Pytorch general question

JhonPool4: <p>Hi :D,</p>
<p>I want to binarize a neural network (BNN) for a project. So I am reading  the work of Rastegari [<a href="https://arxiv.org/pdf/1603.05279.pdf?source=post_page---------------------------" rel="nofollow noopener">1</a>] and his code on GitHub [<a href="https://github.com/jiecaoyu/XNOR-Net-PyTorch/blob/master/CIFAR_10/util.py" rel="nofollow noopener">2</a>].</p>
<p>One part of the code is the following:</p>
<pre><code class="lang-auto">    def updateBinaryGradWeight(self):
        for index in range(self.num_of_params):
            weight = self.target_modules[index].data
            n = weight[0].nelement()
            s = weight.size()
            m = weight.norm(1, 3, keepdim=True)\
                    .sum(2, keepdim=True).sum(1, keepdim=True).div(n).expand(s)
            m[weight.lt(-1.0)] = 0 
            m[weight.gt(1.0)] = 0
            # m = m.add(1.0/n).mul(1.0-1.0/s[1]).mul(n)
            # self.target_modules[index].grad.data = \
            #         self.target_modules[index].grad.data.mul(m)
            m = m.mul(self.target_modules[index].grad.data)
            m_add = weight.sign().mul(self.target_modules[index].grad.data)
            m_add = m_add.sum(3, keepdim=True)\
                    .sum(2, keepdim=True).sum(1, keepdim=True).div(n).expand(s)
            m_add = m_add.mul(weight.sign())
            self.target_modules[index].grad.data = m.add(m_add).mul(1.0-1.0/s[1]).mul(n)
</code></pre>
<p>I am starting with PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> . I don’t understand what’s mean that " \ "  and  how its working “.sum(2, keepdim = True)”:</p>
<pre><code class="lang-auto"> m = weight.norm(1, 3, keepdim=True)\
                    .sum(2, keepdim=True).sum(1, keepdim=True).div(n).expand(s)
</code></pre>
<p>Thanks for your help,<br>
Regards,</p>

albanD: <p><code>.sum(2, keepdim=True)</code> means to perform the sum operation over the 2nd dimension. <code>keepdim</code> means that after reduction, that dimension remains but has size 1.</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: a = torch.rand(2, 3, 4)

In [3]: a.sum(2, keepdim=False).size()
Out[3]: torch.Size([2, 3])

In [4]: a.sum(2, keepdim=True).size()
Out[4]: torch.Size([2, 3, 1])
</code></pre>

JhonPool4: <p>Thanks, and do you know how is working " \ " in :</p>
<pre><code class="lang-auto"> m = weight.norm(1, 3, keepdim=True)\
                    .sum(2, keepdim=True).sum(1, keepdim=True).div(n).expand(s)
</code></pre>

albanD: <p>This is a python thing. It allows you to write something that should be on a single line in multiple lines. This has nothing to do with pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

JhonPool4: <p>Hi, do you know what a have this warning:</p>
<pre><code class="lang-auto">/pytorch/torch/csrc/autograd/python_function.cpp:622: UserWarning: Legacy autograd function with non-static forward method is deprecated and will be removed in 1.3. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">class BinActive(torch.autograd.Function):
    '''
    Binarize the input activations and calculate the mean across channel dimension.
    '''

    def forward(self, input):
        self.save_for_backward(input)
        size = input.size()
        mean = torch.mean(input.abs(), 1, keepdim=True)
        input = input.sign()
        return input, mean

    def backward(self, grad_output, grad_output_mean):
        input, = self.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input.ge(1)] = 0
        grad_input[input.le(-1)] = 0
        return grad_input
</code></pre>

albanD: <p>Hi,</p>
<p>As mentionned, this is the old style of autograd.Function.<br>
You can find info about this <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a> as mentioned in the warning.</p>