How to properly implement an autograd.Function in Pytorch?

username: Shisho_Sama
cooked: <p>Hello everyone, I hope you are having a great time.<br>
I recently wanted to create a simple autoencoder and for that used <a href="https://discuss.pytorch.org/t/how-to-create-a-sparse-autoencoder-neural-network-with-pytorch/3703/6">this thread</a> where <a class="mention" href="/u/smth">@smth</a> provided an example on how to create an autograd Function for the aformentioned autoencoder.<br>
and the code he wrote is this :</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Function

class L1Penalty(Function):

    @staticmethod
    def forward(ctx, input, l1weight):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(self.l1weight)
        grad_input += grad_output
        return grad_input
</code></pre>
<p>However, this code fails completely on newer versions of Pytorch (e.g 1.1.0) with the error indicating the backward method needs to return as many values as the forward method received.</p>
<p>I asked but got no answer and I myself also couldnt specify I dont need a gradient for the second argument. I tried to set <code>ctx.needs_input_grad</code> but thats read-only .<br>
What should I do here? should I simply return None, or 0 for the arguments that I’m not interested in?</p>
<p>by the way what does this part exactly doing?</p>
<pre><code class="lang-auto"> grad_input = input.clone().sign().mul(self.l1weight)
</code></pre>
<p>Can anyone please also clarify this? Thanks a lot</p>

username: Shisho_Sama
cooked: <p>Based on the discussions <a href="https://discuss.pytorch.org/t/solved-what-is-the-correct-way-to-implement-custom-loss-function/3568">here</a> I found out that I should be using None for any inputs that I dont want the gradients for. so this is done.<br>
I however would appreciate if anyone could tell me what that last snippet do though.<br>
its greatly appreciated.</p>

username: smth
cooked: <p>The last snippet is applying the L1 penalty (penalizing the direction of gradient a constant factor of input)</p>

username: Shisho_Sama
cooked: <p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I really appreciate it</p>

username: Shisho_Sama
cooked: <p>May I ask another question if you dont mind?<br>
I noticed we have different ways for imposing/enforcing the sparsity constraint. one other way we can achieve sparsity is likely to get the average of the layers output and treat that as the loss for sparsity and add it e.g. to the reconstruction loss and do the backward pass.<br>
That is simply do in the forward pass :</p>
<pre><code class="lang-auto">    def forward(self, input):
        input = input.view(input.size(0), -1)
        sparsity_loss = 0.0
        output = self.encoder(input)
        sparsity_loss += torch.mean(abs(output))
        if self.use_l1_penalty:
            output = L1Penalty.apply(output, self.l1_weight)
        output = self.decoder(output)
        sparsity_loss += torch.mean(abs(output))
        output = output.view(input.size(0), 1, 28, 28)
        return output, sparsity_loss
</code></pre>
<p>and in the trianing loop :</p>
<pre><code class="lang-auto">for e in range(epochs):
    for imgs,_ in dataloader_train:
        imgs = imgs.to(device)
        output, sparsity_loss = sae_model(imgs)
        loss = criterion(output, imgs)
        
        loss_f = loss + (sparsity_ratio * sparsity_loss)
        optimizer.zero_grad()
        loss_f.backward()
        optimizer.step()
    print(f'epoch: {e}/{epochs} loss_f: {loss_f.item():.6f} loss: { loss.item():.6f}'\
          f' sparsity loss: {sparsity_loss.item():.6f} lr = {scheduler.get_lr()}')
    scheduler.step()
</code></pre>
<p>However I noticed the outcome of these two methods are not the same exactly . why is that?<br>
This is the weights outputs using the first method(L1Penalty) :</p>
<ul>
<li>sample reconstructions at each epoch</li>
<li>encoder_weights_trained:</li>
<li>weight_diff (initial weight - trained_weights)</li>
<li>decoder_weights_trained:</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/149dd5c1ffee072d71bb263bd87965418805f61e.png" data-download-href="https://discuss.pytorch.org/uploads/default/149dd5c1ffee072d71bb263bd87965418805f61e" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/149dd5c1ffee072d71bb263bd87965418805f61e_2_690x257.png" alt="1" data-base62-sha1="2WnHYCtMEU2WKVBc4M7AUJtaTdY" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/149dd5c1ffee072d71bb263bd87965418805f61e_2_690x257.png, https://discuss.pytorch.org/uploads/default/original/2X/1/149dd5c1ffee072d71bb263bd87965418805f61e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/149dd5c1ffee072d71bb263bd87965418805f61e.png 2x" data-dominant-color="373737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">772×288 39.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/954845acf7392c65102c0b3654de8d6f0204f453.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/954845acf7392c65102c0b3654de8d6f0204f453" title="1_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/954845acf7392c65102c0b3654de8d6f0204f453_2_512x500.jpeg" alt="1_1" data-base62-sha1="liBZ6iaqLvTrQ2igL3EFeh7rWoz" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/954845acf7392c65102c0b3654de8d6f0204f453_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/9/954845acf7392c65102c0b3654de8d6f0204f453_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/954845acf7392c65102c0b3654de8d6f0204f453_2_1024x1000.jpeg 2x" data-dominant-color="8C8C8C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1_1</span><span class="informations">1513×1477 754 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/acdb2f6f30c106aeffe504a610586baa277feabd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/acdb2f6f30c106aeffe504a610586baa277feabd" title="1_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/acdb2f6f30c106aeffe504a610586baa277feabd_2_512x500.jpeg" alt="1_2" data-base62-sha1="oF9J5uIUZSPYmTSRrD0XyTdwIxv" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/acdb2f6f30c106aeffe504a610586baa277feabd_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/a/acdb2f6f30c106aeffe504a610586baa277feabd_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/acdb2f6f30c106aeffe504a610586baa277feabd_2_1024x1000.jpeg 2x" data-dominant-color="8F8F8F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1_2</span><span class="informations">1513×1477 692 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/499877f4c4f655a3bd2efb8be8768a2f0b56147f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/499877f4c4f655a3bd2efb8be8768a2f0b56147f" title="1_3"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/499877f4c4f655a3bd2efb8be8768a2f0b56147f_2_512x500.jpeg" alt="1_3" data-base62-sha1="av3w6i7HRCYc1h0D0i1lIiGVXuT" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/499877f4c4f655a3bd2efb8be8768a2f0b56147f_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/4/499877f4c4f655a3bd2efb8be8768a2f0b56147f_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/499877f4c4f655a3bd2efb8be8768a2f0b56147f_2_1024x1000.jpeg 2x" data-dominant-color="919191"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1_3</span><span class="informations">1513×1477 1.17 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And using the sparsity loss from activations:</p>
<ul>
<li>encoder_weights_trained:</li>
<li>weight_diff (initial weight - trained_weights)</li>
<li>decoder_weights_trained:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ea6f6f9c4bf0456688cbb15b7f34bfc474142bc4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ea6f6f9c4bf0456688cbb15b7f34bfc474142bc4" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea6f6f9c4bf0456688cbb15b7f34bfc474142bc4_2_512x500.jpeg" alt="2" data-base62-sha1="xrUtFCcTfQ1nopwVHWbfUwwT3Ni" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea6f6f9c4bf0456688cbb15b7f34bfc474142bc4_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea6f6f9c4bf0456688cbb15b7f34bfc474142bc4_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea6f6f9c4bf0456688cbb15b7f34bfc474142bc4_2_1024x1000.jpeg 2x" data-dominant-color="919191"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1513×1477 1.07 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aaa04f70941ea7e7ccb9b1a38f6b99db24274f25.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/aaa04f70941ea7e7ccb9b1a38f6b99db24274f25" title="2_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aaa04f70941ea7e7ccb9b1a38f6b99db24274f25_2_512x500.jpeg" alt="2_2" data-base62-sha1="olqDg9AzwYybJBIrYENQ3r099kh" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aaa04f70941ea7e7ccb9b1a38f6b99db24274f25_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/a/aaa04f70941ea7e7ccb9b1a38f6b99db24274f25_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/aaa04f70941ea7e7ccb9b1a38f6b99db24274f25_2_1024x1000.jpeg 2x" data-dominant-color="919191"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2_2</span><span class="informations">1513×1477 984 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/81ee984b1f21e707505b1803904b312eff94d0c5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/81ee984b1f21e707505b1803904b312eff94d0c5" title="2_3"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/81ee984b1f21e707505b1803904b312eff94d0c5_2_512x500.jpeg" alt="2_3" data-base62-sha1="ixqMpLhQbyLn9HDYlPN0SDlPKE5" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/81ee984b1f21e707505b1803904b312eff94d0c5_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/8/81ee984b1f21e707505b1803904b312eff94d0c5_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/81ee984b1f21e707505b1803904b312eff94d0c5_2_1024x1000.jpeg 2x" data-dominant-color="939393"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2_3</span><span class="informations">1513×1477 1.09 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/79f40a39924f2f8b7fb17812f97600d37c26df93.png" data-download-href="https://discuss.pytorch.org/uploads/default/79f40a39924f2f8b7fb17812f97600d37c26df93" title="2_4"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/79f40a39924f2f8b7fb17812f97600d37c26df93_2_690x257.png" alt="2_4" data-base62-sha1="hoQCSwgJY2mVknKwYfjlOvnZN8T" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/79f40a39924f2f8b7fb17812f97600d37c26df93_2_690x257.png, https://discuss.pytorch.org/uploads/default/original/2X/7/79f40a39924f2f8b7fb17812f97600d37c26df93.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/79f40a39924f2f8b7fb17812f97600d37c26df93.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2_4</span><span class="informations">772×288 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
<p>They use the same exact hyperparameters for optimization which is SGD with lr=0.98 for 20 epochs.<br>
shouldn’t they have the same effect on weights? or is it because of the <code>mean()</code> of the activations, that the gradients will be spread more homogeneously and thus their effect is much milder than when we directly add a term to the gradients?</p>
<p>Thank you very much in advance</p>