Pytorch not optimizing my model

username: Junnan
cooked: <p>Hi all,</p>
<p>I have trained a Generator named <strong>netG</strong> (which is fixed in the following code). Given <strong>x</strong>, then I tried use adam optimizer to optimize the following function to get the <strong>z</strong>:</p>
<pre><code class="lang-auto">class loss(nn.Module):
    def __init__(self, nz):
        super().__init__()
        self.z = nn.Parameter(torch.zeros(1, 10, 1, 1),requires_grad=True)
        self.criterion = nn.BCELoss()
    def forward(self,x,netG, netD):
        l_rec = torch.norm(x-netG(self.z))
        print('L_rec:',l_rec.detach().numpy())
        return l_rec
</code></pre>
<p>The optimization process is:</p>
<pre><code class="lang-auto">#Number of repetitions
epoch = 1000
loss_epoch = np.zeros(epoch)
for i in range(epoch):
    #Gradient initialization used in optimizer
    opt.zero_grad()
    lss = model(x, netG, netD)
    loss_epoch[i] = lss.item()
    #Gradient setting
    lss.backward()
    #Perform optimization
    opt.step()
</code></pre>
<p>But the result shows my <strong>z</strong> doesn’t update (loss not decreasing), the loss is showing in the following figure<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e621bc32d136ea783311b544acfbc1910ef57df.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e621bc32d136ea783311b544acfbc1910ef57df" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e621bc32d136ea783311b544acfbc1910ef57df.png" alt="Capture" data-base62-sha1="6Ck9V6Wskgq3wapDVKthwIm1uFx" width="690" height="382" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">976×541 3.83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Can anyone help me?<br>
Thanks!</p>

username: gphilip
cooked: <p>Can you try printing <code>self.z[0][0][0][0]</code> in <code>forward()</code>, to see if it becomes non-zero?</p>

username: Junnan
cooked: <p>Yes, I printed it:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4065b7c94b62ef6c7845a459f07c3ec3e9e70630.png" data-download-href="https://discuss.pytorch.org/uploads/default/4065b7c94b62ef6c7845a459f07c3ec3e9e70630" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4065b7c94b62ef6c7845a459f07c3ec3e9e70630.png" alt="Capture" data-base62-sha1="9bGueRpjFalSceNBpceotM6APok" width="508" height="500" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">554×545 6.92 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It is zero all the time.</p>

username: gphilip
cooked: <p>Is this expected behaviour? If this tensor stays at all zeroes, will your loss decrease? Or does your loss decreasing depend on this tensor changing?</p>

username: Junnan
cooked: <p>No, I am expecting the this <strong>z</strong> should not be zero all the time. My loss decreasing does depend on this tensor. That is why I am expecting the loss to decrease.</p>

username: gphilip
cooked: <p>In that case you have to write code in your <code>netG</code> class to ensure that gradients on <code>z</code> are propagated (I am not sure if this is the correct term; what I mean is that you should tell PyTorch how gradients with respect to <code>z</code> change, when you call <code>netG.forward(z)</code>.).</p>
<p>This is not something that I have done (yet!), so I am not entirely clear on how to do this. I think you need to implement a <code>backward</code> function inside your <code>netG</code> class. The code in <a href="https://discuss.pytorch.org/t/inplace-relu-doesnt-work-for-custom-autograd-functions/129079/6">this message</a> may help you get started.</p>

username: Junnan
cooked: <p>Thanks! I tried to initialize <code>z</code> as <code>torch.rand(1, 10, 1, 1)</code> and then it works. But don’t know why.</p>