How are optimizer.step() and loss.backward() related?

Yun: <p>Hi.</p>
<p>I am pretty new to Pytorch and keep surprised with the performance of Pytorch <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have followed tutorials and there’s one thing that is not clear.</p>
<p>How the optimizer.step() and loss.backward() related?</p>
<p>Does optimzer.step() function optimize based on the closest loss.backward() function?</p>
<p>When I check the loss calculated by the loss function, it is just a Tensor and seems it isn’t related with the optimizer.</p>
<p>Here’s my questions:</p>
<p>(1) Does optimzer.step() function optimize based on the closest loss.backward() function?</p>
<p>(2) what happens if I call several different backward() from losses and call optimizer.step()?<br>
Does the optimizer optimize based on all previous called losses?</p>
<p>Thank you!</p>

vabh: <ol>
<li>
<p><code>optimizer.step</code> is performs a parameter update based on the <em>current</em> gradient (stored in <code>.grad</code> attribute of a parameter) and the update rule. As an example, the update rule for SGD is defined here:<br>
<a href="https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/cd9b27231b51633e76e28b6a34002ab83b0660fc/torch/optim/sgd.py#L63</a>.</p>
</li>
<li>
<p>Calling <code>.backward()</code> mutiple times accumulates the gradient (by addition) for each parameter. This is why you should call <code>optimizer.zero_grad()</code> after each <code>.step()</code> call. Note that following the first <code>.backward</code> call, a second call is only possible after you have performed another forward pass.</p>
</li>
</ol>
<p>So for your first question, the update is not the based on the “closest” call but on the <code>.grad</code> attribute. How you calculate the gradient is upto you.</p>

greaber: <aside class="quote no-group" data-username="vabh" data-post="2" data-topic="7350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/898d66/48.png" class="avatar"> vabh:</div>
<blockquote>
<p>Note that following the first .backward call, a second call is only possible after you have performed another forward pass.</p>
</blockquote>
</aside>
<p>This is certainly not true if you specify <code>retain_graph=True</code>, and in some simple cases, it seems to be possible to backpropagate multiple times even without specifying <code>retain_graph=True</code> (but I don’t understand why).  Also, the docks for <code>backward</code> say about <code>retain_graph</code>,</p>
<aside class="quote no-group">
<blockquote>
<p>Note that in nearly all cases setting this option to True is not needed and often can be worked around in a much more efficient way.</p>
</blockquote>
</aside>
<p>But I am not sure if this is really true.  In architectures I have worked with I have often had to specify <code>retain_graph=True</code>, and if there are more efficient ways of doing what I needed to do, I couldn’t find them.  (Is there some explanation somewhere of what these more efficient workarounds are and in what cases they work and in what apparently rare cases they fail?)</p>
<p>For instance, two cases I have encountered are when you have two different loss functions, used to update different parameters, but calculated using some of the same graph, and when you have an RNN and want to do backpropagation through time with overlapping backprop regions (like backprop 512 steps and then 256 steps later backprop another 512 steps).</p>

smth: <p><a class="mention" href="/u/greaber">@greaber</a>, if you have two different loss functions, finish the forwards for both of them separately, and then finally you can do <code>(loss1 + loss2).backward()</code>. It’s a bit more efficient, skips quite some computation i believe.</p>

greaber: <p>But this assumes the different loss functions are used to compute grads for the same parameters, right?  It doesn’t work in a GAN-like situation (although if the two loss functions are literally the negative of each other there might be some shortcut).</p>

smth: <aside class="quote no-group" data-username="greaber" data-post="5" data-topic="7350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/greaber/48/1287_2.png" class="avatar"> greaber:</div>
<blockquote>
<p>But this assumes the different loss functions are used to compute grads for the same parameters, right?</p>
</blockquote>
</aside>
<p>Yes it does, it doesn’t work in a GAN-like situation.</p>

vabh: <aside class="quote no-group" data-username="greaber" data-post="3" data-topic="7350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/greaber/48/1287_2.png" class="avatar"> greaber:</div>
<blockquote>
<p>This is certainly not true if you specify retain_graph=True</p>
</blockquote>
</aside>
<p>You are right, I probably should have mentioned this. I left it out because it is a bit more of an “advanced” use case.</p>

Yun: <p>Thank you! I found it really helpful <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Charles_Earl: <p>I’ve also been trying to understand the how things are happening at the lower level of the cudnn API calls. It seems that the loss and weight update is responsibility of the optimizer. In the case of cuda, that will just handle the output and gradient computation.</p>

Dmitry_Rybakov: <p>I wonder this topic too. There is no explicit connection between optimizer and loss objects in a program. Are they related implicitly via global variables i.e. loss.backward() data recorded somewhere?</p>
<p>I suppose there should be more obvious call like this:<br>
optimzer.step(loss)</p>

aerinykim: <p>Remember we defined <code>optimizer = optim.SGD(parameters())</code>?</p>

Ben_Reich: <p>Yes, but the loss function does not deal with parameters only with predictions</p>

SunHaozhe: <p>What do you mean by saying the following sentence? “this assumes the different loss functions are used to compute grads for the same parameters”.</p>
<p>And what does a GAN-like situation refer to?</p>
<p>I am a little confused and not sure when to use <code>(loss1 + loss2).backward()</code> or <code>loss1.backward(retain_graph=True) loss2.backward()</code>. Why is one more efficient than the other? Are these two methods mathematically equivalent?</p>
<p>According to Sam Bobel in <a href="https://stackoverflow.com/questions/46774641/what-does-the-parameter-retain-graph-mean-in-the-variables-backward-method" rel="noopener nofollow ugc">Stack Overflow - What does the parameter retain_graph mean in the Variable’s backward() method?</a> , these two methods are not mathematically equivalent if one uses adaptive gradient optimizers like ADAM, as shown below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2021388bffe6717332a377df3d3397c02bb86be.png" data-download-href="https://discuss.pytorch.org/uploads/default/c2021388bffe6717332a377df3d3397c02bb86be" title="Screenshot 2020-08-18 at 15.35.32"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2021388bffe6717332a377df3d3397c02bb86be_2_660x500.png" alt="Screenshot 2020-08-18 at 15.35.32" data-base62-sha1="rGh4ONrQ29G1GZR9gXfXknkdA2W" width="660" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2021388bffe6717332a377df3d3397c02bb86be_2_660x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2021388bffe6717332a377df3d3397c02bb86be_2_990x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2021388bffe6717332a377df3d3397c02bb86be_2_1320x1000.png 2x" data-dominant-color="ECEDEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-08-18 at 15.35.32</span><span class="informations">1372×1038 204 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you agree with Sam Bobel?</p>

saba: <p><span class="mention">@smthI</span><br>
I add the second loss to the first loss and expect that the gradients, weights and result changes. But there is no change and difference with the time I just use one loss function. The fist one isBCELoss and the second one is L1.<br>
I check the gradients in both case with loss1 and with loss1+loss2 but gradients were same. exact same. adding more loss does not have effect on gradient even if I used loss1+10*loss2</p>
<pre><code class="lang-auto">netG = Generator994(ngpu,nz,ngf).to(device)

optimizerG = optim.Adam(netG.parameters(), lr=lr2, betas=(beta1, 0.999))

netG.zero_grad()

label.fill_(real_label)  
label=label.to(device)
output = netD(fake).view(-1)
# Calculate G's loss based on this output
loss1 = criterion(output, label)


xxx=torch.histc(Gaussy.squeeze(1).view(-1).cpu(),100, min=0, max=1, out=None)
ddGaussy=xxx/xxx.sum()

xxx1=torch.histc(fake.squeeze(1).view(-1).cpu(),100, min=0, max=1, out=None)
ddFake=xxx1/xxx1.sum()

loss2=abs(ddGaussy-ddFake).sum()

# Calculate gradients for G with 2 loss

errG=loss1+loss2
errG.backward()

for param in netG.parameters():
            print(param.grad.data.sum())
# Update G
optimizerG.step()
 
## ------------------
class Generator994(nn.Module):
    def __init__(self,ngpu,nz,ngf):
        super(Generator994, self).__init__()
        self.ngpu=ngpu
        self.nz=nz
        self.ngf=ngf
        self.l1= nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d(self.nz, self.ngf * 8, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 8),
            nn.ReLU(True),)
            # state size. (ngf*8) x 4 x 4
        self.l2=nn.Sequential(nn.ConvTranspose2d(self.ngf * 8, self.ngf * 4, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 4),
            nn.ReLU(True),)
            # state size. (ngf*4) x 8 x 8
        self.l3=nn.Sequential(nn.ConvTranspose2d( self.ngf * 4, self.ngf * 2, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 2),
            nn.ReLU(True),)
            # state size. (ngf*2) x 16 x 16
        self.l4=nn.Sequential(nn.ConvTranspose2d( self.ngf*2, 1, 3, 1, 0, bias=False),nn.Sigmoid()
#            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        out=self.l1(input)
        out=self.l2(out)
        out=self.l3(out)
        out=self.l4(out)
        print(out.shape)
        return out
</code></pre>

ptrblck: <p>Double post with answer from <a href="https://discuss.pytorch.org/t/why-by-adding-second-loss-and-applying-the-back-ward-generated-gradient-is-as-same-as-before/94625/2">here</a>.</p>