Loss.backward() for two different nets

mbehzad: <p>Hi, let’s say I have two networks, “net1” and “net2” with “loss1” and “loss2” representing the loss function of “net1” and “net2”, and “optimizer1” and “optimizer2” are the optimizers of both networks.</p>
<p>My losses are computed as:<br>
loss1 = criterion(outputs1, labels1)<br>
loss2 = criterion(outputs2, labels2)</p>
<p>Now, I want to backprop the loss to net1 and net2. I use:<br>
loss1.backward()<br>
loss2.backward()<br>
optimizer1.step()<br>
optimizer2.step()</p>
<p>Is this a correct way? I am just confused about how is loss1 associated with net1 for computing gradients, and loss2 with net2? I know that the loss1 is computed from outputs of net1, but I want to ask how are the gradients in “loss1.backward()” computed from “net1”.</p>
<p>My goal: compute the loss1 from net1 and just backprop to net1, and same for net2.</p>

albanD: <p>Hi,</p>
<p>The trick we use is to store with the Tensor enough information to know how it was created (to be able to compute the gradients).<br>
So the <code>loss1</code> “knows” that it was computed based on the parameters from net1 and so when you call .backward, it will compute the gradients wrt to these parameters. (whatever you do on the side with other nets).</p>

David_Alford: <p>PyTorch comes with a component called <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">autograd</a>  which provides automatic differentiation for all operations on Tensors, and Tensors which can remember where they “came from”.</p>
<p>From the PyTorch docs:</p>
<blockquote>
<p><code>torch.Tensor</code>  is the central class of the package. If you set its attribute  <code>.requires_grad</code>  as  <code>True</code> , it starts to track all operations on it. When you finish your computation you can call  <code>.backward()</code>  and have all the gradients computed automatically. The gradient for this tensor will be accumulated into  <code>.grad</code>  attribute.</p>
</blockquote>
<p>When a Neural Network is defined in PyTorch it uses the base class of <code>Torch.nn.Module</code>. All your <em>submodules</em> and <em>layers</em> can be initialized in the module - which will lead to them being tracked by the <code>Module</code>.</p>
<p>Let us say <code>Net1</code> looked like this (subclassing <code>nn.Module</code>)</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

class Net1(nn.Module):
    def __init__(self):
        super(Net1, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))
</code></pre>
<p>Then when you call the model you get an output:</p>
<pre><code class="lang-auto">output = Net1(x)
</code></pre>
<p>This output has been propagated through the <code>forward()</code> pass of <code>Net1</code> (among other methods). Then you calculate the loss:</p>
<pre><code class="lang-auto">loss1 = criterion(outputs1, labels1)
</code></pre>
<p>Now we call the <code>.backward()</code> method on the optimizer, autograd will backpropogate through the <code>tensor</code>s  which have <code>requires_grad</code> set to <code>True</code> and calculate the gradient w.r.t the parameters all the way back to where they came from.</p>
<p>Then when you call <code>optimizer1.step()</code> it will look into <code>params.grad</code> and update the value of the <code>params</code> by subtracting the <code>learning_rate</code> times the <code>grad</code> from it.</p>
<p>The key here is that Tensors know where they came from and each Net is backpropogated automatically - so both your Nets know exactly where they came from. This abstracts it away from the user which makes it quite friendly <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Note: I am newer to PyTorch so if I explain something wrong someone please feel free to correct me.</p>
<p>David Alford</p>

mbehzad: <p>Thanks to both of you for your responses, it makes sense now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>