Can I use loss.backward for Backpropagation with respect to the weights of another neural network?

Jeet: <p>I have 2 neural networks (N1 and N2) and 2 loss functions (L1 and L2). The Loss functions of the neural networks are dependent on each other. So the backpropagation of a loss function has to be done with respect to the weights of both the neural networks (say w1 and w2).</p>
<p>Note:- I cannot just backpropagate the two-loss functions separately and record the gradients to form the Jacobian matrix.</p>
<p>D = Jacobian Matrix =</p>
<pre><code>partial derivate of L1 wrt weights of N1, partial derivate of L1 wrt weights of N2
    partial derivate of L2 wrt weights of N1, partial derivate of L2 wrt weights of N2</code></pre>

KFrank: <p>Hi Jeet!</p>
<p>Yes, you can use <code>loss.backward()</code> to calculate gradients with respect<br>
to the weights of multiple networks.</p>
<p>Pytorch’s autograd facility calculates gradients with respect to tensors<br>
that have <code>requires_grad = True</code>.  It doesn’t matter whether such<br>
tensors are wrapped in <code>Parameter</code>s or belong to a <code>Module</code> or do not<br>
all belong to the same network.</p>
<aside class="quote no-group" data-username="Jeet" data-post="1" data-topic="146306" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeet/48/47500_2.png" class="avatar"> Jeet:</div>
<blockquote>
<p>I have 2 neural networks (N1 and N2) and 2 loss functions (L1 and L2).</p>
</blockquote>
</aside>
<p>So far so good.</p>
<aside class="quote no-group">
<blockquote>
<p>The Loss functions of the neural networks are dependent on each other.</p>
</blockquote>
</aside>
<p>I don’t understand specifically what you mean by this.</p>
<aside class="quote no-group">
<blockquote>
<p>So the backpropagation of a loss function has to be done with respect to the weights of both the neural networks (say w1 and w2).</p>
</blockquote>
</aside>
<p>This can certainly be done.  Consider a made-up example:</p>
<pre data-code-wrap="python"><code class="lang-python">pred1 = N1 (input1)
pred2 = N2 (torch.cat (input2, pred1))   # the output of N1 is fed into N2
loss1 = L1 (pred1, targ1)   # depends only on weights of N1
loss2 = L2 (pred2, targ2)   # depends on the weights of both N1 and N2
loss = loss1 + loss2        # also depends on the weights of both N1 and N2
loss.backward()             # calculates the gradient of loss with respect to the weights of both N1 and N2
</code></pre>
<p>This will work fine.  Again, autograd only knows and cares that the<br>
weights of <code>N1</code> and <code>N2</code> are leaves of the computation graph it is<br>
processing (with <code>requires_grad = True</code>).  It doesn’t matter that<br>
these leaves happen to belong to different networks.</p>
<p>(As an aside, the pytorch api doesn’t really have the concept of a<br>
network, per se.  The closest thing would be a <code>Module</code>.  What we<br>
call a network would often be packaged as a <code>Module</code>, but it doesn’t<br>
have to be.  Also, there are plenty of <code>Module</code>s that we would not<br>
think of as being networks and some commonly used <code>Module</code>s<br>
contain no <code>Parameter</code>s or other trainable weights.  So autograd<br>
can neither know nor care whether certain weights are in this “network”<br>
or that “network” because, at a technical level, a “network” isn’t really<br>
a thing in pytorch.)</p>
<p>Best.</p>
<p>K. Frank</p>

Jeet: <p>Hi Frank, thanks a lot for the detailed explanation. I am actually interested to apply this concept in the case of a Generative adversarial network (GANs) where we have 2 loss functions and 2 neural networks of the generator and the discriminator respectively. I am training the generator and the discriminator separately with loss.backward() and have also stored the weights of the two neural networks in a large vector. Can I just add both the losses here as well and call loss.backward() to do the job? or do I need to work on the loss functions separately and generate the partial derivatives with respect to the large vector containing the weights of both the neural networks?</p>

KFrank: <p>Hi Jeet!</p>
<aside class="quote no-group quote-modified" data-username="Jeet" data-post="3" data-topic="146306" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeet/48/47500_2.png" class="avatar"> Jeet:</div>
<blockquote>
<p>I am actually interested to apply this concept in the case of a Generative adversarial network (GANs)<br>
…<br>
call loss.backward() to do the job?</p>
</blockquote>
</aside>
<p>Almost, but with an important detail.  When you backpropagate from<br>
the discriminator back up into the generator, you need to flip the sign<br>
of the gradient.</p>
<p>(Also, I see so benefit to storing the network weights in a separate<br>
“large vector.”  Just leave them in the networks themselves.)</p>
<p>I’ve never built a GAN, so I am fuzzy on the details, but the basic<br>
idea is as follows:</p>
<p>You have a generator network (<code>Gen</code>) that produces “fake” images<br>
that look real and you have a discriminator network (<code>Disc</code>) whose<br>
job it is to tell the fake images apart from teal ones.</p>
<p>So <code>Disc</code> is an ordinary classifier – “fake” vs. “real” – and you can<br>
train it with something like <code>BCEWithLogitsLoss</code>.</p>
<p>But the idea of a GAN is to also train <code>Gen</code> to generate fake images<br>
that fool <code>Disc</code> into classifying them as real.  The scheme is to train<br>
<code>Disc</code> so that the loss from <code>Disc</code> goes <em>down</em> but train <code>Gen</code> so that the<br>
loss from <code>Disc</code> goes <em>up.</em></p>
<p>You can do this as follows:</p>
<p>Feed a real image into <code>Disc</code> and calculate the classification loss.<br>
Backpropagate it through <code>Disc</code>, updating <code>Disc</code>’s weights.</p>
<p>Now feed some random input into <code>Gen</code>.  <code>Gen</code> acts sort of like a decoder<br>
and “decodes” the random input into a fake image.  The fake image output<br>
by <code>Gen</code> depends on <code>Gen</code>’s weights and has <code>requires_grad = True</code>.<br>
Feed this fake image into <code>Disc</code>, calculate the classification loss and<br>
backpropagate it.  This also updates <code>Disc</code>’s weights, continuing to train<br>
it to distinguish fake from real.</p>
<p>The key point:</p>
<p>When we further backpropagate <code>Disc</code>’s classification loss for the<br>
fake image through <code>Gen</code> – which we can do because the input to<br>
<code>Disc</code> came from <code>Gen</code>, depends on <code>Gen</code>’s weights, and carries<br>
<code>requires_grad = True</code> – we <em>flip the sign</em> of the gradient.  This<br>
is because we want to penalize <code>Gen</code> if <code>Disc</code> did well, and reward<br>
<code>Gen</code> if <code>Disc</code> did poorly when classifying the fake image.  That is,<br>
we train <code>Gen</code> and <code>Disc</code> at cross-purposes with one another.</p>
<p>One convenient way to effect this gradient sign-flip is to interpose<br>
a “sign-flip” <code>Function</code> between <code>Gen</code> and <code>Disc</code>.  During the forward<br>
pass, the sign-flip <code>Function</code> simply passes its input through unchanged.<br>
(That is, we pass the fake image generated by <code>Gen</code> unchanged into<br>
<code>Disc</code>.)  But on the backward pass the sign-flipper takes the gradient<br>
it’s given and flips its sign before sending it on to <code>Gen</code> for further<br>
backpropagation.</p>
<p>I’m not aware that pytorch offers a pre-packaged sign-flip <code>Function</code>,<br>
but it’s easy enough to write one.</p>
<p>Some additional discussion about flipping the gradient’s sign can be<br>
found in this thread:</p>
<aside class="quote" data-post="1" data-topic="134810">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hadaev8/48/16280_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-reverse-gradient-sign-during-backprop/134810">How to reverse gradient sign during backprop?</a> 
  </div>
  <blockquote>
    For example, I have models net0 and net1 working like this 
out0 = net0(inp0)
out1 = net0(net1(net0(inp1)))
loss = criterion(out0, out1, target)

I want to reverse the gradient sign from the output of model net1 but calculate net0(inp0) as usual. 
In simple case, I would do 
out0 = net0(inp0)
out1 = net0(net1(inp1))
loss = criterion(out0, out1, target)
loss.backward()
[p.grad.data.neg_() for p in net1.parameters()]
opt.step()

Can I do some hook to reverse gradients from output of net1?
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>