Multiple Networks Multiple Losses

username: PavanMV
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/798fe0ec395d4905136205e046c4f6ec64215ad7.png" data-download-href="https://discuss.pytorch.org/uploads/default/798fe0ec395d4905136205e046c4f6ec64215ad7" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/798fe0ec395d4905136205e046c4f6ec64215ad7_2_502x500.png" alt="Capture" data-base62-sha1="hlo20ASgKo5IimPE4c5jFicBXbF" width="502" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/798fe0ec395d4905136205e046c4f6ec64215ad7_2_502x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/798fe0ec395d4905136205e046c4f6ec64215ad7_2_753x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/798fe0ec395d4905136205e046c4f6ec64215ad7_2_1004x1000.png 2x" data-dominant-color="262525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1502×1494 310 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Setup</strong> - <em>Only for scenario explanation, any technical gaps please ignore</em> :</p>
<p>Let’s say for a hypothetical robotic system where all the networks shown above are designed to handle different functionalities based on what it sees, i.e., Input for all the models is the  same (camera) and outputs are :<br>
Out1: Object tracking<br>
Out2: Ego Path estimation and Collision probability<br>
Out3: Face recognition<br>
Out4: Object recognition<br>
Out5: Object segmentation</p>
<p>Opt1, Opt2, Opt3 are optimizers defined for NN1, NN2, NN3.<br>
and Opt45 common for [NN4,NN5]</p>
<p>The assumption is all the networks have to be trained together with losses defined as shown (Just to get a holistic understanding of backprop.)</p>
<p>Note: NN1 and NN2 are inter-dependent on Out1 and Out2, but they don’t explore the same loss function (L1 != L2).</p>
<p>How can we achieve the backward prop shown in figure (in green), what should be the sequence of loss.backward() and optimizer.step() calls for each loss and optimizer?</p>
<pre><code class="lang-auto">L1.backward()
Opt1.step()

L2.backward()
Opt2.step()

L3.backward()
Opt3.step()

L45.backward()
Opt45.step()
</code></pre>
<p>will this achieve it? if yes is this the best way to do?</p>
<p>Also In this example Does L1.backward() update the graph only for NN1 and NN2?</p>

username: ayalaa2
cooked: <p>So when you call backwards on <strong>L1</strong>, it’ll traverse backwards and assign gradients to any parameters who are to blame for its loss. What I’m noticing is that <strong>L1</strong> takes as input, the output of <strong>NN1</strong> and <strong>NN2</strong>, therefore gradients will flow through both of them. But according to your diagram, do you not want <strong>L1</strong> to flow through <strong>NN2</strong>? If this is the case, <strong>L1</strong> <em>should not</em> take <strong>out2</strong> as input. What you can do is detach <strong>out2</strong> from the computational graph and use it as a constant in your loss (as to not allow gradients to flow through it), but since your loss is a sum of the two, this won’t achieve anything.</p>
<p>So I would suggest removing <strong>out2</strong> from <strong>L1</strong> and <strong>out1</strong> from <strong>L2</strong> for similar reasons.</p>
<p><strong>L3</strong> looks completely fine to me.</p>
<p><strong>L45</strong> also looks fine to me. The order at which you call backwards won’t really matter since they’re not overlapping with each other. The key thing is that what parameters that affected your loss comes before the optimizer that is in charge of those parameters; which you seem to be doing in your code.</p>

username: PavanMV
cooked: <p>Hi Alex,</p>
<p>Thanks for the explanation, but i am looking for a scenario where L1 and L2 have to be trained together.</p>
<p>I have edited the post for more clarity on my need. Thanks.</p>

username: ayalaa2
cooked: <p>I guess I’m still confused. Should NN2 be updated based on L1 at all?</p>

username: PavanMV
cooked: <p>No NN2 will not use L1 for its training, it just uses Out1 in its loss L2.</p>
<p>In the given scenario, Out2 and Out1 are inter dependent, Object/obstacle tracking and ego path estimation are correlated tasks, however NN1 (object tracking network) need not have the same loss as NN2, it just uses NN2-Out2 for its own loss (L1) calculation.</p>
<p>Think of it as an unsupervised cross task scenario.</p>

username: goku7
cooked: <p><a class="mention" href="/u/ayalaa2">@ayalaa2</a> if you don’t put the parameters of NN2 in Opt1, it won’t be updated.</p>

username: ayalaa2
cooked: <p><a class="mention" href="/u/goku7">@goku7</a> That’s true, but loss will still be back propagated through NN1 via L2. Even if the parameters aren’t updated, this is a little inefficient.</p>
<p><a class="mention" href="/u/pavanmv">@PavanMV</a> If NN2 shouldn’t use L1 for training and NN1 shouldn’t use L2 for its training, then their loss terms shouldn’t contain the other network’s output.</p>
<p>What I can imagine happening is that on your first L1 backwards call, the parameters of NN1 and NN2 are given gradients due to their output being utilized in the loss. But we don’t want NN2 to be updated via L1 (NN2 will keep those gradients and opt2 will utilize them). You can detach out2 so that the loss can still be computed (but gradients won’t flow through NN2), but adding a constant value won’t affect your gradients. Thus I don’t see why out2 is needed in L1.</p>
<p>Unless there exists an operation that utilizes out2 that <em>will</em> affect your gradients in NN1, I would argue that term is not needed in your loss term. The statement <code>L1 = f1(out1) + f2(out2)</code> suggests that this is the case.</p>
<p>It’s possible that you still need to compute that term for application purposes, but I don’t believe it should be included in your loss term that you perform back propagation on.</p>

username: PavanMV
cooked: <p><a class="mention" href="/u/ayalaa2">@ayalaa2</a> With given formulation I agree with you, maybe below formulation will closely represent my intention. Sorry for the confusion.</p>
<blockquote>
<p>L1 = f1(out1,out2)<br>
L2 = g1(out1,out2)</p>
</blockquote>
<p>Will this take care of training both NN1 and NN2 together</p>
<blockquote>
<p>L1.backward()<br>
Opt1.step()<br>
Opt1.zero_grad(),Opt2.zero_grad()</p>
<p>L2.backward()<br>
Opt2.step()<br>
Opt1.zero_grad(),Opt2.zero_grad()</p>
</blockquote>

username: ayalaa2
cooked: <p>That looks like it would work. I would suggest this for a performance speed up:</p>
<pre><code class="lang-auto">L1 = f1(out1, out2.detach())
L2 = g1(out1.detach(), out2)

L1.backward()
Opt1.step()
Opt1.zero_grad()

L2.backward()
Opt2.step()
Opt2.zero_grad()
</code></pre>
<p>If we detach the tensor, gradients won’t flow through it.</p>