Backpropagating the loss through side network

harisgulzar1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/046c77f5467c7358fd7f2830731ed0c4fa8e5f26.png" data-download-href="https://discuss.pytorch.org/uploads/default/046c77f5467c7358fd7f2830731ed0c4fa8e5f26" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/046c77f5467c7358fd7f2830731ed0c4fa8e5f26.png" alt="image" data-base62-sha1="D8iLNit2JgSWuuDtCG9YzvJ8KW" width="690" height="344" data-dominant-color="F0F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1349×673 7.17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have a Main Network (large transformer) and then I add Side Network.<br>
My goal is to use the Side Network for fine-tuning and not update the weights of Main Network.<br>
One way to do this is to freeze the Main Network weights so the weights are not updated.<br>
But the problem is that even if the weights are not updated, the backpropagation graph (for Main Network + Side Network) is still computed which is time and memory-consuming. As I have direct connections between side network modules. I want to only backpropagate through the side network.<br>
Currently, I have just one loss <code>loss=Y-label</code>. Is there a way to use the same loss for backpropagation of ONLY side network?</p>

smth: <p>look at the <code>detach</code> function:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html</a></p>

harisgulzar1: <p>Thanks for your reply.<br>
I checked it before, and apparently, it does not seem to have enough explanation.</p>
<p>Using detach with Main Network <code>MainNetwork.detach()</code> would exclude it from the computation graph when doing the backpropagation on <code>loss</code>. Is my understanding correct?</p>

smth: <p>Take this example code below. In this example, when you do <code>Y.backward()</code>, then it wont backprop through MainNetwork</p>
<pre data-code-wrap="python"><code class="lang-python">output = MainNetwork(X)
output.detach_()
Y = SideNetwork(output)
</code></pre>

harisgulzar1: <p><a class="mention" href="/u/smth">@smth</a> Thanks for the extra information.<br>
I have two concerns with this:</p>
<ol>
<li>The code seems to be considering a series connection of the Main and Side Networks but in our case, it is parallel (with intermediate connections).</li>
<li>In our case, although the MainNetwork and SideNetwork are shown separately, the SideNetwork layers are defined within the MainNetwork’s forward function, because they have to process the same input.</li>
</ol>
<p>Well, by using the <code>detach_()</code> function at the MainNetwork’s output, the MainNetwork seems to have been excluded from the computation graph and seems to be only backpropagating through the layers of SideNetwork. But, do you have any comments on this?<br>
Thanks!</p>

J_Johnson: <p>Here are the considerations you want to make when fine-tuning a side network:</p>
<ol>
<li>Only pass the side network into the optimizer. In this way, only the side network has all of the optimizer parameters tracked.</li>
<li>Use model.eval() on the main network, so you turn off any Dropout and batchnorm layers operate in eval mode.</li>
<li>You might use the <code>with torch.no_grad():</code> wrapper on the main network forward pass or <code>.detach_()</code> on the outputs, as <a class="mention" href="/u/smth">@smth</a> mentioned.</li>
</ol>