When to use detach

Krishna_Garg: <p>If I have two different neural networks (parametrized by model1 and model2) and corresponding two optimizers, would the below operation using model1.parameters <strong>without detach()</strong> lead to change in its gradients? My requirement is that I want to just compute the mean squared loss between the two model parameters but update the optimizer corresponding to model1.</p>
<pre><code class="lang-auto">opt1 = torch.optim.SGD(self.model.parameters(), lr=1e-3)
opt2 = torch.optim.SGD(self.model.parameters(), lr=1e-3)

loss = (self.lamb / 2.) * ((torch.nn.utils.parameters_to_vector(self.model1.parameters()) - torch.nn.utils.parameters_to_vector(self.model2.parameters()))**2).sum()

loss.backward()
opt1.step()
</code></pre>
<p>How can I decide in general whether to use detach for any operation or not?</p>

albanD: <p>Hi,</p>
<p><code>parameters_to_vector</code> is differentiable and so yes gradients will flow back to both models.</p>
<p>In general, there are very limited cases where you need <code>.detach()</code> within your training function. It is most often used when you want to save the loss for logging, or save a Tensor for later inspection but you don’t need gradient information.</p>

Krishna_Garg: <p>Thanks <a class="mention" href="/u/alband">@albanD</a> for the reply.</p>
<p>Follow-up questions for more clarity:</p>
<p>You mentioned about <code>parameters_to_vector</code> being differentiable. So, how can I check whether a function is differentiable or not?</p>
<p>Also, can you be more specific about the operations in which detach is required. For instance, if I pass the model.parameters to any other function or use it like <code>params = list(self.model1.parameters())</code>, will these require the use of <code>detach()</code>?</p>

albanD: <aside class="quote no-group" data-username="Krishna_Garg" data-post="3" data-topic="98147">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishna_garg/48/17850_2.png" class="avatar"> Krishna_Garg:</div>
<blockquote>
<p>You mentioned about <code>parameters_to_vector</code> being differentiable. So, how can I check whether a function is differentiable or not?</p>
</blockquote>
</aside>
<p>In general, all ops in pytorch are differentiable.<br>
The main exceptions are .detach() and with torch.no_grad. As well as functions that work with <code>nn.Parameter</code> that needs to remain leafs and so cannot have gradient history.</p>
<blockquote>
<p>Also, can you be more specific about the operations in which detach is required</p>
</blockquote>
<p>You most likely never need it actually <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Krishna_Garg: <p>The first part makes sense but not the second part. Sorry if that’s annoying.</p>

albanD: <p>Detach is used to break the graph to mess with the gradient computation.<br>
In 99% of the cases, you never want to do that.</p>
<p>The only weird cases where it can be useful are the ones I mentioned above where you want to use a Tensor that was used in a differentiable function for a function that is not expected to be differentiated. And so you can use detach() to express that. But this is a rare case.</p>

zeck: <p>Hi <a class="mention" href="/u/alband">@albanD</a>  i’m in this situation, i need to detach my tensor because the blackbox model that i use in my model don’t take tensor .  How to do that properly please? everything change but my parameters’gradient stuck to zero, i suspect  .detach(). Thank you…</p>

albanD: <p>Hi,</p>
<p>If you want to get gradients through something that our autograd cannot see, you will have to use a custom Function so that you can tell the autograd what the backward is: <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox">Extending PyTorch — PyTorch 1.11.0 documentation</a><br>
From within the <code>forward</code> there, you can just unpack the Tensor into whatever you want and pass that to your black box!</p>

zeck: <p>Thank you so much for your reply. I will try and get back. I still ask my self why all  parameters gradients stuck zero and not None? Even if i initialize bias and weights to certain values, they still change during inference, i’m a beginner and  that confuse me. Thank you.</p>

albanD: <p>Hi,</p>
<p>If the gradients are all zeros, the optimizer will still move if you have weight decay or momentum!</p>

zeck: <p>Ok thank you. I 'm using Adam without weight decay but it’s momentum so i get it. I’ll try your suggestion and get back. So the lesson for now is to never use .detach() into  a model.</p>