Detach() vs requires_grad_(False) -- generating data

theory_buff: <p>I have a custom data generation pipeline which randomly samples 2 torch tensors (using <code>torch.rand()</code>), multiplies them and the product <code>X</code> is used as input to a PyTorch model. I set <code>X.requires_grad_(False)</code> before input to the model, to avoid any unnecessary gradient accumulation or backprop through the data sampling process.</p>
<p><strong>Question</strong>: it seems like <code>X.detach()</code> would be a better approach here, considering I would like to simply ‘detach’ X from the prior computation graph. Would simply doing <code>X.requires_grad_(False)</code> instead of <code>detach()</code> be correct for my purpose? I am only modifying the model parameters during training, and there are no updates to <code>X</code>.</p>
<p>It looks like it would not make a difference according to [<a href="https://discuss.pytorch.org/t/backpropagation-with-requires-grad-false/79494/2">this answer</a>], but I am not sure if in that context, the variables used for generating <code>X</code> are also frozen by using <code>X.requires_grad_(False)</code>.</p>
<p>Thanks!</p>

ptrblck: <p><code>torch.rand</code> will generate a plain tensor and won’t create a computation graph by default, which is also visible by accessing the <code>.grad_fn</code> of the output, which is <code>None</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.rand(10)
print(x.grad_fn)
# None
print(x.requires_grad)
# False
</code></pre>
<p>Calling <code>.requires_grad_(False)</code> or <code>.detach()</code> on this tensor should not change anything.</p>

theory_buff: <p>Thanks for the reply! However, the setting in question is slightly different – X is not a <code>torch.rand()</code> sample but is product of 2 such random samples (i.e. <code>X = U @ V</code> for <code>U = torch.rand((3,3))</code> and likewise for <code>V</code>). Does that change the answer?</p>

ptrblck: <p>No, this does not change the answer as long as <code>U</code> and <code>V</code> are created via <code>torch.rand</code> without setting <code>requires_grad=True</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">U = torch.rand((3, 3))
V = torch.rand((3, 3))
X = U @ V
print(X.grad_fn)
# None
print(X.requires_grad)
# False
</code></pre>
<p>This would change if <code>U</code> or <code>V</code> are created by a differentiable operation (i.e. not a factory method to create tensors) or if you would explicitly set <code>requires_grad=True</code> in their creation.</p>

Arun_Sandy: <p>While trying to install detectron2 and deepspeed I’m getting below errors :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb93aa41c37e3645d35aeba53ad3636424a19bdd.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb93aa41c37e3645d35aeba53ad3636424a19bdd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb93aa41c37e3645d35aeba53ad3636424a19bdd.png" alt="image" data-base62-sha1="qLnEN2sh8saASK83KXfLbrLJSeV" width="596" height="500" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1097×920 41.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have followed the steps to install detectron2 from here (<a href="https://detectron2.readthedocs.io/en/latest/tutorials/install.html" rel="noopener nofollow ugc">Installation — detectron2 0.6 documentation</a>),<br>
Although the above error says that I do not have a “torch” module, I have already installed it and it still doesn’t work.</p>
<p>This is the same case with installing deepspeed:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26af14e7d0bf4baadea1b440de2e43c4afca99d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/26af14e7d0bf4baadea1b440de2e43c4afca99d2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26af14e7d0bf4baadea1b440de2e43c4afca99d2.png" alt="image" data-base62-sha1="5wdfMIj1F4DTngDUjKekMXi3pfA" width="559" height="500" data-dominant-color="121211"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1109×991 44.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My python version is 3.9 and torch version is :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/faf924d87bde707d86fe62cc6f33366f059a1f81.png" alt="image" data-base62-sha1="zOd9FJaFALiOxw43dkt5qkP6O53" width="279" height="82"></p>
<p>Please let me know what might me the issue here.</p>

ptrblck: <p>Your issue doesn’t seem to be related to this topic.</p>

Arun_Sandy: <p>I have started this issue as a new topic in the below link :</p><aside class="quote quote-modified" data-post="1" data-topic="195519">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arun_sandy/48/58945_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/not-able-to-install-detectron2-and-deepspeed-libraries/195519">Not able to install detectron2 and deepspeed libraries</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    While trying to install detectron2 and deepspeed I’m getting below errors : 
[image] 
I have followed the steps to install detectron2 from here (<a href="https://detectron2.readthedocs.io/en/latest/tutorials/install.html" class="inline-onebox-loading" rel="noopener nofollow ugc">https://detectron2.readthedocs.io/en/latest/tutorials/install.html</a>), 
Although the above error says that I do not have a “torch” module, I have already installed it and it still doesn’t work. 
This is the same case with installing deepspeed: 
[image] 
My python version is 3.9 and torch version is : 
[image] 
Please let me know what might me the issue he…
  </blockquote>
</aside>

<p>Please have a look at it.</p>