akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Net.cuda() move all parameters to GPU, then how to ensure the autograd?

username: zplovekq
cooked: <p>If i do this simply code:</p>
<pre><code class="lang-auto">  w=torch.tensor([1.0],requires_grad=True)
  w=w.cuda()
  #some compute 
  l.backward()
  #w.grad will still be NoneType
</code></pre>
<p>Then the tensor w will not be the leaf node in autograd’s graph (the w will be copybackward).<br>
The parameters in net are all initialized as “requires_grad=True”.<br>
So if i do net.cuda(),  is this equal to do <code>param=param.cuda()</code>? Could meet the same problem above?<br>
Looking forward for apply!</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you do net.cuda(), you should do it <em>before</em> you give the parameters to the optimizer.<br>
And the conversion to cuda is actually not done in a differentiable manner, so the Parameters remains leaf (but now on a different device).</p>

username: zplovekq
cooked: <p>Thanks for your reply!~<br>
I understand by this:<br>
if I do <code>tensor.cuda()</code>, it will return a copy in GPU and this operation is differentiable( copybackward).<br>
But if i do <code>net.cuda()</code> , the return value is <code>self</code>. So this operation is not differentiable( not  copy but just move?)<br>
If i understand right ? There is difference between  <code>tensor.cuda()</code>   and <code>net.cuda()</code> ?</p>

username: albanD
cooked: <p>Yes the two are very different.</p>
<p><code>tensor.cuda()</code> is out of place: <code>tensor</code> itself is not modified.<br>
The operation on Tensor (like all ops on Tensors) is differentiable.</p>
<p><code>net.cuda()</code> is inplace! It changes the nn.Module and the returned value is the same as net.<br>
The operation is not differentiable, you can see it as doing:</p>
<pre><code class="lang-auto"># In pseudo code (this won't work with nested nn.Module
with torch.no_grad():
    for name, param in net.named_parameters():
        net.name = param.cuda()
</code></pre>

username: zplovekq
cooked: <p>Thanks~ I do some experiment:<br>
For tensor, the code like this:</p>
<pre><code class="lang-auto">b=torch.tensor([1.0,2.0],requires_grad=True)

with torch.no_grad():
   b=b.cuda() #b.requires_grad will be false
   b.requires_grad=True

y=(b*b).sum()
y.backward()
b.grad #this will work
</code></pre>
<p>And for net, i can do this:</p>
<pre><code class="lang-auto">for name, param in net.named_parameters():
        with torch.no_grad(): #I find dropout this line not effect the result,why....
            exec(f'net.{name}=torch.nn.Parameter(net.{name}.cuda())') #this will work
</code></pre>
<p>The above code will work as expected. And  for net, what is the effect for <code>with torch.no_grad()</code>? Because i find without this line it still works as expected.<br>
Thanks a lot!</p>

username: albanD
cooked: <p>It has no effect because a <code>nn.Parameter()</code> is <em>always</em> a leaf (so that its .grad field will be populated when doing .backward()). And so wrapping inside the Parameter breaks the differentiability. This is why the <code>no_grad()</code> has no effect.</p>
Output of nn.NLLLoss()

username: hiepnguyen034
cooked: <p>I was a bit confused when testing the <code>nn.NLLLoss()</code> today. For example, let’s say  I have</p>
<pre><code class="lang-auto">loss = nn.NLLLoss()
y_hat = torch.FloatTensor([[0.7,0.1,0.2]])
y=torch.LongTensor([0])
</code></pre>
<p>Then,<code> loss(y_hat,y)</code> returns <code>-0.7</code>. Isn’t it true that by definition, the NLL in this case should be <code>-log(0.7) = 0.35</code>.</p>
<p>Is there something that I am missing? Any help would be much appreciated</p>

username: KFrank
cooked: <p>Hi Hiep!</p>
<aside class="quote no-group" data-username="hiepnguyen034" data-post="1" data-topic="88544" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hiepnguyen034/48/12872_2.png" class="avatar"> hiepnguyen034:</div>
<blockquote>
<p>Then,<code> loss(y_hat,y)</code> returns <code>-0.7</code>. Isn’t it true that by definition, the NLL in this case should be <code>-log(0.7) = 0.35</code>.</p>
</blockquote>
</aside>
<p>The issue is that <code>NLLLoss</code> expects <em>log-probabilities</em>, rather than<br>
probabilities, as its input.</p>
<p>From the documentation for <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">NLLLoss</a>:</p>
<blockquote>
<p>The input given through a forward call is expected to contain<br>
log-probabilities of each class.</p>
</blockquote>
<p>If you calculate <code>log (y_hat)</code> yourself, you will get the expected<br>
result.  Here is a pytorch 0.3.0 script and its output</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__
loss = torch.nn.NLLLoss()
y_hat = torch.autograd.Variable (torch.FloatTensor ([[0.7,0.1,0.2]]))
y = torch.autograd.Variable (torch.LongTensor ([0]))
loss (y_hat,y)
y_hat_log = y_hat.log()
loss (y_hat_log, y)
</code></pre>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt; loss = torch.nn.NLLLoss()
&gt;&gt;&gt; y_hat = torch.autograd.Variable (torch.FloatTensor ([[0.7,0.1,0.2]]))
&gt;&gt;&gt; y = torch.autograd.Variable (torch.LongTensor ([0]))
&gt;&gt;&gt; loss (y_hat,y)
Variable containing:
-0.7000
[torch.FloatTensor of size 1]

&gt;&gt;&gt; y_hat_log = y_hat.log()
&gt;&gt;&gt; loss (y_hat_log, y)
Variable containing:
 0.3567
[torch.FloatTensor of size 1]
</code></pre>
<p>(This is done to avoid the separate calculation of <code>softmax()</code> – to get<br>
the probabilities – and <code>log()</code> – to then get the log-probabilities.  Doing<br>
these separately can be numerically unstable.  Instead, <code>NLLLoss</code> is<br>
designed so that you can use the numerically-stable <code>log_softmax()</code><br>
to calculate its input.)</p>
<p>Best.</p>
<p>K. Frank</p>