When the parameters are set on cuda(), the backpropagation doesnt work

Shisho_Sama: <p>This is a follow up to my previous question asked here <a href="https://discuss.pytorch.org/t/how-can-i-insert-a-branch-variable-in-a-model-graph-in-pytorch/35199">How can I insert a branch variable in a model graph in pytorch?</a>,</p>
<p>The problem is , I noticed when training, the GPU utilization is very bad, and at first I thought maybe its because I didn’t make my parameters use <code>cuda()</code> it seems a lot is being off loaded to CPU instead of GPU. here is a screenshot during training.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/99c4f0e929a7d3ca6b7b536a1273206e15695b7e.gif" data-download-href="https://discuss.pytorch.org/uploads/default/99c4f0e929a7d3ca6b7b536a1273206e15695b7e" title="pytorch_customparameter"><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/99c4f0e929a7d3ca6b7b536a1273206e15695b7e.gif" alt="pytorch_customparameter" data-base62-sha1="lWiZPqmvzxS3VLXT8dBz2IzCSBM" width="404" height="500" data-dominant-color="E2D8D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_customparameter</span><span class="informations">467×577 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, when I simply changed my parameter from</p>
<pre><code class="lang-auto">self.mybranch = nn.Parameter(torch.zeros(1))
</code></pre>
<p>to</p>
<pre><code class="lang-auto">self.mybranch = nn.Parameter(torch.zeros(1)).cuda()
</code></pre>
<p>Not only the GPU utilization is not fixed, even worse, the backpropagation now wont work for my parameter at all and it wont get tuned!<br>
If I remove <code>.cuda()</code>, it gets trained fine!<br>
So here I’m facing two issues,</p>
<ol>
<li>Why using <code>.cuda()</code> prevents my parameter from participating in the optimization process?</li>
<li>Why is GPU not utilized like before?</li>
</ol>
<p>by the way, my model uses <code>.cuda()</code> ( the <code>CrossEntropyLoss</code> also has .cuda() but it seems it works fine without it too! I mean it seems it doesn’t make any difference whether I use <code>.cuda()</code> on criterion)</p>
<p>What is wrong here?<br>
What am I missing here?</p>

albanD: <p>Hi,</p>
<p>To use cuda, you should create your model as usual then call <code>model.cuda()</code> to send all parameters of this model (and other stuff in the model) to the gpu. Then you need to make sure that your inputs are on the gpu as well <code>input = input.cuda()</code>. Then you can forward on the gpu by doing: <code>model(input)</code>.<br>
Note that <code>model.cuda()</code> will change the model inplace while <code>input.cuda()</code> will not change input inplace and you need to do <code>input = input.cuda()</code>.</p>

Shisho_Sama: <p>Thanks a lot, I believe I have done all of these, here is the bulk of my training snippet :</p>
<pre><code class="lang-auto">#moving model to cuda()
model.cuda()

#training core snippet
...
#activate training mode
model.train()

  end = time.time()
  for i, (input, target) in enumerate(train_loader):
    # measure data loading time
    data_time.update(time.time() - end)

    if args.use_cuda:
      target = target.cuda(async=True)
      input = input.cuda()
    input_var = torch.autograd.Variable(input)
    target_var = torch.autograd.Variable(target)

    # compute output
    output = model(input_var)
    loss = criterion(output, target_var) 
    
    # measure accuracy and record loss
    prec1, prec5 = accuracy(output.data, target, topk=(1, 5))
    losses.update(loss.item(), input.size(0))
    top1.update(prec1.item(), input.size(0)) 
    top5.update(prec5.item(), input.size(0)) 

    # compute gradient and do SGD step
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>

albanD: <p>Hi,</p>
<p>This looks fairly good, a few small things:</p>
<ul>
<li>
<code>async=True</code> for the target is most likely useless here (unless you specifically put it in pinned memory) so you can remote it</li>
<li>Variables don’t exist anymore, you can remove <code>input_var = torch.autograd.Variable(input)</code> and just use <code>input</code> where you have <code>input_var</code>. Same for the target.</li>
<li>Replace <code>output.data</code> with <code>output.detach()</code> that is the new way to do it. It works nicely with the autograd engine and will detect some forbidden behavior that the old <code>.data</code> was not able to do.</li>
</ul>

Shisho_Sama: <p>Thanks, I did accordingly, but that didnt fix the gpu utilization problem <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>

albanD: <p>Now if you have <code>args.use_cuda=True</code>, your model will use the GPU.<br>
Keep in mind that if your model or input is very small, you might not use the gpu very efficiently and thus gpu usage will remain small. You need to perform large enough ops for the gpu to be properly used. Increasing the batch size is an easy way to increase the amount of work for each op.</p>

Shisho_Sama: <p>Thanks a lot. that was it:) my model was way too tiny to begin with, even when I used batchsize of 512, the network was only 3~4 layers with like 10~30K parameters and it wouldnt need much GPU load to begin with <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=6" title=":blush:" class="emoji" alt=":blush:"><br>
Thanks a gazillion times <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Rohan_Sharma: <p>Could you please tell why using .cuda() on a parameter prevents it from participating in the optimization process? Removal of .cuda() from nn.Parameter() woks fine. I was facing the same issue.</p>

albanD: <p>Hi,</p>
<p>If your original Tensor is a leaf (a Tensor for which the .grad field is populated when you call .backward()), and you call <code>.cuda()</code> on it. Then the result is not a leaf because it was created in a differentiable manner. So you need to move it to cuda() before creating the nn.Parameter().</p>
<p>If your problem persist, please send a small code example that shows your issue.</p>

Rohan_Sharma: <p>Thank you albanD! That explained it.</p>