[SOLVED] Make Sure That Pytorch Using GPU To Compute

herleeyandi: <p>Hello I am new in pytorch. Now I am trying to run my network in GPU. Some of the articles recommend me to use <code>torch.cuda.set_device(0)</code> as long as my GPU ID is 0. However some articles also tell me to convert all of the computation to Cuda, so every operation should be followed by <code>.cuda()</code> . My questions are:<br>
-) Is there any simple way to set mode of pytorch to GPU, without using <code>.cuda()</code> per instruction?, I just want to set all computation just in 1 GPU.<br>
-) How to check and make sure that our network is running on GPU?, when I am using <code>torch.cuda.set_device(0)</code>, I checked using <code>nvidia-smi</code> and I got 0% in volatile GPU. It is different when I am using Tensorflow or caffe which more than 10%. I am affraid that my pytorch still using CPU.<br>
-Thank you-</p>

hughperkins: <p>generally speaking, the pattern is:</p>
<ul>
<li>
<p>use <code>.cuda()</code> on any input batches/tensors</p>
</li>
<li>
<p>use <code>.cuda()</code> on your network module, which will hold your network, like:</p>
<p>class MyModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
self.layer1 = nn. …<br>
self.layer2 = nn. …<br>
… etc …</p>
</li>
</ul>
<p>then just do:</p>
<pre><code>model = MyModel()
model.cuda()</code></pre>

qbx2: <p>How about using <code>torch.set_default_tensor_type('torch.cuda.FloatTensor')</code>?</p>

pinocchio: <p>From the <a href="http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#autograd" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#autograd</a></p>
<p>tutorial it seems that the way they do to make sure everything is in cuda is to have a dytype for GPUs as in:</p>
<pre><code>dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU
</code></pre>
<p>and they have lines like:</p>
<pre><code># Randomly initialize weights
w1 = torch.randn(D_in, H).type(dtype)
w2 = torch.randn(H, D_out).type(dtype)
</code></pre>
<p>that way its seems possible to me that one can avoid the silly <code>.cuda</code> line everywhere in your code. Right? Im also new so Im checking with others.</p>

herleeyandi: <p>Thanks everyone, your solutions are working well in my case. One reason which makes me really like pytorch is because of discussion forum. It help me a lot!</p>

Ismail_Elezi: <p>In addition to what have been discussed so far, I found that adding this line of code:</p>
<pre><code>cudnn.benchmark = True
</code></pre>
<p>before the training takes place, will improve the speed performance if you are using GPU(s).</p>

Ismail_Elezi: <p><a class="mention" href="/u/hughperkins">@hughperkins</a></p>
<p>Do you know what is the case and is there ever a disadvantage on putting the flag on? And if there are no disadvantages, then why it isn’t the default flag on?</p>

hughperkins: <p>This question sounds familiar somehow <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> . <a href="https://groups.google.com/forum/#!topic/torch7/CkB57025yRY" class="inline-onebox" rel="noopener nofollow ugc">Redirecting to Google Groups</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e37940d048fd39f2f3110f7f70bc7509d84a5ee8.png" data-download-href="https://discuss.pytorch.org/uploads/default/e37940d048fd39f2f3110f7f70bc7509d84a5ee8" title="Screen Shot 2017-07-18 at 7.38.28 AM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e37940d048fd39f2f3110f7f70bc7509d84a5ee8_2_690x37.png" width="690" height="37" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e37940d048fd39f2f3110f7f70bc7509d84a5ee8_2_690x37.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e37940d048fd39f2f3110f7f70bc7509d84a5ee8_2_1035x55.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e37940d048fd39f2f3110f7f70bc7509d84a5ee8_2_1380x74.png 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-18 at 7.38.28 AM.png</span><span class="informations">3318×178 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

hughperkins: <p>Also note <code>.fastest</code>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/ce15edcc9bbedaa62ac854645e488134dbb69edb.png" data-download-href="https://discuss.pytorch.org/uploads/default/ce15edcc9bbedaa62ac854645e488134dbb69edb" title="Screen Shot 2017-07-18 at 7.39.43 AM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/ce15edcc9bbedaa62ac854645e488134dbb69edb_2_690x47.png" width="690" height="47" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/ce15edcc9bbedaa62ac854645e488134dbb69edb_2_690x47.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/ce15edcc9bbedaa62ac854645e488134dbb69edb_2_1035x70.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/c/ce15edcc9bbedaa62ac854645e488134dbb69edb_2_1380x94.png 2x" data-dominant-color="F8F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-18 at 7.39.43 AM.png</span><span class="informations">3302×226 36.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Royi: <p>Is there a simple function which tests the GPU is configured correctly?<br>
As I did what <a class="mention" href="/u/hughperkins">@hughperkins</a> suggested on the following MNIST example:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/rickiepark/pytorch-examples/blob/master/mnist.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/rickiepark/pytorch-examples/blob/master/mnist.ipynb" target="_blank" rel="nofollow noopener">rickiepark/pytorch-examples/blob/master/mnist.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch MNIST Example similar to TensorFlow Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer to https://www.tensorflow.org/versions/master/tutorials/mnist/pros/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
</code></pre>

  This file has been truncated. <a href="https://github.com/rickiepark/pytorch-examples/blob/master/mnist.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Yet my system won’t run (Calculation will stop with error).</p>
<p>Thank You.</p>

hughperkins: <p><a class="mention" href="/u/royi">@Royi</a>  I usually do the following, from bash:</p>
<pre><code>nvidia-smi
python -c 'import torch; print(torch.rand(2,3).cuda())'
</code></pre>
<p>If the first fails, your drivers have some issue, or you dont have an (NVIDIA) GPU <img src="//discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=5" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>If the second fails, your pytorch instalaltion isnt able to contact the gpu for some reason (eg you didnt do <code>conda install cuda80 -c soumith</code> etc…)</p>
<p>(edit: if both the above succeed; I never saw any configuration error beyond that, other than my own coding error <img src="//discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=5" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:">  BUT if you try to run on a V100, using cuda 8 pytorch, the second statement will hang for ~5 minutes, whilst it creates the cache. But it’ll do this each time, so it’s useless, and you’ll need to use cuda 9 pytorch, (or not use a V100))</p>

Brando_Miranda: <p>try:</p>
<p><a href="http://pytorch.org/docs/master/notes/cuda.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/notes/cuda.html</a></p>

xiaoiker: <p>Hi when I try this codes, the second failed with the info. :Segmentation fault (core dumped). But when I add CUDA_VISIBLE_DEVICES=1 it works. only when I using CUDA_VISIBLE_DEVICES=0 it is failed. Can you PLS tell me why and give any suggestion.</p>

spacemeerkat: <p>I’m trying to implement the methods at the beginning of this thread as follows:</p>
<pre><code class="lang-auto">model = model.cuda()

torch.backends.cudnn.benchmark=True

import time
start = time.time()
model.train()
train_loss = []
train_accu = []
i = 0
for epoch in range(20):
    for data, target in train_loader:
        data, target = (Variable(data).double()).cuda(), (Variable(target).long()).cuda()
        optimizer.zero_grad()
        output = model(data.view(batch_size,1,64,64))
        loss = F.nll_loss(output, target) # Negative log likelihood (goes with softmax). 
        loss.backward()    # calc gradients
        train_loss.append(loss.data[0]) # Calculating the loss
        optimizer.step()   # update gradients
        prediction = output.data.max(1)[1]   # first column has actual prob.
        accuracy = (prediction.eq(target.data).sum()/batch_size)*100
        train_accu.append(accuracy)
        if i % 10 == 0:
            print('Epoch:',str(epoch),'Train Step: {}\tLoss: {:.3f}\tAccuracy: {:.3f}'.format(i, loss.data[0], accuracy))
        i += 1
end = time.time()
print('TRAIN TIME:')
print('%.2gs'%(end-start))
</code></pre>
<p>But when I train, I just get a constant accuracy of 0%. Am I missing some part where I need to cast to .cuda() ?</p>

justusschock: <p><code>prediction.eq(target.data)</code>  returns a byte tensor/variable. Summing it up and dividing it by a batchsize would lead to zero.</p>
<p>Try it with</p>
<pre><code class="lang-auto">accuracy = (prediction.eq(target.data).float().sum()/batch_size)*100
</code></pre>

spacemeerkat: <p>Ah yes it would wouldn’t it! Worked beautifully, thanks!</p>

spacemeerkat: <p>Would it matter that I’ve called .cuda() on data before turning it into a variable or should i be doing Variable(data).double().cuda() ?</p>

justusschock: <p>This should both work equally good.<br>
I would recommend switching to pytorch 0.4 as both classes are merged in this release</p>

Shariq_Ali: <p>hello dear i have the same issue. i don’t know how to solve it. could you help me please.</p>

Rui_Zhang: <p>Hi,</p>
<p>I am struggling with running Pytorch on GPU. I created a simple fully connected network, set batch_size very large to make sure all data will be fed for the first time, and put my model, X and y to GPU using <code>to('cuda')</code>. The training takes long time comparing to Keras on GPU, and takes similar time to that if I set <code>os.environ["CUDA_VISIBLE_DEVICES"]="-1"</code> such that training will be run on CPU. I wonder if I miss any import step to run Pytorch on GPU.</p>
<p>In fact I observed timing difference for a CNN network - GPU runs faster than CPU. However, I cannot manage to realise it for a fully connected network. The size of the network won’t change the conclusion.</p>
<p>Is there any test code for a fully connected deep network running on GPU? All examples on the web that I can find are CNNs.</p>