Time of profilering with cuda is almost close to time of cpu

yigu: <p>Hi,everyone! I’m a new coder using Pytorch.I have a question, why the time spent on cpu is equal to time spent on gpu? The time costs on cpu and time costs on cuda is very approximate,and I suspect the code run on cpu, but with tool of top and ‘watch -n 3 nvidia-smi’ in another terminal, I saw the change with the gpu when code run on gpu. Is there any method to show, the code really run on cuda? Thank you very much! My Pytorch version is 1.3.0, CUDA 9.0.<br>
code is:</p>
<pre><code class="lang-auto">import torch
input = torch.randn(1000, 3, 224, 224)
conv1 = torch.Conv2d(3, 5, (3, 3))
conv2 = torch.Conv2d(5, 7, (5, 5))

with torch.autograd.Profiler.profile(use_cuda=True) as prof:
    out = conv1(input)
    out = torch.relu(out)
    out = conv2(out)
    out = torch.relu(out)
    out = torch.relu(out)

print("prof is:\n{}".format(prof))
</code></pre>

yigu: <p>this is the result image, and left kernel name has been hidden by the size:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/faec3f28755d7b87fe2203f8ca0c13fcaef783c4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/faec3f28755d7b87fe2203f8ca0c13fcaef783c4" title="youdu图片20200515220850 (2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faec3f28755d7b87fe2203f8ca0c13fcaef783c4_2_690x399.jpeg" alt="youdu图片20200515220850 (2)" data-base62-sha1="zNLwtHEuFcOP38NKKj9R7Dpi51i" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faec3f28755d7b87fe2203f8ca0c13fcaef783c4_2_690x399.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faec3f28755d7b87fe2203f8ca0c13fcaef783c4_2_1035x598.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faec3f28755d7b87fe2203f8ca0c13fcaef783c4_2_1380x798.jpeg 2x" data-dominant-color="565254"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">youdu图片20200515220850 (2)</span><span class="informations">1976×1145 315 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If you are not transferring the model and data to the GPU, your code will run on the CPU by default.<br>
Use</p>
<pre><code class="lang-python">device = 'cuda:0'
input = torch.randn(1000, 3, 224, 224).to(device)
conv1 = torch.Conv2d(3, 5, (3, 3)).to(device)
conv2 = torch.Conv2d(5, 7, (5, 5)).to(device)
</code></pre>
<p>to push <code>input</code> as well as both layers to the GPU before executing the forward pass.</p>
<p><code>nvidia-smi</code> should not show any usage in your code snippet.<br>
What kind of change did you see using <code>nvidia-smi</code>?</p>

yigu: <p>Thanks for your reply.And I change the code with your addvision,and the time cost on cpu and cuda have reduced largely, but they are still neally equal.<br>
And the worse situation is ,the self time are wrong at same time.</p>

yigu: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b57071596a945c8afe92cbc99868512c30c56456.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b57071596a945c8afe92cbc99868512c30c56456" title="9_8"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b57071596a945c8afe92cbc99868512c30c56456_2_690x413.jpeg" alt="9_8" data-base62-sha1="pT5krh5VGypSqmvfs43sFinoKNg" width="690" height="413" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b57071596a945c8afe92cbc99868512c30c56456_2_690x413.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b57071596a945c8afe92cbc99868512c30c56456_2_1035x619.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b57071596a945c8afe92cbc99868512c30c56456_2_1380x826.jpeg 2x" data-dominant-color="514D51"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">9_8</span><span class="informations">1942×1164 237 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The Self CPU total time is the kernel cost time on self, such as conv2d, it use convolution and _convolution and cudnn_convolution as actual running kernel.The time cost on cudnn_convolution cost time is 17.935ms, and _convolution’s Self CPU total is (17.998 - 17.935=0.063ms),its parent convolution cost (18.021 - 17.998 = 0.023ms), and conv2d cost (18.041 - 18.021 = 0.02ms).And CPU total are right.<br>
So I am wondering, are there something running wrong?</p>

yigu: <p>I watch the running situation use command ‘watch -n 1 nvidia-smi’，but I found that, the first card run normal, are when it run over, the second card started to run, so did 3th and 4th.But I don’t set four threads or processes. Why this happens? Does profile with cuda run all gpu cards to profiler, and only show one card’s profile result?<br>
I am confused, thanks a lot if you can give me addvisions or give me some useful links.<br>
May a good day!</p>