CUDA Tensors vs Pytorch Tensor?

username: miner_tom
cooked: <p>I am new to pytorch but I do program some in CUDA (enough to be dangerous).</p>
<p>My understanding is that when, say,  a numpy array is passed as a tensor, then it is in GPU memory. It can then be manipulated by the GPU and finally transferred back to the CPU memory.</p>
<p>Then, there are CUDA tensors. CUDA also operates on GPU memory.</p>
<ol>
<li>What is the difference?</li>
<li>Can GPU profiling be done in pytorch as it can be done in CUDA with the nvidia profiler?</li>
</ol>
<p>Thank You<br>
Tom</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="miner_tom" data-post="1" data-topic="102346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miner_tom/48/30088_2.png" class="avatar"> miner_tom:</div>
<blockquote>
<p>My understanding is that when, say, a numpy array is passed as a tensor, then it is in GPU memory.</p>
</blockquote>
</aside>
<p>No, numpy arrays will share the same underlying data, if <code>tensor = torch.from_numpy(array)</code> is used (no copy triggered) and can be pushed to the GPU via <code>tensor = tensor.to('cuda')</code>, where operations on this tensor will be executed on the GPU.</p>
<aside class="quote no-group" data-username="miner_tom" data-post="1" data-topic="102346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miner_tom/48/30088_2.png" class="avatar"> miner_tom:</div>
<blockquote>
<p>Then, there are CUDA tensors.</p>
</blockquote>
</aside>
<p>I’m not sure what <code>CUDA tensors</code> are. If you are referring to PyTorch tensors on the GPU, then note that each tensor has a device associated with it and can be stored on different devices, such as the CPU, GPU, TPU, etc.</p>
<aside class="quote no-group" data-username="miner_tom" data-post="1" data-topic="102346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miner_tom/48/30088_2.png" class="avatar"> miner_tom:</div>
<blockquote>
<p>Can GPU profiling be done in pytorch as it can be done in CUDA with the nvidia profiler?</p>
</blockquote>
</aside>
<p>Yes, you can use <a href="https://developer.nvidia.com/nsight-systems">Nsight systems/compute</a>, <a href="https://github.com/NVIDIA/PyProf">pyprof</a>, <a href="https://pytorch.org/tutorials/recipes/recipes/profiler.html">the autograd.profiler</a> etc.</p>

username: miner_tom
cooked: <p>Thank you, most earnestly, for your reply.</p>
<p>I would only request one more bit of clarification, if I may. I can load a .cu file into Nsight Systems Compute (although I had to replace my Graphics card from Pascal architecture to Turing architecture, ouch). But can Nsight read a python script (.py) and interpret the pytorch cuda references?</p>
<p>Thanks Again<br>
Tom</p>

username: ptrblck
cooked: <p>I’m not familiar with reading source files directly in Nsight and am profiling the code first via the CLI.</p>

username: miner_tom
cooked: <p>Thank you again, but are you saying that Nsight has a Command line Interface akin to the Nvidia “nvprof”?</p>
<p>Regards<br>
Tom</p>

username: ptrblck
cooked: <p>Yes, you can fine the CLI options <a href="https://docs.nvidia.com/nsight-systems/UserGuide/index.html#cli-options">here</a>. The most basic would be <code>nsys profile python script.py</code> and you could add more options e.g. to create backtraces etc.</p>

username: GabbyChan
cooked: <p>Hi, I want to get the correct runtime   …</p>
<p>Do I need to set CUDA_LAUNCH_BLOCKING=1 when using nsys profile?</p>
<p>For example, do you need to do the following ? …</p>
<pre><code class="lang-auto">export CUDA_LAUNCH_BLOCKING=1
nsys profile python script.py
</code></pre>

username: ptrblck
cooked: <p>No, you can profile the application without blocking launches.</p>

username: GabbyChan
cooked: <p>I see. Thank you. By the way, as I have run <strong><em>without</em> blocking launches</strong>, I have got this big cudaSteamSychronize in green …</p>
<ul>
<li>
<p>Is that just normally how Pytorch would work?</p>
</li>
<li>
<p>Does the runtime from the profiler representing the GPU and CPU runtime …?</p>
</li>
</ul>
<p>I have actually run it with</p>
<p><code>nsys profile -t cuda,osrt,nvtx,cudnn,cublas -o baseline_1024_sup_photos_async -w true  -f true  python3 pythonfile.py </code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41257f49e4e7c6a985c944dfb14e61c3d010fd47.png" data-download-href="https://discuss.pytorch.org/uploads/default/41257f49e4e7c6a985c944dfb14e61c3d010fd47" title="Screenshot from 2021-02-05 18-31-50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41257f49e4e7c6a985c944dfb14e61c3d010fd47_2_690x388.png" alt="Screenshot from 2021-02-05 18-31-50" data-base62-sha1="9ijn7PBlIVT5NAtgiNRMnc4qQMD" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41257f49e4e7c6a985c944dfb14e61c3d010fd47_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41257f49e4e7c6a985c944dfb14e61c3d010fd47_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41257f49e4e7c6a985c944dfb14e61c3d010fd47_2_1380x776.png 2x" data-dominant-color="CBD6DB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-05 18-31-50</span><span class="informations">1920×1080 266 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="GabbyChan" data-post="9" data-topic="102346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gabbychan/48/18041_2.png" class="avatar"> GabbyChan:</div>
<blockquote>
<p>Is that just normally how Pytorch would work?</p>
</blockquote>
</aside>
<p>It depends on the operations you are using. If the current method synchronizes the code by e.g. pushing a value to the CPU, then you could see synchronizations in the timeline.<br>
The majority of ops in PyTorch do not synchronize besides a few exceptions (e.g. some methods in the <code>torch.linalg</code> namespace check the output info to give you an error in case the linalg operation failed. This should be relaxed in upcoming PyTorch releases).</p>
<aside class="quote no-group" data-username="GabbyChan" data-post="9" data-topic="102346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gabbychan/48/18041_2.png" class="avatar"> GabbyChan:</div>
<blockquote>
<p>Does the runtime from the profiler representing the GPU and CPU runtime …?</p>
</blockquote>
</aside>
<p>Yes, you would see the runtimes in the profile. Note that the profiling itself might add overhead.</p>

username: GabbyChan
cooked: <p>Thank you …<br>
I have just found that the delay was caused by converting my float tensors into double precision at the beginning of my script e.g. <code> tensor_A.double()</code> …  not sure why … But I decided to remove the conversion …</p>

username: GabbyChan
cooked: <p>Hi!  I would like to ask more questions about Nsight profiling vs <code>using torch.cuda.Event</code>…</p>
<p>The problem that I have is that the runtime of Nsight profiling is much faster (about 14 ms) than <code>using torch.cuda.Event</code> (about 74 ms)…</p>
<p><strong>My questions are</strong></p>
<ol>
<li>
<p>What are the differences between the two? Is it because the Nsight profiling is running on Asynchronization mode, but the <code>using torch.cuda.Event</code> is running on Synchronization mode.</p>
</li>
<li>
<p>Which of these two results should I use as the reference for the runtime performance of my network?</p>
</li>
</ol>
<p><em>Details on how I ran Nsight Profiler and <code>torch.cuda.event</code> are as follows</em>:</p>
<ul>
<li><strong>Nsight Profiler.</strong> I have used the Nsight Profiler with command line</li>
</ul>
<pre><code class="lang-auto">nsys profile -t cuda,osrt,nvtx,cudnn,cublas -o profile_inference_runtime -w true  -f true python3 demo_inference_runtime.py
</code></pre>
<p>which gives the following runtime results. Notice that the (below)  light blue range measured at CUDA is <strong>much longer</strong> than the (above) runtime measured at NVTX (also denoted as <code>Net @ [iter 57] 14.693 ms</code>):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a93578f83047392f45e94406b891ffa690d400d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/a93578f83047392f45e94406b891ffa690d400d0" title="Screenshot from 2021-05-02 22-10-59"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a93578f83047392f45e94406b891ffa690d400d0_2_690x388.png" alt="Screenshot from 2021-05-02 22-10-59" data-base62-sha1="o8TfVtDUCpxCXy3atuxnWikvies" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a93578f83047392f45e94406b891ffa690d400d0_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a93578f83047392f45e94406b891ffa690d400d0_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a93578f83047392f45e94406b891ffa690d400d0_2_1380x776.png 2x" data-dominant-color="C3D1D0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-05-02 22-10-59</span><span class="informations">1920×1080 274 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ul>
<li><strong><code>Torch.cuda.event</code></strong>, I measured the runtime with <code>torch.cuda.Event</code> as follows.  :</li>
</ul>
<pre><code class="lang-auto">torch.cuda.synchronize()
start_net_event = torch.cuda.Event(enable_timing=True)
end_net_event = torch.cuda.Event(enable_timing=True)
start_net_event.record()

out_t = net(pred_data) 

end_net_event.record()
torch.cuda.synchronize()   
net_runtime_temp = start_net_event.elapsed_time(end_net_event) 
</code></pre>
<p>which gives the following runtime results.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e6b4ebce92b9428024c8a0e16e06a02f2628636.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e6b4ebce92b9428024c8a0e16e06a02f2628636" title="Screenshot from 2021-05-02 22-20-24"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e6b4ebce92b9428024c8a0e16e06a02f2628636.png" alt="Screenshot from 2021-05-02 22-20-24" data-base62-sha1="dtgMPEskDUU279YOK155x689L5c" width="690" height="100" data-dominant-color="3A3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-05-02 22-20-24</span><span class="informations">1318×192 38.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I assume you are checking the <code>NVTX</code> timeline and compare it to the kernel times in the <code>CUDA GeForce...</code> timeline. If so, it seems that the kernel launches are shown in the first one and you see that the CPU seems to run ahead with the data loading (assuming “Import data…” is used in the dataset) while the GPU is executing the workload.</p>

username: GabbyChan
cooked: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="13" data-topic="102346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>…the CPU seems to run ahead with the data loading (assuming “Import data…” is used in the dataset) while the GPU is executing the workload…</p>
</blockquote>
</aside>
<p>Yes, the CPU part is in my dataset class (torch.utils.data.Dataset) … That is in <code>def __get_item__(self, idx)</code>, I have to download numpy data from .h5 file (I used it for the smaller file size)  and convert them into pytorch tensor and cuda device.</p>
<p>So, could you further confirm how should I decide which part is the runtime of my network (inference)…? I have marked it as <code>Net @ [iter 57] 14.693 ms</code> with NVTX…</p>
<p>However, the reason for me to be quite confused is that …in the previously attached Nsight profiler, the (below) light blue range measured at CUDA is <strong>much longer</strong> than the (above) runtime measured at NVTX (<code>Net @ [iter 57] 14.693 ms</code> ).  So, was it <code>14.693 ms</code>? (But when I measured with <code>Torch.cuda.event</code>, the runtime was much longer (73ms)…? )</p>
<p>Or the longer range at CUDA than NVTX in the Nsight Profiler does not mean that it takes longer? Did I miss measuring anything or doing something wrong?</p>
<p>I am so sorry. I am quite new in understanding the Profiler.  Please help.</p>