I have just installed PyTorch with cuda, but model forward is slow

Eleonor: <p>As in title, I have just installed PyTorch 2.4.1 with cuda 12.4 and also performed the verification steps <a href="https://pytorch.org/get-started/locally/#windows-verification" rel="noopener nofollow ugc">here</a>.</p>
<p>However when I try to run a script for training a relatively simple model (some convolution layers) on a custom dataset, the training speed of  is very slow. I am sure it is not a dataset reading problem and, of course, I’m moving my model instance to cuda device.</p>
<p>So I was wondering if I am missing any component in the installation process (like missing dependencies) or if the slowness is caused by something else (like libraries mismatches).<br>
In short, how do I now if my installation is complete?</p>

ptrblck: <aside class="quote no-group" data-username="Eleonor" data-post="1" data-topic="211863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/a698b9/48.png" class="avatar"> Eleonor:</div>
<blockquote>
<p>I am sure it is not a dataset reading problem and, of course, I’m moving my model instance to cuda device.</p>
</blockquote>
</aside>
<p>I assume you’ve profiled the code section with the native PyTorch profiler or e.g. Nsight Systems? If so, could you post these profiles here showing the actual model execution is slow?</p>

Eleonor: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I hope this is what you asked me. This refers to a single epoch iteration without the loss backward step.</p>
<pre><code class="lang-auto">Running environment analysis...
Running your script with cProfile
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1517/1517 [09:14&lt;00:00,  2.73it/s]
Running your script with the autograd profiler...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1517/1517 [09:13&lt;00:00,  2.74it/s]
C:\Users\&lt;name&gt;\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\bottleneck\__main__.py:96: FutureWarning: The attribute `use_cuda` will be deprecated soon, please use ``use_device = 'cuda'`` instead.
  with profiler.profile(use_cuda=use_cuda) as prof:
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1517/1517 [09:31&lt;00:00,  2.66it/s]
--------------------------------------------------------------------------------
  Environment Summary
--------------------------------------------------------------------------------
PyTorch 2.4.1+cu124 DEBUG compiled w/ CUDA 12.4
Running with Python 3.11 and CUDA 12.4.131

`pip3 list` truncated output:
numpy==2.1.2
torch==2.4.1+cu124
torch_cluster==1.6.3+pt24cu124
torch-geometric==2.6.1
torchaudio==2.4.1+cu124
torchvision==0.19.1+cu124
--------------------------------------------------------------------------------
  cProfile output
--------------------------------------------------------------------------------
         3269346 function calls (3034665 primitive calls) in 556.497 seconds

   Ordered by: internal time
   List reduced from 917 to 15 due to restriction &lt;15&gt;

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     4644  534.605    0.115  534.605    0.115 {method 'to' of 'torch._C.TensorBase' objects}
       16   10.414    0.651   10.415    0.651 {method 'dump' of '_pickle.Pickler' objects}
       18    1.175    0.065    1.176    0.065 C:\Users\&lt;name&gt;\AppData\Local\Programs\Python\Python311\Lib\site-packages\h5py\_hl\dataset.py:765(__getitem__)
    22755    0.956    0.000    0.956    0.000 {built-in method torch.batch_norm}
     6068    0.721    0.000    0.721    0.000 {method 'topk' of 'torch._C.TensorBase' objects}
    10619    0.535    0.000    0.535    0.000 {built-in method torch.conv2d}
     6068    0.491    0.000    1.053    0.000 C:\Magistrale\Visione Artificiale\va-project\models\utils.py:45(graph_features)
        1    0.430    0.430    0.430    0.430 {built-in method _winapi.WaitForMultipleObjects}
     6068    0.421    0.000    0.421    0.000 {built-in method torch.matmul}
     6068    0.413    0.000    1.871    0.000 C:\Magistrale\Visione Artificiale\va-project\models\utils.py:4(knn_graph)
    22755    0.366    0.000    0.366    0.000 {built-in method torch._C._nn.leaky_relu}
    10619    0.365    0.000    0.365    0.000 {built-in method torch.conv1d}
       36    0.315    0.009    0.315    0.009 {built-in method _winapi.WaitForSingleObject}
        1    0.298    0.298  556.497  556.497 .\test.py:1(&lt;module&gt;)
    22755    0.292    0.000    0.292    0.000 {method 'add_' of 'torch._C.TensorBase' objects}


--------------------------------------------------------------------------------
  autograd profiler output (CPU mode)
--------------------------------------------------------------------------------
        top 15 events sorted by cpu_time_total

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
                                               aten::to         0.00%       3.800us        27.50%     557.672ms     557.672ms             1
                                         aten::_to_copy         0.00%      10.300us        27.50%     557.668ms     557.668ms             1
                                            aten::copy_        27.50%     557.654ms        27.50%     557.654ms     557.654ms             1
enumerate(DataLoader)#_MultiProcessingDataLoaderIter...        20.01%     405.891ms        20.01%     405.929ms     405.929ms             1
                                               aten::to         0.00%       4.200us        17.50%     354.898ms     354.898ms             1
                                         aten::_to_copy         0.00%       9.300us        17.50%     354.894ms     354.894ms             1
                                            aten::copy_        17.50%     354.876ms        17.50%     354.876ms     354.876ms             1
                                               aten::to         0.00%       6.500us        17.49%     354.786ms     354.786ms             1
                                               aten::to         0.00%       3.700us        17.49%     354.783ms     354.783ms             1
                                         aten::_to_copy         0.01%     194.900us        17.49%     354.780ms     354.780ms             1
                                         aten::_to_copy         0.00%       9.300us        17.49%     354.779ms     354.779ms             1
                                               aten::to         0.00%       4.500us        17.49%     354.772ms     354.772ms             1
                                         aten::_to_copy         0.00%       8.800us        17.49%     354.767ms     354.767ms             1
                                            aten::copy_        17.49%     354.766ms        17.49%     354.766ms     354.766ms             1
                                            aten::copy_        17.49%     354.754ms        17.49%     354.754ms     354.754ms             1
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 2.028s

--------------------------------------------------------------------------------
  autograd profiler output (CUDA mode)
--------------------------------------------------------------------------------
        top 15 events sorted by cpu_time_total

        Because the autograd profiler uses the CUDA event API,
        the CUDA time column reports approximately max(cuda_time, cpu_time).
        Please ignore this output if your code does not use CUDA.

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
enumerate(DataLoader)#_MultiProcessingDataLoaderIter...        23.03%     373.121ms        23.03%     373.147ms     373.147ms     370.957ms        72.89%     370.972ms     370.972ms             1
enumerate(DataLoader)#_MultiProcessingDataLoaderIter...        22.09%     357.819ms        22.09%     357.819ms     357.819ms     137.395ms        27.00%     137.395ms     137.395ms             1
                                               aten::to         0.00%      11.500us        13.75%     222.814ms     222.814ms       7.000us         0.00%     110.000us     110.000us             1
                                         aten::_to_copy         0.00%      47.500us        13.75%     222.803ms     222.803ms      24.000us         0.00%     103.000us     103.000us             1
                                            aten::copy_        13.75%     222.739ms        13.75%     222.739ms     222.739ms      78.000us         0.02%      78.000us      78.000us             1
                                               aten::to         0.00%      14.000us        13.74%     222.621ms     222.621ms      15.000us         0.00%     143.000us     143.000us             1
                                         aten::_to_copy         0.00%      58.400us        13.74%     222.607ms     222.607ms      15.000us         0.00%     128.000us     128.000us             1
                                            aten::copy_        13.74%     222.538ms        13.74%     222.538ms     222.538ms      94.000us         0.02%      94.000us      94.000us             1
                                               aten::to         0.00%      14.500us        13.70%     221.930ms     221.930ms      10.000us         0.00%     119.000us     119.000us             1
                                         aten::_to_copy         0.00%      53.900us        13.70%     221.916ms     221.916ms      33.000us         0.01%     109.000us     109.000us             1
                                               aten::to         0.00%      49.700us        13.70%     221.883ms     221.883ms       9.000us         0.00%     239.000us     239.000us             1
                                            aten::copy_        13.69%     221.848ms        13.69%     221.848ms     221.848ms      75.000us         0.01%      75.000us      75.000us             1
                                         aten::_to_copy         0.00%      75.100us        13.69%     221.833ms     221.833ms     112.000us         0.02%     230.000us     230.000us             1
                                            aten::copy_        13.69%     221.735ms        13.69%     221.735ms     221.735ms     113.000us         0.02%     113.000us     113.000us             1
                                               aten::to         0.00%      14.300us        13.66%     221.339ms     221.339ms       4.000us         0.00%     116.000us     116.000us             1
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 1.620s
Self CUDA time total: 508.941ms
</code></pre>

Eleonor: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
If it helps I have also collected environment information with <code>python -m torch.utils.collect_env</code>:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 2.4.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22631-SP0
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4060 Ti
Nvidia driver version: 551.78
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3400
DeviceID=CPU0
Family=198
L2CacheSize=24576
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=3400
Name=13th Gen Intel(R) Core(TM) i7-13700K
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] torch==2.4.1+cu124
[pip3] torchaudio==2.4.1+cu124
[pip3] torchvision==0.19.1+cu124
[conda] Could not collect
</code></pre>

jinliang_li: <p>You can profile and dump it into json, and check it in html.<br>
Detail is in <code>https://github.com/pytorch/kineto/tree/main/tb_plugin</code></p>

ptrblck: <p>Your profile shows the largest time is spent in the <code>to()</code> operation, which could point to the data loading pipeline, so I’m still unsure why you don’t think it’s a data loading issue.</p>