Pytroch on GPU training but still on cpu

username: Maths_Electronics_Tu
cooked: <p>cuda installed , when i check if gpu is detected, it is true. I setup training in pytorch, mmaction2 , the training is still on the CPU.</p>
<h2><a name="using-gpu-quadro-rtx-5000-1025-122425-mmengine-info-1" class="anchor" href="#using-gpu-quadro-rtx-5000-1025-122425-mmengine-info-1"></a>Using GPU: Quadro RTX 5000<br>
10/25 12:24:25 - mmengine - INFO -</h2>
<p>System environment:<br>
sys.platform: win32<br>
Python: 3.8.18 (default, Sep 11 2023, 13:39:12) [MSC v.1916 64 bit (AMD64)]<br>
CUDA available: True<br>
numpy_random_seed: 1335494315<br>
GPU 0: Quadro RTX 5000<br>
CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3<br>
NVCC: Cuda compilation tools, release 11.3, V11.3.109<br>
MSVC: n/a, reason: fileno<br>
PyTorch: 1.10.0+cu102<br>
PyTorch compiling details: PyTorch built with:</p>
<ul>
<li>
<p>C++ Version: 199711</p>
</li>
<li>
<p>MSVC 192829337</p>
</li>
<li>
<p>Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications</p>
</li>
<li>
<p>Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)</p>
</li>
<li>
<p>OpenMP 2019</p>
</li>
<li>
<p>LAPACK is enabled (usually provided by MKL)</p>
</li>
<li>
<p>CPU capability usage: AVX512</p>
</li>
<li>
<p>CUDA Runtime 10.2</p>
</li>
<li>
<p>NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37</p>
</li>
<li>
<p>CuDNN 7.6.5</p>
</li>
<li>
<p>Magma 2.5.4</p>
</li>
<li>
<p>Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=C:/w/b/windows/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/w/b/windows/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON,</p>
<p>TorchVision: 0.11.1+cu102<br>
OpenCV: 4.5.1<br>
MMEngine: 0.9.0</p>
</li>
</ul>
<h2><a name="runtime-environment-dist_cfg-backend-nccl-seed-1335494315-distributed-launcher-none-distributed-training-false-gpu-number-1-2" class="anchor" href="#runtime-environment-dist_cfg-backend-nccl-seed-1335494315-distributed-launcher-none-distributed-training-false-gpu-number-1-2"></a>Runtime environment:<br>
dist_cfg: {‘backend’: ‘nccl’}<br>
seed: 1335494315<br>
Distributed launcher: none<br>
Distributed training: False<br>
GPU number: 1</h2>
<h2><a name="h-1025-122426-mmengine-info-distributed-training-is-not-used-all-syncbatchnorm-syncbn-layers-in-the-model-will-be-automatically-reverted-to-batchnormxd-layers-if-they-are-used-1025-122426-mmengine-info-hooks-will-be-executed-in-the-following-order-before_run-very_high-runtimeinfohook-below_normal-loggerhook-3" class="anchor" href="#h-1025-122426-mmengine-info-distributed-training-is-not-used-all-syncbatchnorm-syncbn-layers-in-the-model-will-be-automatically-reverted-to-batchnormxd-layers-if-they-are-used-1025-122426-mmengine-info-hooks-will-be-executed-in-the-following-order-before_run-very_high-runtimeinfohook-below_normal-loggerhook-3"></a>10/25 12:24:26 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.<br>
10/25 12:24:26 - mmengine - INFO - Hooks will be executed in the following order:<br>
before_run:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(BELOW_NORMAL) LoggerHook</h2>
<h2><a name="before_train-very_high-runtimeinfohook-normal-itertimerhook-very_low-checkpointhook-4" class="anchor" href="#before_train-very_high-runtimeinfohook-normal-itertimerhook-very_low-checkpointhook-4"></a>before_train:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(NORMAL      ) IterTimerHook<br>
(VERY_LOW    ) CheckpointHook</h2>
<h2><a name="before_train_epoch-very_high-runtimeinfohook-normal-itertimerhook-normal-distsamplerseedhook-5" class="anchor" href="#before_train_epoch-very_high-runtimeinfohook-normal-itertimerhook-normal-distsamplerseedhook-5"></a>before_train_epoch:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(NORMAL      ) IterTimerHook<br>
(NORMAL      ) DistSamplerSeedHook</h2>
<h2><a name="before_train_iter-very_high-runtimeinfohook-normal-itertimerhook-6" class="anchor" href="#before_train_iter-very_high-runtimeinfohook-normal-itertimerhook-6"></a>before_train_iter:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(NORMAL      ) IterTimerHook</h2>
<h2><a name="after_train_iter-very_high-runtimeinfohook-normal-itertimerhook-below_normal-loggerhook-low-paramschedulerhook-very_low-checkpointhook-7" class="anchor" href="#after_train_iter-very_high-runtimeinfohook-normal-itertimerhook-below_normal-loggerhook-low-paramschedulerhook-very_low-checkpointhook-7"></a>after_train_iter:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(NORMAL      ) IterTimerHook<br>
(BELOW_NORMAL) LoggerHook<br>
(LOW         ) ParamSchedulerHook<br>
(VERY_LOW    ) CheckpointHook</h2>
<h2><a name="after_train_epoch-normal-itertimerhook-low-paramschedulerhook-very_low-checkpointhook-8" class="anchor" href="#after_train_epoch-normal-itertimerhook-low-paramschedulerhook-very_low-checkpointhook-8"></a>after_train_epoch:<br>
(NORMAL      ) IterTimerHook<br>
(LOW         ) ParamSchedulerHook<br>
(VERY_LOW    ) CheckpointHook</h2>
<h2><a name="before_val-very_high-runtimeinfohook-9" class="anchor" href="#before_val-very_high-runtimeinfohook-9"></a>before_val:<br>
(VERY_HIGH   ) RuntimeInfoHook</h2>
<h2><a name="before_val_epoch-normal-itertimerhook-10" class="anchor" href="#before_val_epoch-normal-itertimerhook-10"></a>before_val_epoch:<br>
(NORMAL      ) IterTimerHook</h2>
<h2><a name="before_val_iter-normal-itertimerhook-11" class="anchor" href="#before_val_iter-normal-itertimerhook-11"></a>before_val_iter:<br>
(NORMAL      ) IterTimerHook</h2>
<h2><a name="after_val_iter-normal-itertimerhook-below_normal-loggerhook-12" class="anchor" href="#after_val_iter-normal-itertimerhook-below_normal-loggerhook-12"></a>after_val_iter:<br>
(NORMAL      ) IterTimerHook<br>
(BELOW_NORMAL) LoggerHook</h2>
<h2><a name="after_val_epoch-very_high-runtimeinfohook-normal-itertimerhook-below_normal-loggerhook-low-paramschedulerhook-very_low-checkpointhook-13" class="anchor" href="#after_val_epoch-very_high-runtimeinfohook-normal-itertimerhook-below_normal-loggerhook-low-paramschedulerhook-very_low-checkpointhook-13"></a>after_val_epoch:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(NORMAL      ) IterTimerHook<br>
(BELOW_NORMAL) LoggerHook<br>
(LOW         ) ParamSchedulerHook<br>
(VERY_LOW    ) CheckpointHook</h2>
<h2><a name="after_val-very_high-runtimeinfohook-14" class="anchor" href="#after_val-very_high-runtimeinfohook-14"></a>after_val:<br>
(VERY_HIGH   ) RuntimeInfoHook</h2>
<h2><a name="after_train-very_high-runtimeinfohook-very_low-checkpointhook-15" class="anchor" href="#after_train-very_high-runtimeinfohook-very_low-checkpointhook-15"></a>after_train:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(VERY_LOW    ) CheckpointHook</h2>
<h2><a name="before_test-very_high-runtimeinfohook-16" class="anchor" href="#before_test-very_high-runtimeinfohook-16"></a>before_test:<br>
(VERY_HIGH   ) RuntimeInfoHook</h2>
<h2><a name="before_test_epoch-normal-itertimerhook-17" class="anchor" href="#before_test_epoch-normal-itertimerhook-17"></a>before_test_epoch:<br>
(NORMAL      ) IterTimerHook</h2>
<h2><a name="before_test_iter-normal-itertimerhook-18" class="anchor" href="#before_test_iter-normal-itertimerhook-18"></a>before_test_iter:<br>
(NORMAL      ) IterTimerHook</h2>
<h2><a name="after_test_iter-normal-itertimerhook-below_normal-loggerhook-19" class="anchor" href="#after_test_iter-normal-itertimerhook-below_normal-loggerhook-19"></a>after_test_iter:<br>
(NORMAL      ) IterTimerHook<br>
(BELOW_NORMAL) LoggerHook</h2>
<h2><a name="after_test_epoch-very_high-runtimeinfohook-normal-itertimerhook-below_normal-loggerhook-20" class="anchor" href="#after_test_epoch-very_high-runtimeinfohook-normal-itertimerhook-below_normal-loggerhook-20"></a>after_test_epoch:<br>
(VERY_HIGH   ) RuntimeInfoHook<br>
(NORMAL      ) IterTimerHook<br>
(BELOW_NORMAL) LoggerHook</h2>
<h2><a name="after_test-very_high-runtimeinfohook-21" class="anchor" href="#after_test-very_high-runtimeinfohook-21"></a>after_test:<br>
(VERY_HIGH   ) RuntimeInfoHook</h2>
<h2><a name="after_run-below_normal-loggerhook-22" class="anchor" href="#after_run-below_normal-loggerhook-22"></a>after_run:<br>
(BELOW_NORMAL) LoggerHook</h2>
<p>10/25 12:24:26 - mmengine - WARNING - Dataset CustomVideoDataset has no metainfo. <code>dataset_meta</code> in visualizer will be None.<br>
10/25 12:24:26 - mmengine - WARNING - The prefix is not set in metric class Accuracy.<br>
10/25 12:24:26 - mmengine - WARNING - Dataset CustomVideoDataset has no metainfo. <code>dataset_meta</code> in the evaluator, metric and visualizer will be None.<br>
10/25 12:24:26 - mmengine - WARNING - “FileClient” will be deprecated in future. Please use io functions in <a href="https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io" class="inline-onebox" rel="noopener nofollow ugc">mmengine.fileio — mmengine 0.9.0 documentation</a><br>
10/25 12:24:26 - mmengine - INFO - Checkpoints will be saved to F:\HOG_HAR\work_dir.</p>

username: JuanFMontesinos
cooked: <p>In order a model to be trained in gpu, it needs to be allocated there.<br>
If the exposed class by the library is a nn.Module it’s as simple as doing <code>model=model.cuda()</code> and the analogous call for the inputs.</p>
<p>Ofc it’s hard to know what the library you are using is handling this.</p>

username: ptrblck
cooked: <p>It would also be interesting to know how you’ve confirmed the GPU wasn’t used.</p>

username: Maths_Electronics_Tu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/418ea465177962ec7a58d3a01e2da32b08e87944.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/418ea465177962ec7a58d3a01e2da32b08e87944" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea465177962ec7a58d3a01e2da32b08e87944_2_690x364.jpeg" alt="image" data-base62-sha1="9lWDZqORRZ5EST29tHxgnhBEklK" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea465177962ec7a58d3a01e2da32b08e87944_2_690x364.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea465177962ec7a58d3a01e2da32b08e87944_2_1035x546.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea465177962ec7a58d3a01e2da32b08e87944_2_1380x728.jpeg 2x" data-dominant-color="2B383E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1767×933 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebea9541c95abc3f8f2890dd2513ba9911cd74ef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ebea9541c95abc3f8f2890dd2513ba9911cd74ef" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebea9541c95abc3f8f2890dd2513ba9911cd74ef_2_690x362.jpeg" alt="image" data-base62-sha1="xF0Nraq977YD1ADQyfZ3dqO1ybB" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebea9541c95abc3f8f2890dd2513ba9911cd74ef_2_690x362.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebea9541c95abc3f8f2890dd2513ba9911cd74ef_2_1035x543.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebea9541c95abc3f8f2890dd2513ba9911cd74ef_2_1380x724.jpeg 2x" data-dominant-color="2F3B40"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1784×938 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6d26125b249821914aa949f7a95b87e8baf36d2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a6d26125b249821914aa949f7a95b87e8baf36d2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d26125b249821914aa949f7a95b87e8baf36d2_2_690x374.jpeg" alt="image" data-base62-sha1="nNLZKPK7U1IAsIIsGKyR7LTyyPM" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d26125b249821914aa949f7a95b87e8baf36d2_2_690x374.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d26125b249821914aa949f7a95b87e8baf36d2_2_1035x561.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d26125b249821914aa949f7a95b87e8baf36d2_2_1380x748.jpeg 2x" data-dominant-color="F8F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1783×969 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The Windows task manager is misleading, so either check the “compute” section of it or use <code>nvidia-smi</code>.</p>

username: Maths_Electronics_Tu
cooked: <p>ok , i used the mmaction2 framework with the same GPU, which was fast, but now it is slow , very slow and the strange issue is that the gpu memory is used highly</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60a59164ad5c64623a85fa07772b2f9062051ffa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/60a59164ad5c64623a85fa07772b2f9062051ffa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60a59164ad5c64623a85fa07772b2f9062051ffa_2_476x500.jpeg" alt="image" data-base62-sha1="dMYz1g6tBhAHJ2ilfOFA3lVACo2" width="476" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60a59164ad5c64623a85fa07772b2f9062051ffa_2_476x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60a59164ad5c64623a85fa07772b2f9062051ffa_2_714x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/0/60a59164ad5c64623a85fa07772b2f9062051ffa.jpeg 2x" data-dominant-color="1A1919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">905×949 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Maths_Electronics_Tu
cooked: <p>any help or advise i am blocked. i have work to deliver</p>

username: ptrblck
cooked: <p>You didn’t describe any issue properly. The initial claim your GPU isn’t used is wrong. In your last post you claim your run “was fast, but now it is slow , very slow”, which is not sufficient information to give you any advice. Also, you didn’t explain why a high memory usage wasn’t expected.</p>

username: Maths_Electronics_Tu
cooked: <p>i describe what i observed: the code is based on pytorch to train model on a GPU  , the gpu is detected; I use use.to.device  for data and model , but i observed training is slow; the cpu are 70%, the GPU is 10% and memory is 80%</p>

username: Gillian
cooked: <p>I’ve more or less the same issue also with two quadro RTX 5000.</p>
<p>I run the code</p>
<pre><code class="lang-auto">def benchmark(device):
    x = torch.randn(4000, 4000, device=device)
    torch.cuda.synchronize()
    start = time.time()
    for _ in range(100):
        y = x @ x
    torch.cuda.synchronize()
    print(f"{device}: {time.time() - start:.2f} seconds")

</code></pre>
<p>benchmark(‘cuda:1’) runs in 1.4s</p>
<p>benchmark(‘cuda:0’) runs in 16s</p>
<p>benchmark(‘cpu’) runs in 16s</p>
<p>I also see memory usage of gpu:0 in the task manager although everything indicates that it runs on CPU</p>

username: Arunprakash-A
cooked: <p>You may need to wrap the code in a <code>torch.cuda.device(device) </code>context manager to ensure correct results</p>
<pre data-code-wrap="python"><code class="lang-python">def benchmark(device):
    with torch.cuda.device(device): # &lt;-- select the device
        x = torch.randn(4000, 4000, device=device)
        torch.cuda.synchronize()
        start = time.time()
        for _ in range(100):
            y = x @ x
        torch.cuda.synchronize()
        print(f"{device}: {time.time() - start:.2f} seconds")
</code></pre>
<p>Here are the results<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd561ac28f6e900d600b0ef7214e246a99c3d882.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd561ac28f6e900d600b0ef7214e246a99c3d882" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd561ac28f6e900d600b0ef7214e246a99c3d882.png" alt="image" data-base62-sha1="tiumi6HGUXTkPpXLH7WT02Y4rtM" width="383" height="261"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">383×261 10.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Recommendation:</strong> Avoid using the Task Manager for performance measurements. For complex networks, use the PyTorch Profiler (<code>ProfilerActivity</code>) to measure both execution time and memory consumption.</p>
<p>Hope it helps</p>

username: Gillian
cooked: <p>Just a feedback from my side that can be helpful for someone. For me DDU was the way to the solution.</p>