YOLOv5 2022-3-25 torch 1.8.1+cu111 CUDA:0 (NVIDIA GeForce RTX 3060 Laptop GPU, 6144MiB) -> Invalid CUDA '--device 0'

luka123: <p>I’m trying to train yolov5s with bdd100k on my GPU locally and getting this error although my GPU check is correctly done. I installed latest version of cuda and pytorch.</p>
<p>ERROR: github: up to date with <a href="https://github.com/ultralytics/yolov5" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ultralytics/yolov5: YOLOv5 🚀 in PyTorch &gt; ONNX &gt; CoreML &gt; TFLite</a><br>
fatal: cannot change to ‘C:\Users\Luka\Desktop\Berkeley’: No such file or directory<br>
Traceback (most recent call last):<br>
File “C:\Users\Luka\Desktop\Berkeley dataset\yolov5s_bdd100k\yolov5\train.py”, line 643, in <br>
main(opt)<br>
File “C:\Users\Luka\Desktop\Berkeley dataset\yolov5s_bdd100k\yolov5\train.py”, line 525, in main<br>
device = select_device(opt.device, batch_size=opt.batch_size)<br>
File “C:\Users\Luka\Desktop\Berkeley dataset\yolov5s_bdd100k\yolov5\utils\torch_utils.py”, line 61, in select_device<br>
assert torch.cuda.is_available() and torch.cuda.device_count() &gt;= len(device.replace(‘,’, ‘’)), <br>
AssertionError: Invalid CUDA ‘–device 0’ requested, use ‘–device cpu’ or pass valid CUDA device(s)</p>
<p>AND after that I do nvidia-smi</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e99ebda2e1db4249dc17e8bf9012c1e27af7ca3.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e99ebda2e1db4249dc17e8bf9012c1e27af7ca3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e99ebda2e1db4249dc17e8bf9012c1e27af7ca3.png" alt="image" data-base62-sha1="fMqioaih6Pryw3fmfTFYrL2OEDx" width="690" height="389" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">888×501 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It seems your PyTorch installation cannot find the GPU.<br>
What is <code>torch.version.cuda</code> returning and could you post the output of <code>python -m torch.utils.collect_env</code>, please?</p>

luka123: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="147476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.version.cuda</code></p>
</blockquote>
</aside>
<p>Torch version = 11.1</p>
<p>…collect_env output:<br>
Collecting environment information…<br>
PyTorch version: 1.8.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: Could not collect<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Education<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: version 3.22.3</p>
<p>Python version: 3.9 (64-bit runtime)<br>
Is CUDA available: False<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU<br>
Nvidia driver version: 512.15<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.22.3</p>
<p>[pip3] torch==1.11.0</p>
<p>[pip3] torchaudio==0.11.0+cu113</p>
<p>[pip3] torchvision==0.12.0<br>
[conda] blas                      1.0                         mkl</p>
<p>[conda] cpuonly                   1.0                           0    pytorch</p>
<p>[conda] cudatoolkit               11.3.1               h59b6b97_2</p>
<p>[conda] mkl                       2021.4.0           haa95532_640</p>
<p>[conda] mkl-include               2022.0.0           haa95532_115</p>
<p>[conda] mkl-service               2.4.0            py39h2bbff1b_0</p>
<p>[conda] mkl_fft                   1.3.1            py39h277e83a_0</p>
<p>[conda] mkl_random                1.2.2            py39hf11a4ad_0</p>
<p>[conda] numpy                     1.22.3                   pypi_0    pypi</p>
<p>[conda] pytorch                   1.8.0               py3.9_cpu_0  [cpuonly]  pytorch</p>
<p>[conda] pytorch-mutex             1.0                        cuda    pytorch</p>
<p>[conda] torch                     1.11.0                   pypi_0    pypi</p>
<p>[conda] torchaudio                0.11.0+cu113             pypi_0    pypi</p>
<p>[conda] torchvision               0.12.0                   pypi_0    pypi</p>

ptrblck: <p>It seems your environments might be a bit mixed up as you claim to use the CUDA 11.1 runtime, while the setup output shows no CUDA at all:</p>
<pre><code class="lang-auto">CUDA used to build PyTorch: Could not collect
...
Is CUDA available: False
</code></pre>
<p>And also shows different installed packages:</p>
<pre><code class="lang-auto">PyTorch version: 1.8.0

[pip3] torch==1.11.0

[pip3] torchaudio==0.11.0+cu113

[pip3] torchvision==0.12.0

[conda] cpuonly 1.0 0 pytorch

[conda] pytorch 1.8.0 py3.9_cpu_0 [cpuonly] pytorch

[conda] torch 1.11.0 pypi_0 pypi

[conda] torchaudio 0.11.0+cu113 pypi_0 pypi

[conda] torchvision 0.12.0 pypi_0 pypi
</code></pre>
<p>Based on this a few <code>cpuonly</code> packages are installed, other libraries with a CUDA 11.3 runtime etc.<br>
I would recommend to create a new and empty virtual environment and install the latest stable release there to avoid these issues.</p>

luka123: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="147476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>python -m torch.utils.collect_env</p>
</blockquote>
</aside>
<p>Now have this error:</p>
<p>OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading “C:\Users\Luka\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\lib\cublas64_11.dll” or one of its dependencies.</p>
<p>and millions of this error:<br>
File “C:\Users\Luka\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\multiprocessing\reductions.py”, line 36, in <strong>del</strong><br>
File “C:\Users\Luka\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\storage.py”, line 520, in _free_weak_ref<br>
AttributeError: ‘NoneType’ object has no attribute ‘_free_weak_ref’<br>
Exception ignored in: &lt;function StorageWeakRef.<strong>del</strong> at 0x00000286F54F7C10&gt;</p>
<p>There is python -m torch.utils.collect_env output.<br>
$ python -m torch.utils.collect_env<br>
Collecting environment information…<br>
PyTorch version: 1.11.0+cu113<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.3<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Education<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: N/A</p>
<p>Python version: 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.19041-SP0<br>
Is CUDA available: True<br>
CUDA runtime version: 11.6.124<br>
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU<br>
Nvidia driver version: 512.15<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.22.3<br>
[pip3] torch==1.11.0+cu113<br>
[pip3] torchaudio==0.11.0+cu113<br>
[pip3] torchvision==0.12.0+cu113<br>
[conda] Could not collect</p>

luka123: <p>Problem is SOLVED.<br>
I make new venv with conda and install pytorch firstly, then yolov5 requirements with pip inside env directory. Now packages look like this:<br>
Versions of relevant libraries:<br>
[pip3] numpy==1.21.5<br>
[pip3] torch==1.11.0<br>
[pip3] torchaudio==0.11.0<br>
[pip3] torchvision==0.12.0<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               11.3.1               h59b6b97_2<br>
[conda] mkl                       2021.4.0           haa95532_640<br>
[conda] mkl-service               2.4.0            py39h2bbff1b_0<br>
[conda] mkl_fft                   1.3.1            py39h277e83a_0<br>
[conda] mkl_random                1.2.2            py39hf11a4ad_0<br>
[conda] numpy                     1.21.5           py39ha4e8547_0<br>
[conda] numpy-base                1.21.5           py39hc2deb75_0<br>
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8_0    pytorch<br>
[conda] pytorch-mutex             1.0                        cuda    pytorch<br>
[conda] torchaudio                0.11.0               py39_cu113    pytorch<br>
[conda] torchvision               0.12.0               py39_cu113    pytorch</p>
<p>Lastly, I set ‘–workers 4’ and my GPU can hold out  batch size of 25 with this settings. Training time is 12 minutes per epoch. Thanks for help :D.</p>

10756007: <p>I have a same problem, but the solution doesn’t work for me. Please help me.</p>

10756007: <p>Could you help me,please.<br>
I have the same problem.</p>

luka123: <p>Hi, you should create virtual env and install cuda 11.3 from PyTorch official page. After installing cuda (torchvision, torch, etc…) clone YOLOv5 and install requirements with pip. Installing packages in this order doesn’t make problems for me.</p>