Can not use gpu with torch

username: ct2034
cooked: <p>When I run any torch to work with the GPU, I always get this error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: CUDA error: out of memory</p>
</blockquote>
<p>For example, when running …</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 usr/bin/python3 -c "import torch; x = torch.linspace(0, 1, 10, device=torch.device(\"cuda:0\"))
</code></pre>
<p>Even if i select a GPU that has definitely memory left …</p>
<pre><code class="lang-auto">nvidia-smi -i 3
Wed Feb 16 21:13:11 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   3  GeForce RTX 3090    On   | 00000000:61:00.0 Off |                  N/A |
| 30%   26C    P8    18W / 350W |     15MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    3   N/A  N/A      5084      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A     11272      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A   2461850      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>When running<br>
<code>CUDA_VISIBLE_DEVICES=3 CUDA_LAUNCH_BLOCKING=1 /usr/bin/python3 -c "import torch; x = torch.linspace(0, 1, 10, device=torch.device(\"cuda:0\"))"</code></p>
<p>, I get:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/chenkel/.local/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 214, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory</p>
</blockquote>
<p>My torch version is <code>1.10.0+cu113</code><br>
Cuda and driver version you can see from <code>nvidida-smi</code> above.</p>

username: eqy
cooked: <p>What output do you get with</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=3 CUDA_LAUNCH_BLOCKING=1 /usr/bin/python3 -c "import torch; x = torch.linspace(0, 1, 10, device='cuda')"
</code></pre>
<p>?</p>

username: ct2034
cooked: <p>It’s just the same…</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 1, in<br>
File “/home/chenkel/.local/lib/python3.8/site-packages/torch/cuda/<strong>init</strong> .py”, line 214, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory</p>
</blockquote>

username: AlphaBetaGamma96
cooked: <p>Could you try running pytorch with a different CUDA version?</p>
<p>From your <code>nvidia-smi</code> command it seems that your CUDA version of your drivers currently support 11.2 yet your current install of PyTorch is CUDA 11.3, so perhaps this could cause an issue? You can check the actual version of CUDA you’re using via <code>nvcc --version</code> if you’re working on Linux.</p>
<p>You could always create a new environment and try and install pytorch with CUDA 11.2 (or whatever version <code>nvcc --version</code> states) and see if that resolves the issue?</p>

username: ct2034
cooked: <p>Okay, I know what the issue was: I had set <code>ulimit -v 10000000</code> in my <code>.bashrc</code>.</p>
<p>I don’t know how this is related. But after removing it, all works fine. I can clearly reproduce this to have been the issue. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:"></p>