CUDA installation cryptic errors

username: ntrstd11
cooked: <p>I’m trying to setup torch 1.1 with CUDA, but I’m encountering random errors difficult to debug. After installation, when I try to run the minimal code to test CUDA functionality:</p>
<pre><code class="lang-auto">import torch
torch.zeros(5).cuda()
</code></pre>
<p>I get random cryptic errors, one example being</p>
<pre><code class="lang-auto">fatal   : Memory allocation failure
fatal   : Memory allocation failure
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: unknown error
</code></pre>
<p>or a CUDA out of memory error (even tough I’m only trying to copy a small array) or even a segmentation fault, causing Python to crash.</p>
<p>To give some context, I’m using a conda environment with Python 3.7, and I have a CUDA 9.0 installation with cuDNN 7.5.1. I install PyTorch with the command from the website:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
</code></pre>
<p>I have also tried to using Pytorch 0.4 and CUDA 10.0/10.1, but the error persists. The installed driver version is 418.74. Finally, I do not have root access since I’m computing on a shared server and due to this reason CUDA is installed in a custom directory.</p>

username: ptrblck
cooked: <p>The PyTorch binaries ship with the CUDA and cudnn runtimes, so you would just need to install the driver on your machine.</p>
<p>Which GPU(s) are you using?<br>
Does <code>nvidia-smi</code> run successfully?<br>
Did you update the driver without restarting?</p>

username: ntrstd11
cooked: <p>Thank you for the reply,</p>
<p>The GPUs I tested were GTX TITAN X, TITAN Xp and Tesla K40, and all three reported the same errors, though for the Xp Python hangs indefinitely rather than an explicit message.</p>
<p>In all cases <code>nvidia-smi</code> runs without issues. The drivers were not installed by me (as I do not have root access), but I think they work fine since other users of the server have not reported a problem. However the drivers were updated by the admin some time ago, just before the problems started. I clean installed each component I could, however could not make any progress.</p>
<p>Apparently cleaning the cache at <code>~/.nv</code> might be relevant in some cases after a driver change, but it did not help for me. I wonder if PyTorch keeps other driver-related cache files somewhere that might be causing a mismatch.</p>
<p>The servers run Debian and I connect via ssh, if that might be relevant.</p>

username: ptrblck
cooked: <p>That’s really weird.<br>
Does your system have <code>docker</code> and <code>nvidia-docker</code> on it?<br>
If so, could you try to run the PyTorch Docker container as explained <a href="https://github.com/pytorch/pytorch#docker-image" rel="nofollow noopener">here</a>?<br>
I would like to try to create a clean environment and see, if that’s working.</p>

username: ntrstd11
cooked: <p>Unfortunately the system does not have docker installed, not sure if there is another way to test this without root access. I also tried clean installing Python using <code>pyenv</code> and installing torch via <code>pip</code> to see if it was a conda related issue, this did not work either.</p>

username: ptrblck
cooked: <p>Could you also try to create a clean conda environment and reinstall PyTorch there?</p>
<p>If that doesn’t help and CUDA is installed on the machine, could you try to build PyTorch from source?</p>
<p>Are you successfully running any CUDA code in this machine?</p>

username: ntrstd11
cooked: <p>I tried deleting miniconda entirely, reinstalling it, creating a new empty environment (just in case) and installing PyTorch with the command from the website. Unfortunately I ran into the exact same issue, once again.</p>
<p>On the other hand I can successfully compile and run the CUDA samples distributed with version 9.0.176, so CUDA and the <code>nvcc</code> tool seem to be working and configured properly, even though I understand it’s irrelevant for PyTorch.</p>
<p>I will shortly try to compile PyTorch from scratch and report the results.</p>

username: ntrstd11
cooked: <p>So I tried to compile PyTorch from scratch with CUDA support. I installed CUDA toolkit 9.2 locally, configured the environment variables and compile-installed PyTorch to a clean conda environment (as described in the PyTorch repo).</p>
<p>Unfortunately, this seemed to not work either. CMake successfully detects CUDA and cudnn during compilation. Weirdly enough, after installation running</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.is_available())
print(torch.version.cuda)
print(torch.backends.cudnn.enabled)
print()

torch.zeros(5).cuda()
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">False
9.2.148
True

THCudaCheck FAIL file=../aten/src/THC/THCGeneral.cpp line=51 error=2 : out of memory
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/yardima/miniconda3/lib/python3.7/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (2) : out of memory at ../aten/src/THC/THCGeneral.cpp:51
</code></pre>
<p>What could be wrong here? The CUDA 9.2 samples still compile and run cleanly, but PyTorch does not work.</p>
<p>As a side note, I tried compiling with CUDA 9.0, but CMake complained GCC version &gt;6 is incompatible with CUDA &lt;9.2. The server I use has GCC 6.3.0 but not GCC 5. Is there a way to circumvent this without going through the hassle of compiling GCC-5, as I do not have root priveleges?</p>

username: ptrblck
cooked: <p>Could you check the memory usage in <code>nvidia-smi</code> and see, if the GPU is used by other processes or if some dead processes are filling it up?</p>
<p>I’m not sure, if there is a simple way of changing gcc in your current setup.<br>
The cleanest way would probably still be to use docker, since you can just wipe it in the worst case.</p>

username: ntrstd11
cooked: <p>Checking <code>nvidia-smi</code> shows that the the server is almost entirely free. Among 8 GPUs available, one is assigned to me (set by <code>CUDA_VISIBLE_DEVICES</code>), which is always unused. But the problem persists even in cases where the server usage is zero.</p>
<p>I also realised that although docker is not supported, the server supports Singularity containers, maybe that could help?</p>

username: ptrblck
cooked: <p>Could you check, what <code>torch.cuda.device_count()</code> returns?<br>
Are you seeing more than a single GPU and might accidentally use the wrong one?</p>
<p>I’m not familiar with Singularity containers, but you could give it a try.</p>

username: ntrstd11
cooked: <p>Interestingly, even though <code>CUDA_VISIBLE_DEVICES</code> is set correctly, <code>torch.cuda.device_count()</code> returns 0.</p>

username: ptrblck
cooked: <p>Could you skip setting <code>CUDA_VISIBLE_DEVICES</code> and check the returned counts then?</p>

username: ntrstd11
cooked: <p>It still returns zero, even if I do not set <code>CUDA_VISIBLE_DEVICES</code>.</p>

username: ptrblck
cooked: <p>Does <code>torch.cuda.is_available()</code> also return <code>False</code>?<br>
Could you check, if the flag is set in your <code>.bashrc</code> or somewhere else by using<br>
<code>echo $CUDA_VISIBLE_DEVICES</code> before running your script?</p>

username: ntrstd11
cooked: <p>As I mentioned, running the minimal script</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.is_available())
print(torch.version.cuda)
print(torch.backends.cudnn.enabled)
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">False
9.2.148
True
</code></pre>
<p>I double checked the <code>CUDA_VISIBLE_DEVICES</code> flag, but it is not set explicitly prior to the PyTorch code.</p>

username: ptrblck
cooked: <p>I’m not sure what’s going on, and would personally try the brute force approach:</p>
<ul>
<li>create a new conda env for each run and try to install all current binaries</li>
<li>start from the nightly build with CUDA10, then go down to CUDA9</li>
<li>end with the current stable version</li>
<li>try the pip versions next</li>
</ul>
<p>Also, could you post the build log here which you have gotten while building from source?</p>