Libtorch CUDA initialization: Unexpected error out of memory

Xander_Dunn: <p>I’m developing on GCP instances with A100 GPUs. Ubuntu 18.04. I’ve had no trouble running Python scripts with pytorch on GPU. I’ve recreated one of our models in C++ using the libtorch C++ interface. It runs successfully on CPU but I’ve been unable to get it to run on GPU.</p>
<p>Running this script:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
    std::cout &lt;&lt; torch::cuda::device_count() &lt;&lt; std::endl;
}
</code></pre>
<p>produces this error:</p>
<pre><code class="lang-auto">[W CUDAFunctions.cpp:109] Warning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (function operator())
0
</code></pre>
<p>You can see CUDA initialization failed with <code>out of memory (function operator())</code> and no CUDA devices are found.</p>
<p>This is a bad error because no other process is using the GPU and you can see the VRAM is completely free:</p>
<pre><code class="lang-auto">$ nvidia-smi
Sat May 15 00:16:02 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.32.00    Driver Version: 455.32.00    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:00:04.0 Off |                    0 |
| N/A   35C    P0    44W / 350W |      0MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p>In addition, the system RAM is using 2 out of 80GB.</p>
<pre><code class="lang-auto">$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0
</code></pre>
<p>The CUDA installation appears to be fine. The samples execute without issue:</p>
<pre><code class="lang-auto">$ /usr/local/cuda/samples/bin/x86_64/linux/release/cppIntegration
GPU Device 0: "Ampere" with compute capability 8.0

Hello World.
Hello World.
$ /usr/local/cuda/samples/bin/x86_64/linux/release/matrixMulCUBLAS
[Matrix Multiply CUBLAS] - Starting...
GPU Device 0: "Ampere" with compute capability 8.0

GPU Device 0: "A100-SXM4-40GB" with compute capability 8.0

MatrixA(640,480), MatrixB(480,320), MatrixC(640,320)
Computing result using CUBLAS...done.
Performance= 8408.76 GFlop/s, Time= 0.023 msec, Size= 196608000 Ops
Computing result using host CPU...done.
Comparing CUBLAS Matrix Multiply with CPU results: PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
$ /usr/local/cuda/samples/bin/x86_64/linux/release/cudaTensorCoreGemm
Initializing...
GPU Device 0: "Ampere" with compute capability 8.0

M: 4096 (16 x 256)
N: 4096 (16 x 256)
K: 4096 (16 x 256)
Preparing data for GPU...
Required shared memory size: 64 Kb
Computing... using high performance kernel compute_gemm
Time: 1.990656 ms
TFLOPS: 69.04
</code></pre>
<p>It appears to be correctly identified by CMake during the build:</p>
<pre><code class="lang-auto">-- Found CUDA: /usr/local/cuda-11.1 (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.1
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found cuDNN: v8.2.0  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
CMake Warning at /home/xander/libtorch/share/cmake/Caffe2/public/cuda.cmake:198 (message):
  Failed to compute shorthash for libnvrtc.so
Call Stack (most recent call first):
  /home/xander/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/xander/dev/vcpkg/scripts/buildsystems/vcpkg.cmake:861 (_find_package)
  /home/xander/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  /home/xander/dev/vcpkg/scripts/buildsystems/vcpkg.cmake:861 (_find_package)
  CMakeLists.txt:30 (find_package)


-- Autodetected CUDA architecture(s):  8.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_80,code=sm_80
-- Found Torch: /home/xander/libtorch/lib/libtorch.so
</code></pre>
<p>Note that the warning <code>Failed to compute shorthash for libnvrtc.so</code> no longer appears on the nightly build of libtorch, but the CUDA initialization failure remains.</p>
<p>Restarting the machine has no effect on the issue.</p>
<p>I did <code>sudo rm -r /usr/local/cuda*</code> and re-installed CUDA, the NVIDIA driver, and cudnn from NVIDIA’s .deb packages.  Again, same result. Nothing changed.</p>
<p>On the same machine I am able to run pytorch models from Python on the GPU without issue. I do NOT have conda’s <code>cudatoolkit</code> installed, so it’s using the same cuda installation as libtorch. This is confirmed when I delete the /usr/ installation of cuda, the Python code ceases to be able to access the GPU.</p>
<p>My issue appears to be similar to <a href="https://discuss.pytorch.org/t/pytorch-v1-7-1-cu110-cuda-initialization-error/113027">this</a> issue except that my CUDA samples run fine and I am having trouble only in libtorch, not in pytorch.</p>
<p>pytorch was installed with:</p>
<pre><code class="lang-auto">pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>libtorch was installed with:</p>
<pre><code class="lang-auto">wget https://download.pytorch.org/libtorch/cu111/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu111.zip
unzip libtorch-cxx11-abi-shared-with-deps-1.8.1+cu111.zip
</code></pre>
<p>I’ve tried building with GCC 10.3.0 as well as clang 12.0.0. Same result with both. cmake 3.20.1. In-between builds I am cleaning with <code>rm -r CMakeFiles CMakeCache.txt</code>.</p>
<p>Does anyone have any additional debugging steps or thoughts on why I am unable to access the GPU specifically in libtorch C++?</p>

Xander_Dunn: <p>Lord have mercy. It was this in my CMakeLists.txt:</p>
<pre><code class="lang-auto">set(CMAKE_CXX_FLAGS "-fsanitize=undefined -fsanitize=address")
</code></pre>
<p>leftover from some debugging.</p>
<p>Sanitizers breaking CUDA isn’t too surprising. <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:"></p>
<p>libtorch runs fine on GPU in the absence of these compiler flags.</p>

tuero: <aside class="quote no-group" data-username="Xander_Dunn" data-post="2" data-topic="121311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xander_dunn/48/16513_2.png" class="avatar"> Xander_Dunn:</div>
<blockquote>
<p>absence</p>
</blockquote>
</aside>
<p>Thanks for the update, just ran into the same issue banging my head against the wall trying to figure out what in my config was causing the issues.</p>