Using pytorch with cuda

Izan_C_G: <p>Hello, I am trying to execute Pytorch with Cuda on two machines. In both cases, I didn’t get the expected results. I will try to explain both cases to see if I am doing something wrong or if there is any incompatibility I cannot see.</p>
<p>Case 1)</p>
<ul>
<li>Cuda installed (nvcc --version): 11.2</li>
<li>torch                           1.12.1+cu113</li>
<li>torchaudio                   0.12.1+cu113</li>
<li>torchvision                  0.13.1+cu113</li>
</ul>
<p>I run the following code (some parts of it are omitted for simplifications purposes):</p>
<pre><code class="lang-auto">print(f"Is CUDA supported by this system? {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")

# Storing ID of current CUDA device
cuda_id = torch.cuda.current_device()
print(f"ID of current CUDA device: {torch.cuda.current_device()}")
       
print(f"Name of current CUDA device: {torch.cuda.get_device_name(cuda_id)}")

device = 'cuda' if torch.cuda.is_available() else 'cpu'

model = model.cuda()
model = model.eval() 

with torch.no_grad():
    for i, (x, y) in enumerate(val_loader):
        x = x.cuda()
        y = y.cuda()
        y_pred = model(x)
</code></pre>
<p>The result I get is the following:</p>
<pre><code class="lang-auto">
Is CUDA supported by this system? True
CUDA version: 11.3
ID of current CUDA device: 0
Name of current CUDA device: NVIDIA A100-PCIE-40GB
inputs torch.Size([1024, 3, 224, 224])  \ rep  0  \ labels  torch.Size([1024])
Segmentation fault (core dumped)
</code></pre>
<p>When the model does the inference, fails. I don’t know why, because when I run the example with gdb, it gets stuck, and does not finish.</p>
<p>If I run some commands like nvidia-smi -lms to see if there is some program in execution on GPU, it appears, but with a 0% usage of GPU.</p>
<p>If, for instance, I eliminate x.cuda and y.cuda, the program finishes okay, and with Nvidia-semi, it appears like there is some MiB of the GPU allocated but with 0% usage. I suppose this happens because the model is in GPU, but the inference is not.</p>
<p>I change the code, instead of using model.cuda(), model.to(device) or x.to(device) with the same output.</p>
<p>case 2)</p>
<ul>
<li>Cuda installed (nvcc --version): 11.8</li>
<li>torch                        2.4.1+cu118</li>
<li>torchaudio                   2.4.1+cu118</li>
<li>torchvision                  0.19.1+cu118</li>
</ul>
<p>same code, and this is the output:</p>
<pre><code class="lang-auto">Is CUDA supported by this system? True
CUDA version: 11.8
ID of current CUDA device: 0
Name of current CUDA device: Tesla V100-PCIE-32GB
Segmentation fault (core dumped)
</code></pre>
<p>I get the same output with different Cuda configurations and different GPUs, and I can detect not only the Cuda version but also the GPU in my code with functions like current_device() , get_device() or is_available().</p>
<p>Again, if a don’t use x.to(device) or y.to(device) works but with 0% usage. Surprisingly, model.to(device) or model.cuda() never fails.</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:3D:00.0 Off |                    0 |
| N/A   35C    P0    35W / 250W |   1684MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
</code></pre>
<p>The only possible answer I find is that somehow I am using the wrong code to send the model and the tensors to GPU, or I don’t have a compatible torch version.</p>
<p>Any help would be appreciated,</p>
<p>Thank you.</p>

JuanFMontesinos: <p>Have you tried to run the model on cpu?<br>
You are getting segmentation fault, core dump. It’s a bit generic but typically means error on the C++ side.</p>

ptrblck: <p>Try to run your workload via:</p>
<pre><code class="lang-auto">gdb--args python script.py
...
run
...
bt
</code></pre>
<p>to print the backtrace of the segfault.</p>

Izan_C_G: <p>This is the output from gdb:</p>
<pre><code class="lang-auto">Thread 66 "python3.8" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff5b1aa700 (LWP 25254)]
__memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2509
2509    ../sysdeps/x86_64/multiarch/memcpy-ssse3.S: No such file or directory.
(gdb) bt
#0  __memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2509
#1  0x00007ffeefdb5754 in void fbgemm::pack_a_with_im2col_opt&lt;2, 512&gt;(fbgemm::conv_param_t&lt;2&gt; const&amp;, fbgemm::block_type_t const&amp;, unsigned char const*, unsigned char*, int, int*, int, int, bool) ()
   from /mnt/beegfs/gap/izcagal/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#2  0x00007ffeefdb60ce in fbgemm::PackAWithIm2Col&lt;unsigned char, int, 2&gt;::pack(fbgemm::block_type_t const&amp;) () from /mnt/beegfs/gap/izcagal/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#3  0x00007ffeefd3d8a4 in void fbgemm::fbgemmPacked&lt;fbgemm::PackAWithIm2Col&lt;unsigned char, int, 2&gt;, fbgemm::PackBMatrix&lt;signed char, int&gt;, unsigned char, fbgemm::ReQuantizeOutput&lt;true, (fbgemm::QuantizationGranularity)2, float, unsigned char, int, fbgemm::DoNothing&lt;unsigned char, unsigned char&gt; &gt; &gt;(fbgemm::PackMatrix&lt;fbgemm::PackAWithIm2Col&lt;unsigned char, int, 2&gt;, fbgemm::PackAWithIm2Col&lt;unsigned char, int, 2&gt;::inpType, fbgemm::PackAWithIm2Col&lt;unsigned char, int, 2&gt;::accType&gt;&amp;, fbgemm::PackMatrix&lt;fbgemm::PackBMatrix&lt;signed char, int&gt;, fbgemm::PackBMatrix&lt;signed char, int&gt;::inpType, fbgemm::PackBMatrix&lt;signed char, int&gt;::accType&gt;&amp;, unsigned char*, int*, unsigned int, fbgemm::ReQuantizeOutput&lt;true, (fbgemm::QuantizationGranularity)2, float, unsigned char, int, fbgemm::DoNothing&lt;unsigned char, unsigned char&gt; &gt; const&amp;, int, int, fbgemm::BlockingFactors const*) ()
   from /mnt/beegfs/gap/izcagal/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#4  0x00007ffeefd7e815 in int fbgemm::fbgemmConv&lt;fbgemm::ReQuantizeOutput&lt;true, (fbgemm::QuantizationGranularity)2, float, unsigned char, int, fbgemm::DoNothing&lt;unsigned char, unsigned char&gt; &gt;, 2, int&gt;(fbgemm::conv_param_t&lt;2&gt; const&amp;, unsigned char const*, fbgemm::PackWeightsForConv&lt;2, signed char, int&gt;&amp;, fbgemm::ReQuantizeOutput&lt;true, (fbgemm::QuantizationGranularity)2, float, unsigned char, int, fbgemm::DoNothing&lt;unsigned char, unsigned char&gt; &gt;::outType*, int*, fbgemm::ReQuantizeOutput&lt;true, (fbgemm::QuantizationGranularity)2, float, unsigned char, int, fbgemm::DoNothing&lt;unsigned char, unsigned char&gt; &gt;&amp;, int, int, fbgemm::BlockingFactors const*) ()
   from /mnt/beegfs/gap/izcagal/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#5  0x00007ffeebd020cb in void at::internal::invoke_parallel&lt;PackedConvWeight&lt;2&gt;::apply_impl&lt;true&gt;(at::Tensor const&amp;, double, long)::{lambda(long, long)#2}&gt;(long, long, long, PackedConvWeight&lt;2&gt;::apply_impl&lt;true&gt;(at::Tensor const&amp;, double, long)::{lambda(long, long)#2} const&amp;) [clone ._omp_fn.0] ()
   from /mnt/beegfs/gap/izcagal/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#6  0x00007fffdc5c0405 in ?? ()
   from /mnt/beegfs/gap/izcagal/.local/lib/python3.8/site-packages/torch/lib/libgomp-a34b3233.so.1
#7  0x00007ffff7bbb6db in start_thread (arg=0x7fff5b1aa700) at pthread_create.c:463
--Type &lt;RET&gt; for more, q to quit, c to continue without paging--c
#8  0x00007ffff713f61f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
</code></pre>

Izan_C_G: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, could you take a look at the backtrace?</p>

nickums: <p>I like using accelerate to optionally place all my data on the gpu.</p>

ptrblck: <p>Thanks for providing the stacktrace! It seems <code>fbgemm</code> is failing with some <code>SSE3</code> instructions but I don’t know which CPU features are required for its execution. CC <a class="mention" href="/u/malfet">@malfet</a> do you know if SSE3 is supported in current <code>fbgemm</code> builds?</p>

malfet: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I think sse3 is a red herring here (note, that it fails with SIGSEGV not SIGILL), i.e. looks like either src or dst pointer is invalid.<br>
Also, fbgemm just calls to <code>memcpy</code> (for example see <a href="https://github.com/pytorch/FBGEMM/blob/49fa9a55ba97a0d655cf2c51bf595caab37638e1/src/PackAWithIm2Col.cc#L260" class="inline-onebox" rel="noopener nofollow ugc">FBGEMM/src/PackAWithIm2Col.cc at 49fa9a55ba97a0d655cf2c51bf595caab37638e1 · pytorch/FBGEMM · GitHub</a> ) and than glibc decides to make a dispatch to one or another accelerated flavor based on the CPU instruction set support.</p>
<p><a class="mention" href="/u/izan_c_g">@Izan_C_G</a> do you mind running <code>info registers</code> and <code>disasm</code> commands when exception happens?</p>

Izan_C_G: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/malfet">@malfet</a> problem solved (more or less)<br>
The source of the problem was not the CUDA installation/version or compatibility with Pytorch. It was a problem with the NN model used to make the inference.</p>
<p>I was working with the Resnet50 Quantized model. If Resnet50 is used with the fp32 datatype, it works fine, and the GPU is used as it should be. However, quantized models always get segmentation faults.</p>
<p>I realized that in <a href="https://pytorch.org/vision/stable/models/generated/torchvision.models.quantization.resnet50.html#torchvision.models.quantization.resnet50" rel="noopener nofollow ugc">torchvision 0.19 API</a>, there is a note saying that quantized models don’t work with GPU. I think this is the problem. I ignored that.</p>
<p>However, the problem I have now, is that I want to use quantized models, and I like to use them in GPU.  Is there any possible solution to that?</p>