Segmentation fault (Core dump) when using model.cuda

ellenW: <p>Hi, I’m getting a Segmentation Fault when using model.cuda.</p>
<p>Torch version =1.2.0  , gpu Quadro RTX 5000 , Cuda :11.2</p>
<p>Here is output of gdb:</p>
<p>New Thread 0x7fff63ff5700 (LWP 110466)]</p>
<p>Thread 1 “python” received signal SIGSEGV, Segmentation fault.</p>
<p>0x00007ffef9e3faae in ?? () from /lib64/libcuda.so.1</p>
<p>(gdb)</p>
<p>(gdb) where</p>
<p><span class="hashtag">#0</span> 0x00007ffef9e3faae in ?? () from /lib64/libcuda.so.1</p>
<p><span class="hashtag">#1</span> 0x00007ffef9e2b2f9 in ?? () from /lib64/libcuda.so.1</p>
<p><span class="hashtag">#2</span> 0x00007ffef9c4ab7e in ?? () from /lib64/libcuda.so.1</p>
<p><span class="hashtag">#3</span> 0x00007ffef9cac3a0 in ?? () from /lib64/libcuda.so.1</p>
<p><span class="hashtag">#4</span> 0x00007ffef9c66b7f in ?? () from /lib64/libcuda.so.1</p>
<p><span class="hashtag">#5</span> 0x00007ffef9da56ec in cuDevicePrimaryCtxRetain () from /lib64/libcuda.so.1</p>
<p><span class="hashtag">#6</span> 0x00007fffe35b8d30 in ?? () from …/lib/python3.7/site-packages/torch/lib/libcudart-1581fefa.so.10.0</p>
<p><span class="hashtag">#7</span> 0x00007fffe35b9832 in ?? () from …/lib/python3.7/site-packages/torch/lib/libcudart-1581fefa.so.10.0</p>
<p><span class="hashtag">#8</span> 0x00007fffe35ba2e8 in ?? () from …/lib/python3.7/site-packages/torch/lib/libcudart-1581fefa.so.10.0</p>
<p><span class="hashtag">#9</span> 0x00007fffe35ad43e in ?? () from …./lib/python3.7/site-packages/torch/lib/libcudart-1581fefa.so.10.0</p>
<p><span class="hashtag">#10</span> 0x00007fffe359cde8 in ?? () from …/lib/python3.7/site-packages/torch/lib/libcudart-1581fefa.so.10.0</p>
<p><span class="hashtag">#11</span> 0x00007fffe35ce23c in cudaMalloc () from …/lib/python3.7/site-packages/torch/lib/libcudart-1581fefa.so.10.0</p>

ptrblck: <p>Could you update to the latest stable or nightly release and check the code again?<br>
If you are still running into the issue, could you post a minimal code snippet to reproduce the issue and the output of <code>python -m torch.utils.collect_env</code>?</p>

ellenW: <p>I created a new virtual environment and install latest torch version using the following command:</p>
<p>conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch</p>
<p>Unfortunately I encountered the same error again. Server has 4 gpus but I use gpu-0. Gpu-1</p>
<p>and 2 are not appropriate for torch.</p>
<p>Here is output  python -m torch.utils.collect_env :</p>
<p>PyTorch version: 1.8.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: CentOS Linux release 8.3.2011 (x86_64)<br>
GCC version: (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)<br>
Clang version: Could not collect<br>
CMake version: version 3.11.4</p>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: Quadro RTX 5000<br>
GPU 1: Tesla K10.G1.8GB<br>
GPU 2: Tesla K10.G1.8GB<br>
GPU 3: Quadro RTX 5000</p>
<p>Nvidia driver version: 460.32.03<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.20.2<br>
[pip3] torch==1.8.0<br>
[pip3] torchaudio==0.8.0a0+a751e1d<br>
[pip3] torchvision==0.9.0<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.2.89              hfd86e86_1<br>
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
[conda] mkl                       2021.2.0           h06a4308_296<br>
[conda] mkl-service               2.3.0            py38h27cfd23_1<br>
[conda] mkl_fft                   1.3.0            py38h42c9631_2<br>
[conda] mkl_random                1.2.1            py38ha9443f7_2<br>
[conda] numpy                     1.20.2           py38h2d18471_0<br>
[conda] numpy-base                1.20.2           py38hfae3a4d_0<br>
[conda] pytorch                   1.8.0           py3.8_cuda10.2_cudnn7.6.5_0    pytorch<br>
[conda] torchaudio                0.8.0                      py38    pytorch<br>
[conda] torchvision               0.9.0                py38_cu102 pytorch</p>
<p>Minimal code snippet :</p>
<p><span class="hashtag">#pdb</span>.set_trace()<br>
train_size = len(train_dataset)<br>
print(train_size)<br>
train_batch = data.DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True, num_workers=args.num_workers, drop_last=True)</p>
<p>model = convAE(args.c, args.t_length, args.psize, args.fdim[0], args.pdim[0])<br>
<span class="hashtag">#pdb</span>.set_trace()<br>
model.cuda() -------- Execution of this is problematic.</p>
<p>params_encoder =  list(model.encoder.parameters())<br>
params_decoder = list(model.decoder.parameters())</p>
<p>Thank you…</p>

ellenW: <p>I solve the problem ! :)) .  I add</p>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
print(device)<br>
print(f"running with device: {torch.cuda.get_device_name(torch.cuda.current_device())}"</p>
<p>and change model.cuda() to model.to(device). I think the problem was wrong gpu selection.</p>

iucario: <p>I solved my error by using <code>model.to(torch.device('cuda'))</code></p>