Cublas runtime error : the GPU program failed to execute... with CUDA 10 and torch 1.2.0

SamB: <p>Hi,</p>
<p>I am using pytorch 1.2.0 with CUDA 10.0.130 to train a neural network on a GeForce 2080 Ti GPU. I am receiving a cublas runtime error (traceback pasted at the bottom). My code runs successfully with pytorch 1.1, CUDA 9, and GeForce GTX 1080, so the problem probably relates to my environment configuration. I thought PyTorch 1.2.0, CUDA 10, and the 2080 Tis were all compatible? More specifics about my environment are pasted below the error traceback. I appreciate any help resolving this issue – thank you!</p>
<p><strong>Error traceback</strong></p>
<p>CUDA runtime error: misaligned address (74) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda100_1549065924616/work/interface_cuda/interface.cpp:944<br>
CUDA runtime error: misaligned address (74) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda100_1549065924616/work/interface_cuda/interface.cpp:945<br>
CUDA runtime error: misaligned address (74) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda100_1549065924616/work/interface_cuda/interface.cpp:946<br>
Traceback (most recent call last):<br>
File “/dscrhome/snb21/autoencoded-vocal-analysis-master/mouse_sylls_mwe.py”, line 150, in <br>
model.train_loop(loaders, epochs=600, test_freq=None)<br>
File “/hpchome/mooneylab/snb21/autoencoded-vocal-analysis-master/ava/models/vae.py”, line 422, in train_loop<br>
loss = self.train_epoch(loaders[‘train’])<br>
File “/hpchome/mooneylab/snb21/autoencoded-vocal-analysis-master/ava/models/vae.py”, line 354, in train_epoch<br>
loss.backward()<br>
File “/dscrhome/snb21/.conda/envs/ava2/lib/python3.7/site-packages/torch/tensor.py”, line 118, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/dscrhome/snb21/.conda/envs/ava2/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 93, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/THCBlas.cu:331</p>
<p><strong>Environment Details</strong><br>
PyTorch version: 1.2.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0.130</p>
<p>OS: Red Hat Enterprise Linux Server release 7.7 (Maipo)<br>
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)<br>
CMake version: version 2.8.12.2</p>
<p>Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.0.130<br>
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti<br>
Nvidia driver version: 418.39<br>
cuDNN version: /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.5.0</p>
<p>Versions of relevant libraries:<br>
[pip] numpy==1.16.4<br>
[pip] torch==1.2.0<br>
[pip] torchvision==0.4.0a0+6b959ee<br>
[conda] blas                      1.0                         mkl<br>
[conda] magma-cuda100             2.5.1                         1    pytorch<br>
[conda] mkl                       2019.4                      243<br>
[conda] mkl-service               2.3.0            py37he904b0f_0<br>
[conda] mkl_fft                   1.0.14           py37ha843d7b_0<br>
[conda] mkl_random                1.0.2            py37hd81dba3_0<br>
[conda] pytorch                   1.2.0           py3.7_cuda10.0.130_cudnn7.6.2_0    pytorch<br>
[conda] torchvision               0.4.0                py37_cu100    pytorch</p>

ptrblck: <p>Could you post a (minimal) code snippet to reproduce this issue?</p>