MultivariateNormal on GPU segmentation fault

Peter_Ham: <p>I try to generate a distribution on gpu, but got segmentation fault.</p>
<p>Code is here:</p>
<pre><code>from torch.distributions.multivariate_normal import MultivariateNormal
import torch
mean =  torch.ones(3).cuda()
scale = torch.ones(3).cuda()
mvn = MultivariateNormal(mean, torch.diag(scale))</code></pre>

ptrblck: <p>I cannot reproduce this issue using the latest PyTorch release (<code>1.7.1</code>) on a TitanV and <code>print(mvn)</code> yields:</p>
<pre><code class="lang-python">MultivariateNormal(loc: torch.Size([3]), covariance_matrix: torch.Size([3, 3]))
</code></pre>
<p>Which PyTorch and CUDA version as well as GPU are you using?</p>

Hanqing_Wang: <p>I met the same problem when running the code:</p>
<pre><code class="lang-python">from torch.distributions.multivariate_normal import MultivariateNormal
import torch
mean =  torch.ones(3).cuda()
scale = torch.ones(3).cuda()
mvn = MultivariateNormal(mean, torch.diag(scale))
</code></pre>
<p>I got a <code>Segmentation fault (core dumped)</code>.<br>
My PyTorch version is <code>1.7.1</code>, my GPU is RTX 3090, and the CUDA version is <code>11.0</code></p>

ptrblck: <p>I cannot reproduce this issue on an RTX3090 using the PyTorch <code>1.7.1</code> conda binaries with <code>cudatoolkit=11.0</code>:</p>
<pre><code class="lang-python"># tmp.py
from torch.distributions.multivariate_normal import MultivariateNormal
import torch
print(torch.cuda.get_device_name(0))
print(torch.__version__)
print(torch.version.cuda)
mean =  torch.ones(3).cuda()
scale = torch.ones(3).cuda()
mvn = MultivariateNormal(mean, torch.diag(scale))
print(mvn)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">GeForce RTX 3090
1.7.1
11.0
MultivariateNormal(loc: torch.Size([3]), covariance_matrix: torch.Size([3, 3]))
</code></pre>

Hanqing_Wang: <p>Thanks for your reply!<br>
I ran the <code>tmp.py</code> script and get the following output:</p>
<pre><code class="lang-auto">GeForce RTX 3090
1.7.1
11.0
Segmentation fault (core dumped)
</code></pre>
<p>If <code>mean</code> and <code>scale</code> are CPU tensors, everything is fine.</p>
<pre><code class="lang-auto">GeForce RTX 3090
1.7.1
11.0
MultivariateNormal(loc: torch.Size([3]), covariance_matrix: torch.Size([3, 3]))
</code></pre>
<p>My python environment is created by conda and the python version is <code>3.6</code>.</p>

ptrblck: <p>Could you check the backtrace via:</p>
<pre><code class="lang-python">gdb --args python tmp.py
...
run
...
bt
</code></pre>
<p>and post it here?</p>

Hanqing_Wang: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I got the following output:</p>
<pre><code class="lang-auto">Starting program: /home/user/anaconda3/envs/vln/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fff17bd3700 (LWP 30501)]
GeForce RTX 3090
1.7.1
11.0
[New Thread 0x7fff173d2700 (LWP 30502)]
[New Thread 0x7fff16bd1700 (LWP 30503)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffcfc35507 in ?? () from /usr/local/cuda-11.0/lib64/libcublasLt.so.11
(gdb) bt
#0  0x00007fffcfc35507 in ?? () from /usr/local/cuda-11.0/lib64/libcublasLt.so.11
#1  0x00007fffcfc36a24 in free_gemm_select () from /usr/local/cuda-11.0/lib64/libcublasLt.so.11
#2  0x00007fffda9fdaf5 in cublasDestroy_v2 () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/../../../../libcublas.so.11
#3  0x00007fff77b71380 in magma_queue_destroy_internal () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so
#4  0x00007fff77b91912 in magma_spotrf_LL_expert_gpu () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so
#5  0x00007fff77b919c4 in magma_spotrf_gpu () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so
#6  0x00007fff76460358 in void at::native::magmaCholesky&lt;float&gt;(magma_uplo_t, int, float*, int, int*) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so
#7  0x00007fff7646dfcc in at::native::_cholesky_helper_cuda(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so
#8  0x00007fff75a34a8f in at::CUDAType::_cholesky_helper(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so
#9  0x00007fffc7df8af0 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoRuntimeFunctor_&lt;at::Tensor (*)(at::Tensor const&amp;, bool), at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, bool&gt; &gt;, at::Tensor (at::Tensor const&amp;, bool)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#10 0x00007fffc868c6db in at::Tensor c10::Dispatcher::callWithDispatchKey&lt;at::Tensor, at::Tensor const&amp;, bool&gt;(c10::TypedOperatorHandle&lt;at::Tensor (at::Tensor const&amp;, bool)&gt; const&amp;, c10::DispatchKey, at::Tensor const&amp;, bool) const () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007fffc8574dc7 in at::_cholesky_helper(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#12 0x00007fffc7f1ccc3 in at::native::cholesky(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#13 0x00007fffc874c2ff in at::TypeDefault::cholesky(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#14 0x00007fffc7df8af0 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoRuntimeFunctor_&lt;at::Tensor (*)(at::Tensor const&amp;, bool), at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, bool&gt; &gt;, at::Tensor (at::Tensor const&amp;, bool)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007fffc868c6db in at::Tensor c10::Dispatcher::callWithDispatchKey&lt;at::Tensor, at::Tensor const&amp;, bool&gt;(c10::TypedOperatorHandle&lt;at::Tensor (at::Tensor const&amp;, bool)&gt; const&amp;, c10::DispatchKey, at::Tensor const&amp;, bool) const () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007fffc8575207 in at::cholesky(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007fffc9b371e5 in torch::autograd::VariableType::(anonymous namespace)::cholesky(at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#18 0x00007fffc7df8af0 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoRuntimeFunctor_&lt;at::Tensor (*)(at::Tensor const&amp;, bool), at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, bool&gt; &gt;, at::Tensor (at::Tensor const&amp;, bool)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, bool) () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#19 0x00007fffc868c6db in at::Tensor c10::Dispatcher::callWithDispatchKey&lt;at::Tensor, at::Tensor const&amp;, bool&gt;(c10::TypedOperatorHandle&lt;at::Tensor (at::Tensor const&amp;, bool)&gt; const&amp;, c10::DispatchKey, at::Tensor const&amp;, bool) const () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#20 0x00007fffc87f8d07 in at::Tensor::cholesky(bool) const () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so
#21 0x00007fffcebd7905 in torch::autograd::THPVariable_cholesky () from /home/user/anaconda3/envs/vln/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#22 0x0000555555666a14 in _PyCFunction_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/methodobject.c:231
#23 0x00005555556eea5c in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4851
#24 0x000055555571125a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#25 0x00005555556e8166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#26 0x00005555556e932c in _PyFunction_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:5084
#27 0x0000555555666ddf in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2310
#28 0x000055555566b873 in _PyObject_Call_Prepend () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2373
#29 0x000055555566681e in PyObject_Call () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2261
#30 0x00005555556bf88b in slot_tp_init () at /tmp/build/80754af9/python_1599604603603/work/Objects/typeobject.c:6420
#31 0x00005555556eed97 in type_call () at /tmp/build/80754af9/python_1599604603603/work/Objects/typeobject.c:915
#32 0x0000555555666bfb in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2331
#33 0x00005555556eebae in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4875
#34 0x000055555571125a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#35 0x00005555556e9969 in _PyEval_EvalCodeWithName (qualname=0x0, name=&lt;optimized out&gt;, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=&lt;optimized out&gt;, kwargs=0x0, kwnames=0x0, argcount=0, args=0x0,  locals=0x7ffff7f3f0d8, globals=0x7ffff7f3f0d8, _co=0x7ffff7efaa50) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#36 PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4187
#37 0x00005555556ea70c in PyEval_EvalCode (co=co@entry=0x7ffff7efaa50, globals=globals@entry=0x7ffff7f3f0d8, locals=locals@entry=0x7ffff7f3f0d8) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:731
#38 0x000055555576a574 in run_mod () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:1025
#39 0x000055555576a971 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:978
#40 0x000055555576ab73 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:419
#41 0x000055555576ac7d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:81
#42 0x000055555576e663 in run_file (p_cf=0x7fffffffdecc, filename=0x5555558aa780 L"test.py", fp=0x5555558e7940) at /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:340
#43 Py_Main () at /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:811
#44 0x000055555563843e in main () at /tmp/build/80754af9/python_1599604603603/work/Programs/python.c:69
#45 0x00007ffff77e4bf7 in __libc_start_main (main=0x555555638350 &lt;main&gt;, argc=2, argv=0x7fffffffe0d8, init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, rtld_fini=&lt;optimized out&gt;, stack_end=0x7fffffffe0c8) at ../csu/libc-start.c:310
#46 0x0000555555717d0b in _start () at ../sysdeps/x86_64/elf/start.S:103
</code></pre>

ptrblck: <p>Thank you for the stack trace. That’s really helpful!<br>
Could you install the nightly binaries as described on the <a href="https://pytorch.org/get-started/locally/">website</a> and see, if you are still running into this issue?<br>
<a href="https://github.com/pytorch/pytorch/pull/50957#issuecomment-769664974">This PR</a> should have fixed the out of bounds reads in cholesky, but is not available in <code>1.7.1</code>.</p>