Help understand CUDA error:device-side assert triggered

Yoark_YANG: <p>Hello, all,<br>
A CUDA error occurs when I try to <code>seq_range_expand = seq_range_expand.cuda()</code>, the error message says:<br>
<code>/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [0,0,0] Assertion </code>val &gt;= zero<code>failed. /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [1,0,0] Assertion</code>val &gt;= zero<code>failed. /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [2,0,0] Assertion</code>val &gt;= zero<code>failed. /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [3,0,0] Assertion</code>val &gt;= zero<code>failed. /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [4,0,0] Assertion</code>val &gt;= zero<code>failed. /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [5,0,0] Assertion</code>val &gt;= zero<code>failed. /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::&lt;unnamed&gt;::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [1,0,0], thread: [6,0,0] Assertion</code>val &gt;= zero<code>failed.</code></p>
<p>Sorry if the question is too naive</p>

richard: <p>Try to run with CUDA_LAUNCH_BLOCKING=1 python </p>
<p>The error message probably comes from a different line than the one you linked, but it is showing up there because of the asynchronous nature of CUDA. In particular, it looks like you called <code>multinomial</code> with a negative probability somewhere.</p>

albanD: <p>Still the warning seems to be that <code>sampleMultinomialOnce</code> fails to get <code>val &gt;= zero</code>.<br>
So I guess you try to sample from a multinomial with negative values? Which is not supported?</p>

Yoark_YANG: <p>Thank you! I am looking at other people’s code, and couldn’t find the part with multinomial, do you have any clues?</p>

Yoark_YANG: <p>Thank you! Richard, yeah it is from another function.</p>

albanD: <p>You can run this on CPU or on GPU with the env variable <code>CUDA_LAUNCH_BLOCKING=1</code> to get the exact python stack trace of where this happens.</p>

Timo: <p>I get similar error message when loss of pytorch model gets very big ie infinity. For example if I decrease learning rate then I wont get this error.</p>
<p>Here is example of errors I am getting:</p>
<blockquote>
<p>/opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/ATen/native/cuda/MultinomialKernel.cu:243: void at::native::::sampleMultinomialOnce(long *, long, int, scalar_t *, scalar_t *, int, int) [with scalar_t = float, accscalar_t = float]: block: [0,0,0], thread: [2,0,0] Assertion <code>val &gt;= zero</code> failed.</p>
</blockquote>

albanD: <p>As the error mentions, you try to sample from a multinational kernel with a negative weight.<br>
You should check your code how can this happen.<br>
This most likely happens with larger learning rate because you sample value based on trained parameters and a high learning rate lead to this negative value (either due to instability or because you did not run the low learning rate version long enough).</p>

Timo: <p>Thanks! I use simple actor-critic to give simple commands. So trained parameters effects to data I sample.</p>