Random CUDA error: device-side assert triggered (once every week)

username: Emil_Wallner
cooked: <p>Hey everyone,</p>
<p>I’ve been using TorchServe in production for over a year, handling several million requests weekly without issues. Recently, it started crashing every few days.</p>
<p>I’ve added more details in a github issue: <a href="https://github.com/pytorch/serve/issues/3114" class="inline-onebox" rel="noopener nofollow ugc">CUDA out of Memory with low Memory Utilization (CUDA error: device-side assert triggered) · Issue #3114 · pytorch/serve · GitHub</a></p>
<p>To diagnose, I set CUDA_LAUNCH_BLOCKING=1 and encountered a “CUDA error: device-side assert triggered” and a “CUDA out of memory” error when transferring data to the GPU.</p>
<p>Here are some details:</p>
<ul>
<li>I log torch.cuda.max_memory_allocated() and torch.cuda.memory_allocated().</li>
<li>Typically, the models use about 6180 MiB out of 23028 MiB available.</li>
<li>The torch.cuda.max_memory_allocated() logs around 366 MB.</li>
<li>I suspected a memory leak, shape mismatches, or NaN values, but nothing stood out. However, I’ve added a lot of checks to make sure it’s not the case.</li>
</ul>
<p>There are a few things that are a bit odd about this issue:</p>
<ul>
<li>The server has been running fine for over a year, I only made a few updates a few months back, and all of a sudden it started crashing frequently</li>
<li>I thought it was some edge case that crashed the server, but it only crashes some of the running instances</li>
<li>It happens randomly every 3-5 days, that’s why I assumed it was some memory leak, but I can’t find any evidence of it</li>
<li>I get a device-side assert triggered, and CUDA out of memory, however the available memory seems to be plenty, and I check for any NaN value or wrong shape before placing it on the GPU.</li>
</ul>
<p>I’ve run out of ideas, any thought or feedback would be much appreciated.</p>

username: Emil_Wallner
cooked: <p>Before the device-side assert triggered error, I get 32 warnings which probably overloads the memory and causes the problem:</p>
<p>/opt/pytorch/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [28,0,0], thread: [53,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</p>
<p>/opt/pytorch/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [28,0,0], thread: [54,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</p>
<p>/opt/pytorch/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [28,0,0], thread: [55,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Emil_Wallner" data-post="2" data-topic="202844">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emil_wallner/48/69571_2.png" class="avatar"> Emil_Wallner:</div>
<blockquote>
<p>/opt/pytorch/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [28,0,0], thread: [53,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.</p>
</blockquote>
</aside>
<p>You are running into an indexing error and should isolate which line of code calls into this <code>scatter/gather</code> kernel.</p>
<aside class="quote no-group" data-username="Emil_Wallner" data-post="1" data-topic="202844">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emil_wallner/48/69571_2.png" class="avatar"> Emil_Wallner:</div>
<blockquote>
<p>I only made a few updates a few months back, and all of a sudden it started crashing frequently</p>
</blockquote>
</aside>
<p>Try to correlate these changes to any indexing, scatter, gather operation and verify these ops are using valid indices.</p>
<aside class="quote no-group" data-username="Emil_Wallner" data-post="1" data-topic="202844">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emil_wallner/48/69571_2.png" class="avatar"> Emil_Wallner:</div>
<blockquote>
<p>I suspected a memory leak, shape mismatches, or NaN values, but nothing stood out.</p>
</blockquote>
</aside>
<p>I don’t think the OOM is real and just reported after running into the sticky assert, so I would ignore it for now and focus in the indexing error, which is real.</p>