When I use amp for accelarate the model, i met the problem“RuntimeError: CUDA error: device-side assert triggered”?

Mr.Crab: <p>I wangt to use amp for mixed-precision training. But when i set amp.autocast is true:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/373e40bbe4114e08f7d0b6945f97ca0c28a5464b.png" data-download-href="https://discuss.pytorch.org/uploads/default/373e40bbe4114e08f7d0b6945f97ca0c28a5464b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/373e40bbe4114e08f7d0b6945f97ca0c28a5464b_2_690x331.png" alt="image" data-base62-sha1="7SHDuJonX3UNGco7GIOHvzF0iDx" width="690" height="331" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/373e40bbe4114e08f7d0b6945f97ca0c28a5464b_2_690x331.png, https://discuss.pytorch.org/uploads/default/original/3X/3/7/373e40bbe4114e08f7d0b6945f97ca0c28a5464b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/7/373e40bbe4114e08f7d0b6945f97ca0c28a5464b.png 2x" data-dominant-color="2D3138"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">889×427 42.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I met the problem as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4ceaa93598c5dc3313b4b680c069b033b3c175b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ceaa93598c5dc3313b4b680c069b033b3c175b3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4ceaa93598c5dc3313b4b680c069b033b3c175b3.png" alt="image" data-base62-sha1="aYr2T1v6AYoWItsu0TJ2ceHzrqz" width="690" height="64" data-dominant-color="35323C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">950×89 7.54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
What puzzles me is i can train the model normally, when i set the amp.autocast is false. So how can i to solve the problem?</p>

ptrblck: <p>Could you rerun the script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stadcktrace here by wrapping it into three backticks ```, please?</p>

Mr.Crab: <p>The same error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/725104aa64c721d24e9ee63bf92dffbbb832ce94.png" data-download-href="https://discuss.pytorch.org/uploads/default/725104aa64c721d24e9ee63bf92dffbbb832ce94" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/725104aa64c721d24e9ee63bf92dffbbb832ce94_2_690x495.png" alt="image" data-base62-sha1="gji0LzXdeUkstAI9BDja6B7GEgk" width="690" height="495" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/725104aa64c721d24e9ee63bf92dffbbb832ce94_2_690x495.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/725104aa64c721d24e9ee63bf92dffbbb832ce94_2_1035x742.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/2/725104aa64c721d24e9ee63bf92dffbbb832ce94.png 2x" data-dominant-color="31303B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1152×827 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for the update! The stacktrace points to an invalid indexing operation in:</p>
<pre><code class="lang-python">points[batch_indices, idx, :]
</code></pre>
<p>so check the shape of <code>points</code> as well as the min. and max. values of <code>batch_indices</code> and <code>idx</code> and make sure they are valid.</p>

Mr.Crab: <p>But i can train the model normally, when i set the amp.autocast is false. So i don’t think it is the points’value that is the key to the problem.</p>