HELP : RuntimeError: CUDA error: device-side assert triggered

guanghui_ma: <p>I’ve been trying for days and still can’t solve the problem.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e98548f516e53430748f3982a542550f806c3491.png" data-download-href="https://discuss.pytorch.org/uploads/default/e98548f516e53430748f3982a542550f806c3491" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e98548f516e53430748f3982a542550f806c3491.png" alt="image" data-base62-sha1="xjOOmOR9W4PsvfRREpj87WXbHJD" width="690" height="146" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1901×403 39.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40baf50016d385bea90ebfd85dd493ac0cb9c684.png" data-download-href="https://discuss.pytorch.org/uploads/default/40baf50016d385bea90ebfd85dd493ac0cb9c684" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40baf50016d385bea90ebfd85dd493ac0cb9c684.png" alt="image" data-base62-sha1="9eD6Vo8uWpwtFrcfrrrlwDY5Y5C" width="690" height="141" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1427×292 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It always comes up with the above error. On line 248, I simply take out the label corresponding to the hidden. I make sure that this operation is without error.</p>
<blockquote>
<pre><code>    for one_class in label_set:
        source_index = source_label == one_class
        target_index = target_label == one_class
        select_source_hidden = source_hidden[source_index]
        select_target_hidden = target_hidden[target_index]
</code></pre>
</blockquote>

ptrblck: <p>An indexing operation fails and you could rerun the code via <code>CUDA_LAUNCH_BLOCKING=1</code> which would then point towards the failing operation in the stacktrace. Often e.g. an embedding layer is failing if the input tensor contains values which are out of bounds.</p>

guanghui_ma: <p>In fact, I have used CUDA_LAUNCH_BLOCKING=1 very early on, and it shows me a tensor passing error. But I can ensure that it is passed correctly.<br>
Also, the CUDA error: device-side assert triggered is reproduced in many cases in the current project, and not in others. In several other projects, my virtual environment works fine without any errors. Therefore, I think there is some potential conflict in the package installation.</p>
<p>I spent a day reconfiguring the new virtual environment, but still could not resolve the issue.</p>

guanghui_ma: <p>What I can’t understand is that in the new virtual environment when setting DDPStrategy(find_unused_parameters=True) it does not give an error. When DDPStrategy(find_unused_parameters=Fasle), it gives the above error.</p>

guanghui_ma: <aside class="quote no-group" data-username="guanghui_ma" data-post="1" data-topic="182378">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/guanghui_ma/48/43387_2.png" class="avatar"> guanghui_ma:</div>
<blockquote>
<pre><code class="lang-auto">    for one_class in label_set:
        source_index = source_label == one_class
        target_index = target_label == one_class
        select_source_hidden = source_hidden[source_index]
        select_target_hidden = target_hidden[target_index]
</code></pre>
</blockquote>
</aside>
<p>The code where the error occurs is always shown here. I just did a very simple operation here, i.e. I took out the corresponding hidden by label. source_hidden and target_hidden have the dimensions batch_size * hidden . The dimension of source_label and source_label is batch_size . I don’t think there is anything wrong with this code.</p>

ptrblck: <p>The line of code and the assert both point towards a failing indexing operation so I would recommend checking the tensor shapes and the values of the indexing tensors to make sure they contain only valid indices.</p>