Assertion `srcIndex < srcSelectDimSize` failed on Transformer Decoder

JaniceXiong: <p>Hi, I built a fairseq transformer model and encountered the following assertion error when running my code on <strong>GPU</strong>. According to traceback, it seems to happen in <strong>Transformer Decoder “extract_features_scriptable”</strong> below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee145816ffef695f1e151e4deb3f5053b4bc371c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee145816ffef695f1e151e4deb3f5053b4bc371c" title="ce4fc50179795c593a6eb7872c6536d"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee145816ffef695f1e151e4deb3f5053b4bc371c_2_690x113.png" alt="ce4fc50179795c593a6eb7872c6536d" data-base62-sha1="xY9dWUj9aRnbOq5DPHX3gkNvmQk" width="690" height="113" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee145816ffef695f1e151e4deb3f5053b4bc371c_2_690x113.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee145816ffef695f1e151e4deb3f5053b4bc371c_2_1035x169.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee145816ffef695f1e151e4deb3f5053b4bc371c_2_1380x226.png 2x" data-dominant-color="363636"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ce4fc50179795c593a6eb7872c6536d</span><span class="informations">1658×272 125 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">if self.cross_self_attention or prev_output_tokens.eq(self.padding_idx).any():
            self_attn_padding_mask = prev_output_tokens.eq(self.padding_idx) # this line
</code></pre>
<p>And according to the training process, this error didn’t occur at the very beginning, but occurred in the batch sample 800 / 10000.</p>
<p>Could anyone give some advice on how to fix it or how to catch the assertion error so that I can analyze the error in detail? Thanks!</p>

ptrblck: <p>To get a better stacktrace which should point to the failing operation you could rerun your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> or use the CPU (might be too slow depending on the actual workload).<br>
If you are sure the “Transformer Decoder” module raised this error, then check all indexing operations there and make sure the indexing tensor contains valid indices.</p>

JaniceXiong: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="148985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
</aside>
<p>Thanks for replying!<br>
Using CPU is too low and because the error did not occur at the very beginning, so there has been no result for a long time <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
I wonder why the eq operation can produce the <strong>index assertion error</strong> like the picture above?</p>

ptrblck: <p>Did you rerun the code with the blocking launches and was the <code>eq</code> operation still failing?<br>
If not, then note that the stacktrace might be wrong due to the asynchronous execution of the CUDA kernels and you would need to rerun the script as mentioned before.</p>

JaniceXiong: <p>Thanks. I reproduce the error using CPU, and it occurred in fairseq<br>
Here I use the TransformerEncoder. And the <code>src_tokens.shape</code> is [3,333], and the src_select_index is tensor[0,1,2]</p>
<pre><code class="lang-auto">encoder_out = self.encoder(
            src_tokens,
            src_lengths=src_lengths,
            token_embeddings=token_embeddings,
            return_all_hiddens=return_all_hiddens
        )

</code></pre>
<p>But a strange shape error occurred in <code>encoder_out</code>. <code>encoder_out["encoder_out"][0].shape</code> is [333,3,1024] which is right, but <code>encoder_out["encoder_states"][0].shape</code> is [333,2,1024]. The batch size in dim 1 is different.<br>
So when I use <code>reorder_encoder_out</code> function, shape error occurred.</p>
<pre><code class="lang-auto">facet_encoder_out = self.encoder.reorder_encoder_out(encoder_out, src_select_index)
</code></pre>
<p>Do you have any suggestion about it? I really could not find why it occurred <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I’m unsure which module you are using exactly, as <code>nn.TransfrmerEncoder</code> doesn’t seem to return a <code>dict</code> with outputs:</p>
<pre><code class="lang-python">encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
src = torch.rand(10, 32, 512)
out = transformer_encoder(src)
print(type(out))
# &lt;class 'torch.Tensor'&gt;
</code></pre>