Runetime Error: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/THCTensorMathCompare.cuh:82

username: phinum
cooked: <p>I’m training a model; after running for 50k to 0.5 million epochs, it gives me this error, again and again! I’ve attached the terminal error and also the piece of code where the error occurs. Kindly help me!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9163b848f1893f22be879245be28a46a47be387.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9163b848f1893f22be879245be28a46a47be387" title="Screenshot 2021-04-01 at 10.52.42 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9163b848f1893f22be879245be28a46a47be387_2_690x379.png" alt="Screenshot 2021-04-01 at 10.52.42 PM" data-base62-sha1="zxwwqxTwDu4HUsqSBg8qrFl4IkL" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9163b848f1893f22be879245be28a46a47be387_2_690x379.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9163b848f1893f22be879245be28a46a47be387_2_1035x568.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9163b848f1893f22be879245be28a46a47be387.png 2x" data-dominant-color="393A3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-01 at 10.52.42 PM</span><span class="informations">1178×648 294 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The code is attached below:</p>
<p>def train_batch_MLE(self, enc_out, enc_hidden, enc_padding_mask, ct_e, extra_zeros, enc_batch_extend_vocab, batch):<br>
‘’’ Calculate Negative Log Likelihood Loss for the given batch. In order to reduce exposure bias,<br>
pass the previous generated token as input with a probability of 0.25 instead of ground truth label<br>
Args:<br>
:param enc_out: Outputs of the encoder for all time steps (batch_size, length_input_sequence, 2*hidden_size)<br>
:param enc_hidden: Tuple containing final hidden state &amp; cell state of encoder. Shape of h &amp; c: (batch_size, hidden_size)<br>
:param enc_padding_mask: Mask for encoder input; Tensor of size (batch_size, length_input_sequence) with values of 0 for pad tokens &amp; 1 for others<br>
:param ct_e: encoder context vector for time_step=0 (eq 5 in <a href="https://arxiv.org/pdf/1705.04304.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1705.04304.pdf</a>)<br>
:param extra_zeros: Tensor used to extend vocab distribution for pointer mechanism<br>
:param enc_batch_extend_vocab: Input batch that stores OOV ids<br>
:param batch: batch object<br>
‘’’<br>
dec_batch, max_dec_len, dec_lens, target_batch = get_dec_data(batch)                        <span class="hashtag-raw">#Get</span> input and target batchs for training decoder<br>
step_losses = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<pre><code>    copy_loss = []
    s_t = (enc_hidden[0], enc_hidden[1])                                                        #Decoder hidden states
    x_t = get_cuda(T.LongTensor(len(enc_out)).fill_(self.start_id))                             #Input to the decoder
    prev_s = None                                                                               #Used for intra-decoder attention (section 2.2 in https://arxiv.org/pdf/1705.04304.pdf)
    sum_temporal_srcs = None                                                                    #Used for intra-temporal attention (section 2.1 in https://arxiv.org/pdf/1705.04304.pdf)
    for t in range(min(max_dec_len, config.max_dec_steps)):
        use_gound_truth = get_cuda((T.rand(len(enc_out)) &gt; 0.25)).long()                        #Probabilities indicating whether to use ground truth labels instead of previous decoded tokens
        x_t = use_gound_truth * dec_batch[:, t] + (1 - use_gound_truth) * x_t                   #Select decoder input based on use_ground_truth probabilities
        x_t = self.model.embeds(x_t)
        final_dist, s_t, ct_e, sum_temporal_srcs, prev_s = self.model.decoder(x_t, s_t, enc_out, enc_padding_mask, ct_e, extra_zeros, enc_batch_extend_vocab, sum_temporal_srcs, prev_s)
        target = target_batch[:, t]
        log_probs = T.log(final_dist + config.eps)

        step_loss = F.nll_loss(log_probs, target, reduction="none", ignore_index=self.pad_id)
        # except:
        #     print('')
        step_losses.append(step_loss)
        x_t = T.multinomial(final_dist, 1).squeeze()                                            #Sample words from final distribution which can be used as input in next time step
        is_oov = (x_t &gt;= config.vocab_size).long()                                              #Mask indicating whether sampled word is OOV
        x_t = (1 - is_oov) * x_t.detach() + (is_oov) * self.unk_id                              #Replace OOVs with [UNK] token
        # print("ssss",x_t)
        #######
        copy_loss.append(1-is_oov)
    losses = T.sum(T.stack(step_losses, 1), 1)                                                  #unnormalized losses for each example in the batch; (batch_size)
    batch_avg_loss = losses / dec_lens                                                          #Normalized losses; (batch_size)
    mle_loss = T.mean(batch_avg_loss)                                                           #Average batch loss
    # copy_losses = T.sum(T.stack(copy_loss, 1), 1)  # unnormalized losses for each example in the batch; (batch_size)
    # batch_avg_copy_loss = copy_losses.float() / dec_lens  # Normalized losses; (batch_size)
    # avg_copy_loss = T.mean(batch_avg_copy_loss)
    # mle_loss += avg_copy_loss*5
    return mle_loss
</code></pre>

username: ptrblck
cooked: <p>Based on the error message it seems that an internal check failed for the <code>sampleMultinomialOnce</code> kernel.<br>
However, I cannot find the <code>THCNumerics&lt;T&gt;::ge(val, zero)</code> operationand guess that you might be hitting one of <a href="https://github.com/pytorch/pytorch/blob/593295daacc28ed016f1d7456bd2ecd21c786544/aten/src/ATen/native/cuda/MultinomialKernel.cu#L37-L47">these checks</a>. Which PyTorch version are you using and could you update to the latest release in case you are using an older one?</p>

username: phinum
cooked: <p>I was earlier using the updated one, but I read through some other articles on the error, and they suggested using PyTorch 1.1.0. However, using the updated PyTorch also throws the same error!</p>

username: phinum
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7afe5c101a0b885cbe1121eb540a5f20651138b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/7afe5c101a0b885cbe1121eb540a5f20651138b8" title="Screenshot 2021-04-09 at 12.48.33 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7afe5c101a0b885cbe1121eb540a5f20651138b8_2_690x401.png" alt="Screenshot 2021-04-09 at 12.48.33 PM" data-base62-sha1="hy3di254LH6EQwXZwfYUL2Laf5C" width="690" height="401" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7afe5c101a0b885cbe1121eb540a5f20651138b8_2_690x401.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7afe5c101a0b885cbe1121eb540a5f20651138b8_2_1035x601.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7afe5c101a0b885cbe1121eb540a5f20651138b8.png 2x" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-09 at 12.48.33 PM</span><span class="informations">1138×662 276 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I reran it and got another error similar to the old one!</p>

username: ptrblck
cooked: <p>Could you check the inputs to the <code>multinomial</code> call and make sure they contain valid values?</p>