RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/generic/THCTensorCopy.c:21

Aki58: <p>Hello, Getting this error in the middle of the training. I have tried training several times but the same error comes at different steps.</p>
<pre><code class="lang-auto">/opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/THCTensorIndex.cu:360: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [5,0,0], thread: [96,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/THCTensorIndex.cu:360: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [5,0,0], thread: [97,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
.
.
.

/opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/THCTensorIndex.cu:360: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [6,0,0], thread: [31,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/generic/THCTensorCopy.c line=21 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "train.py", line 40, in &lt;module&gt;
    main(opt)
  File "train.py", line 27, in main
    single_main(opt)
  File "/home/aki/OpenNMT-py/onmt/train_single.py", line 262, in main
    opt.valid_steps)
  File "/home/aki/OpenNMT-py/onmt/trainer.py", line 223, in train
    report_stats)
  File "/home/aki/OpenNMT-py/onmt/trainer.py", line 384, in _gradient_accumulation
    dec_state)
  File "/home/aki/anaconda3/envs/zeroshot/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aki/OpenNMT-py/onmt/models/model.py", line 75, in forward
    memory_lengths=lengths)
  File "/home/aki/anaconda3/envs/zeroshot/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aki/OpenNMT-py/onmt/decoders/decoder.py", line 139, in forward
    tgt, memory_bank, state, memory_lengths=memory_lengths)
  File "/home/aki/OpenNMT-py/onmt/decoders/decoder.py", line 350, in _run_forward_pass
    memory_lengths=memory_lengths)
  File "/home/aki/anaconda3/envs/zeroshot/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aki/OpenNMT-py/onmt/modules/global_attention.py", line 181, in forward
    mask = sequence_mask(memory_lengths, max_len=align.size(-1))
  File "/home/aki/OpenNMT-py/onmt/utils/misc.py", line 23, in sequence_mask
    .type_as(lengths)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/THC/generic/THCTensorCopy.c:21

</code></pre>

vdw: <p>So far I got this error for two reasons:</p>
<ul>
<li>In case of classification, my labels where not in <code>[0...(label_size-1)]</code>
</li>
<li>My input sequence contained indexes there where not in the embedding layer.</li>
</ul>
<p>Since you don’t say what you’re doing or show any code, one can only guess. Does it work on CPU? And if not, what’s the error there?</p>

Aki58: <p>I am working on a multilingual nmt model using opennmt. Getting the following error when running on cpu. As you said its index out of range error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "train.py", line 40, in &lt;module&gt;

    main(opt)

  File "train.py", line 27, in main

    single_main(opt)

  File "/home/gamut/OpenNMT-py/onmt/train_single.py", line 262, in main

    opt.valid_steps)

  File "/home/gamut/OpenNMT-py/onmt/trainer.py", line 223, in train

    report_stats)

  File "/home/gamut/OpenNMT-py/onmt/trainer.py", line 384, in _gradient_accumulation

    dec_state)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/gamut/OpenNMT-py/onmt/models/model.py", line 61, in forward

    enc_final, memory_bank = encoder(src, lengths)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/gamut/OpenNMT-py/onmt/encoders/rnn_encoder.py", line 57, in forward

    emb = self.embeddings(src)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/gamut/OpenNMT-py/onmt/modules/embeddings.py", line 205, in forward

    source = self.make_embedding(source)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward

    input = module(input)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/gamut/OpenNMT-py/onmt/modules/util_class.py", line 43, in forward

    outputs = [f(x) for f, x in zip(self, inputs_)]

  File "/home/gamut/OpenNMT-py/onmt/modules/util_class.py", line 43, in &lt;listcomp&gt;

    outputs = [f(x) for f, x in zip(self, inputs_)]

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 108, in forward

    self.norm_type, self.scale_grad_by_freq, self.sparse)

  File "/home/gamut/anaconda3/envs/zeroshotpy/lib/python3.6/site-packages/torch/nn/functional.py", line 1076, in embedding

    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

RuntimeError: index out of range at /opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/TH/generic/THTensorMath.c:343
</code></pre>

vdw: <p>I best guess would be that your input sequences contain indexes that not match with the embedding layer. The embedding layer has a shape of <code>(vocabulary_size, embedding_dim)</code>, e.g., (20000, 300). This means your input sequences can only contain values in the range <code>[0..19999]</code>. You might want to check if any value in your input sequences is 20000 or higher.</p>

n0obcoder: <p>Yeah it was wrong embedding index in my case. Thanks <a class="mention" href="/u/vdw">@vdw</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Melody2333: <p>Thank you very much!!! I solve my problem <img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"></p>