[solved] Assertion `srcIndex < srcSelectDimSize` failed on GPU for `torch.cat()`

handesy: <p>Hi, I encountered the following assertion error when running my code on GPU (things are fine on CPU):</p>
<pre><code>/b/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [179,0,0], thread: [0,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/b/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [179,0,0], thread: [1,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
/b/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [179,0,0], thread: [2,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu line=226 error=59 : device-side assert triggered
Traceback (most recent call last):
....
x = torch.cat([y_tm1_embed.squeeze(0), ctx_tm1], 1)
File "torch/autograd/variable.py", line 836, in cat
return Concat(dim)(*iterable)
File "torch/autograd/_functions/tensor.py", line 310, in forward
return torch.cat(inputs, self.dim)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /b/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu:226
</code></pre>
<p>I used the flag <code>CUDA_LAUNCH_BLOCKING=1</code>. The full code is a simple attention-based encoder-decoder, with <code>y_tm1_embed</code> being the embedding of the previous word, and <code>ctx_tm1</code> is the previous context vector, which is initialized by:</p>
<pre><code>ctx_tm1 = Variable(torch.zeros(batch_size, self.args.hidden_size * 2), requires_grad=False)
if self.args.cuda:
    ctx_tm1 = ctx_tm1.cuda()
</code></pre>
<p>Any suggestions? Thanks!</p>

smth: <p>you are giving an out of bounds index somewhere.</p>
<p>Can you reproduce this error with a small snippet?</p>
<p>Usually these device-side asserts are easier to debug if you run the same code on CPU (i.e. without .cuda()) and you know right away what the out of bounds indices are and where they’re coming from.</p>

handesy: <p>Hi smth, thank you for your reply! The weird thing is that when running the <em>same</em> code on CPU (i.e., without <code>.cuda()</code>) everything is fine. I am trying to reproduce this with a small snippet. May I know if there’re any other possibilities that could cause this error?</p>

handesy: <p>update: problem solved. This is due to an out of bounds index in the embedding matrix. Thanks for the help!</p>

py0ker: <p>hello, excuse me, I’d like to know how did you find it? I face the same problem, but I don’t know why it could be out of index bound. thank you!</p>

ptrblck: <p>If you are seeing this error using an <code>nn.Embedding</code> layer, you might add a print statement which shows the min and max values for each input. Some batches might have an out of bounds index.<br>
Once you find, the erroneous batch you should have a look how it was created so that you can fix this error.</p>

Gaurav_Koradiya: <p>Excatly, I solved my problem. Thank u very much.</p>

Chang_Liao: <p>How you guys solve this problem? I am facing the same issue. It’s running well on CPU but it doesn’t work on GPU.</p>

verified.human: <p>I think the PyTorch error messages should be improved here. Having the wrong number of classes for nn.Embedding throws a bunch of C++ errors and returns CUDNN_STATUS_NOT_INITIALIZED on the latest version. Quite hard to debug this problem given these non-informative error messages.</p>

ptrblck: <p>CUDA errors might be sometimes cryptic, so I generally recommend to debug the code on the CPU, if possible. If that’s not possible, I would try to execute the script via:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>to get the right line of code which raised the error in the stack trace.</p>

addisonklinke: <p>For others stumbling on this thread, be careful to choose a positive index (i.e. 0 instead of -1) for padding sequences as input to an embedding layer. Even if you specify the negative index in the embedding constructor, you will still get a runtime error on both CPU and GPU</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

emb = nn.Embedding(20, 100, padding_idx=-1)
inp = torch.tensor([5, 2, 7, 12, 3])
bad_padding = torch.cat((inp, torch.tensor([-1] * 3)))
good_padding = torch.cat((inp, torch.tensor([0] * 3)))
out = emb(good_padding)
out = emb(bad_padding)  # RuntimeError
</code></pre>

Andrew_Brockman: <p>Was nice to see an explanation for why trying with CPU was a good start:</p>
<p>In my case I now see:</p>
<blockquote>
<p>“RuntimeError: index out of range: Tried to access index 20000 out of table with 19999 rows. at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418”</p>
</blockquote>
<p>When running an LSTM as such:</p>
<pre><code class="lang-auto">class LSTMClassifier(nn.Module):

	"""
	USAGE:
		model = LSTMClassifier( HIDDEN_SIZE, INPUT_SIZE, VOCAB_SIZE )
		model.to( DEVICE )
	"""

	# initial setup of the RNN, ..
	# .. given user parameters, notice we have [at least] 3 layers: 
	# 		1. embedding, 
	#  		2. encoder [x N_LAYERS], 
	# 		3. predictor

	def __init__(self, hidden_size, embedding_dim, vocab_size, n_lstm_layers):  # bespoke @ANDY:@DEBUG:@1818 this may be causing a bug, below is default
	#def __init__(self, hidden_size, embedding_dim, vocab_size):  # ^
		super(LSTMClassifier, self).__init__()

		#self.embedding 	= nn.Embedding(vocab_size, embedding_dim) # @ANDY:@DEBUG:@1818 this leads to error: "RuntimeError: index out of range: Tried to access index 20000 out of table with 19999 rows. at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418", so a crude fix is the line below
		self.embedding 	= nn.Embedding(vocab_size, embedding_dim) # see ^
		self.encoder 	= nn.LSTM( 	input_size  = embedding_dim, 
									hidden_size = hidden_size,
								#	num_layers = 2) 
									num_layers  = n_lstm_layers)     # @ANDY:@DEBUG:@1818 this leads to error: "RuntimeError: index out of range: Tried to access index 20000 out of table with 19999 rows. at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418", so a crude fix is the line below
		#self.predictor 	= nn.Linear(hidden_size, N_OUT_CLASSES )  # bespoke @ANDY:@DEBUG:@1818 this may be causing a bug, below is default // arg1 = size of input, arg2 = number of output classes, see: https://pytorch.org/docs/stable/nn.html (CTRL+F: "nn.Linear")
		self.predictor 	= nn.Linear(hidden_size, 2 )  # ^ // arg1 = size of input, arg2 = number of output classes, see: https://pytorch.org/docs/stable/nn.html (CTRL+F: "nn.Linear")

		#self.flatten_parameters()


	# This is how the model makes predictions, 
	# .. given an input (training: u/ later to calculate losses &amp; backprops )
	def forward( self, seq ):

		output, (hidden,_) 	= self.encoder(self.embedding(seq))
		preds 				= self.predictor(hidden.squeeze(0))  # e.g. remove 1D entries from the shape of an array, see: https://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html

		return preds
</code></pre>
<hr>
<p>There are two places this 20000 is used:</p>
<p>Instantiating the model:</p>
<p><code>lstm_classifier = LSTMClassifier( HIDDEN_SIZE=150, INPUT_SIZE=300, vocab_size=20000, N_LAYERS=4 )</code></p>
<p>And  when making the vocabulary, in the dataset generation phase:</p>
<pre><code class="lang-auto">VOCAB_SIZE = 20000
vocab_size = VOCAB_SIZE  # to restrict the vocabulary, which saves memory
TWEET.build_vocab(train, max_size = vocab_size)
</code></pre>
<p>Any help would greatly be appreciated!</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji only-emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

Andrew_Brockman: <p>I’ve come to learn what was going wrong…</p>
<p>When you build a vocabulary using <code>torchtext.data.Label</code> class:</p>
<pre><code class="lang-auto">from torchtext import data
print("Building vocabulary...")
TWEET = data.Field( tokenize="spacy", lower=True ) # https://spacy.io/usage/
vocab_size = 20000  # to restrict the vocabulary, which saves memory
TWEET.build_vocab(train, max_size = vocab_size)
</code></pre>
<p>Since I told it 20000 is the maximum vocab size, I would have expected the maximum sequence input would be 19999th element.</p>
<p>BUT</p>
<p>when you ask for the length of this vocabulary, it is always +2 from the maximum vocabulary size you ASKED it to restrict the vocab to:</p>
<pre><code class="lang-auto">In [12]: len(TWEET.vocab)                                                 
Out[12]: 20002
</code></pre>
<p>This is because two additional tokens are added, one of then is &lt;unk&gt; for unknown, etc.</p>
<p>So you need to tell your classifier this:</p>
<p><code>self.embedding 	= nn.Embedding(vocab_size +2, embedding_dim)</code></p>
<p>Full code for the LSTM class:</p>
<pre><code class="lang-auto">class LSTMClassifier(nn.Module):

	"""
	USAGE:
		model = LSTMClassifier( HIDDEN_SIZE, INPUT_SIZE, VOCAB_SIZE )
		model.to( DEVICE )
	"""

	# initial setup of the RNN, ..
	# .. given user parameters, notice we have [at least] 3 layers: 
	# 		1. embedding, 
	#  		2. encoder [x N_LAYERS], 
	# 		3. predictor

	def __init__(self, hidden_size, embedding_dim, vocab_size, n_lstm_layers):  # bespoke @ANDY:@DEBUG:@1818 this may be causing a bug, below is default
	#def __init__(self, hidden_size, embedding_dim, vocab_size):  # ^
		super(LSTMClassifier, self).__init__()

		#self.embedding 	= nn.Embedding(vocab_size, embedding_dim) # @ANDY:@DEBUG:@1818 this leads to error: "RuntimeError: index out of range: Tried to access index 20000 out of table with 19999 rows. at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418", so a crude fix is the line below
		self.embedding 	= nn.Embedding(vocab_size +2, embedding_dim) # see ^
		self.encoder 	= nn.LSTM( 	input_size  = embedding_dim, 
									hidden_size = hidden_size,
								#	num_layers = 2) 
									num_layers  = n_lstm_layers)     # @ANDY:@DEBUG:@1818 this leads to error: "RuntimeError: index out of range: Tried to access index 20000 out of table with 19999 rows. at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418", so a crude fix is the line below
		#self.predictor 	= nn.Linear(hidden_size, N_OUT_CLASSES )  # bespoke @ANDY:@DEBUG:@1818 this may be causing a bug, below is default // arg1 = size of input, arg2 = number of output classes, see: https://pytorch.org/docs/stable/nn.html (CTRL+F: "nn.Linear")
		self.predictor 	= nn.Linear(hidden_size, 2 )  # ^ // arg1 = size of input, arg2 = number of output classes, see: https://pytorch.org/docs/stable/nn.html (CTRL+F: "nn.Linear")

		#self.flatten_parameters()


	# This is how the model makes predictions, 
	# .. given an input (training: u/ later to calculate losses &amp; backprops )
	def forward( self, seq ):

		try:
			output, (hidden,_) 	= self.encoder(self.embedding(seq))
			preds 				= self.predictor(hidden.squeeze(0))  # e.g. remove 1D entries from the shape of an array, see: https://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html

		except:
			[max(seq[i]) for i in range(seq.shape[0])]
			pdb.set_trace()

		return preds
</code></pre>

attardi: <p>I run into the same error when using a model from Huggingface transformers (BertModel). The code runs fine on CPU:</p>
<p>/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexT<br>
ype&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, Ds<br>
tDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [145,0,0], thread: [95,0,0] Assertion <code>srcIndex &lt; srcS electDimSize</code> failed.<br>
Traceback (most recent call last):<br>
File “run.py”, line 58, in <br>
cmd(args)<br>
File “/project/piqasso/tools/biaffine-parser/parser/cmds/train.py”, line 82, in <strong>call</strong><br>
self.train(train.loader)<br>
File “/project/piqasso/tools/biaffine-parser/parser/cmds/cmd.py”, line 83, in train<br>
arc_scores, rel_scores = self.model(words, feats)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/project/piqasso/tools/biaffine-parser/parser/model.py”, line 90, in forward<br>
feat_embed = self.feat_embed(*feats)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/project/piqasso/tools/biaffine-parser/parser/modules/bert.py”, line 43, in forward<br>
bert = bert[bert_mask].split(bert_lens[mask].tolist())<br>
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered</p>
<p>How can I figure out what is the culprit?</p>

LeonardoEmili: <p>I got the same error as <a class="mention" href="/u/attardi">@attardi</a> when running on GPU the ‘bert-large-cased’ model from HuggingFace. Is anybody aware of a solution for this problem?</p>

LeonardoEmili: <p>By chance did you find you what was causing the problem?</p>

stas: <p>Possibly related to <a href="https://github.com/pytorch/pytorch/issues/46020" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/46020</a></p>

LeonardoEmili: <p>It seems that there were some input instances there were exceeding the maximum number of workpiece embeddings that BERT could handle. What I did is simply check the dimensions of the input batches and pass as input to the model only those that were not exceeding that limit.</p>

david.waterworth: <p>I’ve been having this issue with RoBERTa. The problem was the max_position_embeddings parameter must be larger than the max_seq_length, otherwise the position embedding can generate indices &gt; max_position_embeddings</p>

widdiot: <p>Could also be due to some issue with the vocab file. As it was in my case</p>