Embedding Runtime Error Index Out of Range

eprox: <p>Hi,<br>
I am getting the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 549, in &lt;module&gt;
    task.execute()
  File "train.py", line 271, in execute
    train_loss, train_acc = self.train(phase)
  File "train.py", line 147, in train
    answer = self.model(batch)
  File "/home/eprox/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eprox/uom/clnli/clnli-code/rnn-impl/models/bilstm.py", line 69, in forward
    premise_embed = self.embedding(batch.premise)
  File "/home/eprox/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eprox/.local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/eprox/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 12209 out of table with 4979 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
</code></pre>
<p>Module code:</p>
<pre><code class="lang-auto">class bilstm(nn.Module):
	def __init__(self, sent_embed, d_hidden, dp_ratio, device):
		super(BiLSTM, self).__init__()
		self.hidden_size = d_hidden
		self.directions = 2
		self.num_layers = 2
		self.concat = 4
		self.device = device
		self.out_dim = 3
		self.embedding = nn.Embedding.from_pretrained(sent_embed)
		print('SentEmbed:'); print(sent_embed.shape) 
		# Output: torch.Size([4980, 1024])
		self.embed_dim = sent_embed_dim
		
		print(self.embedding) # Output: Embedding(4980, 1024)
		print('EMBED DIM IS: {}'.format(self.embed_dim)) # EMBED DIM IS: 1024
		self.projection = nn.Linear(self.embed_dim, self.hidden_size)
		self.lstm = nn.LSTM(self.hidden_size, self.hidden_size, self.num_layers,
									bidirectional = True, batch_first = True, dropout = dp_ratio)
		self.relu = nn.LeakyReLU()
		self.dropout = nn.Dropout(p = dp_ratio)

		self.lin1 = nn.Linear(self.hidden_size * self.directions * self.concat, self.hidden_size)
		self.lin2 = nn.Linear(self.hidden_size, self.hidden_size)
		self.lin3 = nn.Linear(self.hidden_size, self.out_dim)

		for lin in [self.lin1, self.lin2, self.lin3]:
			nn.init.xavier_uniform_(lin.weight)
			nn.init.zeros_(lin.bias)

		self.out = nn.Sequential(
			self.lin1,
			self.relu,
			self.dropout,
			self.lin2,
			self.relu,
			self.dropout,
			self.lin3
		)

	def forward(self, batch):
		print('*premiseshape:'); print(batch.premise.shape)
		# Output: torch.Size([128, 35]
		print('*embeddingshape:'); print(self.embedding)
		# Output: Embedding(4980, 1024)
		print('*projectionshape:'); print(self.projection)
		# Output: Linear(in_features=1024, out_features=100, bias=True)
		premise_embed = self.embedding(batch.premise)
		hypothesis_embed = self.embedding(batch.hypothesis)

		premise_proj = self.relu(self.projection(premise_embed))
		hypothesis_proj = self.relu(self.projection(hypothesis_embed))

		h0 = c0 = torch.tensor([]).new_zeros((self.num_layers * self.directions, batch.batch_size, self.hidden_size)).to(self.device)

		_, (premise_ht, _) = self.lstm(premise_proj, (h0, c0))
		_, (hypothesis_ht, _) = self.lstm(hypothesis_proj, (h0, c0))

		premise = premise_ht[-2:].transpose(0, 1).contiguous().view(batch.batch_size, -1)
		hypothesis = hypothesis_ht[-2:].transpose(0, 1).contiguous().view(batch.batch_size, -1)

		combined = torch.cat((premise, hypothesis, torch.abs(premise - hypothesis), premise * hypothesis), 1)
		return self.out(combined)
</code></pre>
<p>Based on the outputs in the comments, it seems like the embedding and the sentence embedding are the correct size. So I cannot understand why it is trying to access the index 12209.<br>
Could someone please advise me on this?</p>

ptrblck: <p>Based on the error message it seems that the input to the embedding contains an index of <code>12209</code>, while the number of embeddings is set to <code>4949</code>.<br>
Could you check it via <code>print(batch.premise.max())</code> and make sure the indices are in the valid range (<code>[0, num_embeddings-1]</code>)?</p>

jianlue_zhang: <p>hi ,i also meet this problem when i fine tuned roberta model. but i still do not know how to resolve it after seeing your solution repeatedly.<br>
it shows this problem. but i do not understand your solution. do you mean that every number in the input_ids after convert_token_to_id needs to be smaller than the embedding size?</p>

ptrblck: <p>I’m unsure what “embedding size” means in this context, but the input to <code>nn.Embedding</code> has to be in the range <code>[0, num_embeddings-1]</code> while <code>embedding_dim</code> can be chosen to a desired value and will be used as the feature vector dimension for each dense output tensor.<br>
Here is a small code snippet showing the issue:</p>
<pre><code class="lang-python">num_embeddings = 100
embedding_dim = 10
emb = nn.Embedding(num_embeddings=num_embeddings, embedding_dim=embedding_dim)
x = torch.randint(0, num_embeddings, (64, 1))

print(x.min(), x.max())
# &gt; tensor(1) tensor(99)

out = emb(x) # works since x is in [0, num_embeddings-1]

x[0] = 100
out = emb(x)
# &gt; IndexError: index out of range in self
# fails because x is out of range
</code></pre>