Getting index out of self error

username: Manaswi_Mancha
cooked: <p>I am trying to do a forward pass on the LEDForConditionalGeneration with the PRIMER checkpoint, but I am getting an error. How do I fix this?</p>
<h2><a name="error-message-1" class="anchor" href="#error-message-1"></a>Error Message:</h2>
<p>IndexError                                Traceback (most recent call last)<br>
 in &lt;cell line: 12&gt;()<br>
38     global_attention_mask[input_ids == DOCSEP_TOKEN_ID] = 1<br>
39<br>
—&gt; 40     outputs = MODEL(input_ids) # &lt;---------------------------------------------------------------------------------------------- causing a bug<br>
41     # outputs = MODEL(input_ids=input_ids_all, global_attention_mask=global_attention_mask)<br>
42</p>
<p>13 frames<br>
/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)<br>
2231         # remove once script supports set_grad_enabled<br>
2232         <em>no_grad_embedding_renorm</em>(weight, input, max_norm, norm_type)<br>
 → 2233     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
2234<br>
2235</p>
<p>IndexError: index out of range in self</p>
<p>Model Architecture:<br>
LEDForConditionalGeneration(<br>
(led): LEDModel(<br>
(shared): Embedding(50266, 1024, padding_idx=1)<br>
(encoder): LEDEncoder(<br>
(embed_tokens): Embedding(50266, 1024, padding_idx=1)<br>
(embed_positions): LEDLearnedPositionalEmbedding(4096, 1024)<br>
(layers): ModuleList(<br>
(0-11): 12 x LEDEncoderLayer(<br>
(self_attn): LEDEncoderAttention(<br>
(longformer_self_attn): LEDEncoderSelfAttention(<br>
(query): Linear(in_features=1024, out_features=1024, bias=True)<br>
(key): Linear(in_features=1024, out_features=1024, bias=True)<br>
(value): Linear(in_features=1024, out_features=1024, bias=True)<br>
(query_global): Linear(in_features=1024, out_features=1024, bias=True)<br>
(key_global): Linear(in_features=1024, out_features=1024, bias=True)<br>
(value_global): Linear(in_features=1024, out_features=1024, bias=True)<br>
)<br>
(output): Linear(in_features=1024, out_features=1024, bias=True)<br>
)<br>
(self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
(activation_fn): GELUActivation()<br>
(fc1): Linear(in_features=1024, out_features=4096, bias=True)<br>
(fc2): Linear(in_features=4096, out_features=1024, bias=True)<br>
(final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
)<br>
)<br>
(layernorm_embedding): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
)<br>
(decoder): LEDDecoder(<br>
(embed_tokens): Embedding(50266, 1024, padding_idx=1)<br>
(embed_positions): LEDLearnedPositionalEmbedding(1024, 1024)<br>
(layers): ModuleList(<br>
(0-11): 12 x LEDDecoderLayer(<br>
(self_attn): LEDDecoderAttention(<br>
(k_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
(v_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
(q_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
(out_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
)<br>
(activation_fn): GELUActivation()<br>
(self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
(encoder_attn): LEDDecoderAttention(<br>
(k_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
(v_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
(q_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
(out_proj): Linear(in_features=1024, out_features=1024, bias=True)<br>
)<br>
(encoder_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
(fc1): Linear(in_features=1024, out_features=4096, bias=True)<br>
(fc2): Linear(in_features=4096, out_features=1024, bias=True)<br>
(final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
)<br>
)<br>
(layernorm_embedding): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)<br>
)<br>
)<br>
(lm_head): Linear(in_features=1024, out_features=50266, bias=False)<br>
)</p>
<p>Input Shape:<br>
tensor([[    0, 43070, 26945,  …,     4, 50265,     2],<br>
[    0, 34543,    35,  …,     1,     1,     1]])<br>
torch.Size([2, 1047])</p>

username: ptrblck
cooked: <p>The error is raised in an embedding operation so check the min./max. values in the index tensor as well as the shape of the <code>weight</code> matrix.</p>