Problem in finetuning the pretrained model

Ahtsham_Manzoor: <p>Hi,<br>
I Have been working on fine-tuning a pre-trained language model, however, in the training part, I am receiving this error.</p>
<blockquote>
<p>File “training.py”, line 296, in <br>
record_loss, perplexity = train_one_iter(batch, update_count, fp16=False)<br>
File “training.py”, line 105, in train_one_iter<br>
logits, past = model_A(dial_turn_inputs, past=past)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py”, line 528, in forward<br>
head_mask=head_mask)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py”, line 418, in forward<br>
inputs_embeds = self.wte(input_ids)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py”, line 114, in forward<br>
self.norm_type, self.scale_grad_by_freq, self.sparse)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py”, line 1467, in embedding<br>
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)<br>
RuntimeError: index out of range: Tried to access index 50297 out of table with 50296 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237</p>
</blockquote>
<p>I have checked the model weights’ size and vocab size. It tuned out like this respectively.<br>
<em>torch.Size([50297, 768])</em><br>
<em>50297</em></p>
<p><strong>Here is my code:</strong></p>
<pre><code class="lang-auto"># continue training
# Initialize models
model_A_states, model_B_states = torch.load(inspired_data_path+"model_weight.pth",map_location=torch.device('cpu'))
print('model weights loaded')
#print(len(model_A_states))
added_embedding_plot = [model_A_states["transformer.wte.weight"][[tokenizer.encode("movie plot")], :].mean(1)]
added_embedding_plot = torch.cat(added_embedding_plot, 0)
added_embedding_sep = [model_A_states["transformer.wte.weight"][[tokenizer.encode("separate")], :].mean(1)]
added_embedding_sep = torch.cat(added_embedding_sep, 0)

new_weight = torch.cat([model_A_states["transformer.wte.weight"], added_embedding_plot, added_embedding_sep], 0)
print(new_weight.size())
#print(new_weight.shape())
model_A_states["transformer.wte.weight"] = new_weight
model_B_states["transformer.wte.weight"] = new_weight
model_A_states["lm_head.weight"] = new_weight
model_B_states["lm_head.weight"] = new_weight

config = GPT2Config()
config.vocab_size = model_A_states["transformer.wte.weight"].shape[0]
#config.vocab_size = config.vocab_size-1
print(config.vocab_size)
model_A = GPT2LMHeadModel(config)
model_B = GPT2LMHeadModel(config)

model_A.load_state_dict(model_A_states)
model_B.load_state_dict(model_B_states)

device = torch.device("cpu")
#print(device)
model_A = model_A.to(device)
model_B = model_B.to(device)
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you please assist in this regard?</p>

InnovArul: <aside class="quote no-group" data-username="Ahtsham_Manzoor" data-post="1" data-topic="139874">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahtsham_manzoor/48/32203_2.png" class="avatar"> Ahtsham_Manzoor:</div>
<blockquote>
<p>RuntimeError: index out of range: Tried to access index 50297 out of table with 50296 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237</p>
</blockquote>
</aside>
<p>I see that there are 2 things confusing here.</p>
<ol>
<li>
<p>In your debug prints, the embedding seems to have 50297 rows. But error message says it has 50296 rows?</p>
</li>
<li>
<p>Accessing index 50297 will not be possible even in the embedding with size <code>50297 x D</code>. You can access only from <code> 0 to 50296</code>. Can you check along these lines?</p>
</li>
</ol>