Finetuning GPT2 produces IndexError: index out of range in self error
Author: aclifton314
Pull request: no
Labels: name: wontfix

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 3.0.2
Tokenizers: 0.8.1rc1
Python: 3.7.6
Pretrained Model: GPT2
Pretrained Tokenizer: GPT2

## Question
I am finetuning the pretrained GPT2 model on my dataset, using a custom defined loss function. I am getting this error below, but don't really have an idea as to what the issue might be. Here is the error:
```python
Traceback (most recent call last):
  File "run_finetune_gpt2.py", line 143, in <module>
    main()

  File "run_finetune_gpt2.py", line 130, in main
    trainer.train()

  File "/usr/local/anaconda3/lib/python3.6/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/transformers/trainer.py", line 622, in _training_step
    outputs = model(**inputs)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)

  File "/data/ric-2020/textgen/finetune_gpt2.py", line 90, in forward
    top_p=0.95

  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/transformers/generation_utils.py", line 483, in generate
    model_specific_kwargs=model_specific_kwargs

  File "/usr/local/anaconda3/lib/python3.6/site-packages/transformers/generation_utils.py", line 524, in _generate_no_beam_search
    outputs = self.generate_text_while_finetuning(**model_inputs)

  File "/data/ric-2020/textgen/finetune_gpt2.py", line 44, in generate_text_while_finetuning
    output_hidden_states=output_hidden_states,

  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 470, in forward
    position_embeds = self.wpe(position_ids)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)

  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

IndexError: index out of range in self
```
I save many of the inputs that go into the training process:
```python
TRAINING ARGS
output_dir: models/textgen/out
overwrite_output_dir: False
do_train: True
do_eval: False
do_predict: False
evaluate_during_training: False
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
learning_rate: 5e-05
weight_decay: 0.0
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 3.0
max_steps: -1
warmup_steps: 0
logging_dir: models/textgen/logs
logging_first_step: False
logging_steps: 500
save_steps: 500
save_total_limit: None
no_cuda: False
seed: 42
fp16: False
fp16_opt_level: O1
local_rank: -1
tpu_num_cores: None
tpu_metrics_debug: False
debug: False
dataloader_drop_last: False
eval_steps: 1000
past_index: -1
---------------------------------------------
MODEL ARGS
activation_function: gelu_new
architectures: ['GPT2LMHeadModel']
attn_pdrop: 0.1
bos_token_id: 50256
embd_pdrop: 0.1
eos_token_id: 50256
initializer_range: 0.02
layer_norm_epsilon: 1e-05
model_type: gpt2
n_ctx: 1024
n_embd: 768
n_head: 12
n_layer: 12
n_positions: 1024
resid_pdrop: 0.1
summary_activation: None
summary_first_dropout: 0.1
summary_proj_to_labels: True
summary_type: cls_index
summary_use_proj: True
task_specific_params: {'text-generation': {'do_sample': True, 'max_length': 50}}
vocab_size: 50257
---------------------------------------------
TRAINER ARGS
args: TrainingArguments(
output_dir='models/textgen/out', 
overwrite_output_dir=False, 
do_train='True', 
do_eval=False, 
do_predict=False, 
evaluate_during_training=False, 
per_device_train_batch_size=8, 
per_device_eval_batch_size=8, 
per_gpu_train_batch_size=None, 
per_gpu_eval_batch_size=None, 
gradient_accumulation_steps=1, 
learning_rate=5e-05, 
weight_decay=0.0, 
adam_epsilon=1e-08, 
max_grad_norm=1.0, 
num_train_epochs=3.0, 
max_steps=-1, 
warmup_steps=0, 
logging_dir='models/textgen/logs', 
logging_first_step=False, 
logging_steps=500, 
save_steps=500, 
save_total_limit=None, 
no_cuda=False, 
seed=42, 
fp16=False, 
fp16_opt_level='O1', 
local_rank=-1, 
tpu_num_cores=None, 
tpu_metrics_debug=False, 
debug=False, 
dataloader_drop_last=False, 
eval_steps=1000, 
past_index=-1)

data_collator: <function sd_data_collator at 0x7ffaba8f8e18>
train_dataset: <custom_dataset.SDAbstractsDataset object at 0x7ffa18c8c400>
eval_dataset: None
compute_metrics: None
prediction_loss_only: False
optimizers: None
tb_writer: <torch.utils.tensorboard.writer.SummaryWriter object at 0x7ff9f79e45c0>
```

The interesting thing is that the dataset that generates this error consists of 5426 examples. However, if I use another dataset that has 1024 examples, the training completes. So I'm not really sure what's going on. I'm happy to provide any other information. Thanks in advance for the help!


Comments:
looks similar to #6192

Hey @aclifton314, 

Could you post a complete code example here so that we can reproduce the error?

@patil-suraj thanks for your reply. I checked the vocab and input embed size and they are the same:
```python
model.transformer.wte.weight.shape[0] = 50257
len(model.tokenizer) = 50257
```
The OP in the issue you linked to figured out that, for him, the issue was coming from the cross entropy loss:

> I figured it out, it was due to the change in ignore_index=-100 instead of -1 in the cross entropy loss which was causing the issue. I'll close this.

For me, I calculate a loss with respect to an n-grams model I made so I'm not entirely sure that is the source of the problem. What's more, doing a test on the training with a dataset that has 1024 examples completes, while using a dataset with 5426 examples produces the `IndexError: index out of range in self` error.

@patrickvonplaten sure, let me see what I can put together.

@patrickvonplaten I'm having problems coming up with some sample code that reproduces the problem exactly. Since I am generating text during the training process, my hunch is that I generate a sequence that is longer than what GPT2 allows. It looks like another user posted some code that might serve as a sample to reproduce the error here: https://github.com/huggingface/transformers/issues/6599

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.