`run_glue.py` fails with models `bert-base-cased`, `distil-bert-cased`, others
Author: jxmorris12
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `bert-base-cased`

Language I am using the model on (English, Chinese ...): `English`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. download GLUE data
2. run `python run_glue.py --model_name_or_path bert-base-cased`

Error message:

```
06/07/2020 00:01:06 - INFO - transformers.trainer -   ***** Running training *****
06/07/2020 00:01:06 - INFO - transformers.trainer -     Num examples = 3668
06/07/2020 00:01:06 - INFO - transformers.trainer -     Num Epochs = 3
06/07/2020 00:01:06 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
06/07/2020 00:01:06 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 4
06/07/2020 00:01:06 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/07/2020 00:01:06 - INFO - transformers.trainer -     Total optimization steps = 2751
Iteration:   1%|█▊                                                                                                                                                                                            | 9/917 [00:24<40:42,  2.69s/it]
Epoch:   0%|                                                                                                                                                                                                            | 0/3 [00:24<?, ?it/s]
Traceback (most recent call last):
  File "./run_glue.py", line 247, in <module>
    main()
  File "./run_glue.py", line 174, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1143, in forward
    inputs_embeds=inputs_embeds,
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 727, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to a
ccess index 29597 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```
## Expected behavior

Model successfully trains. The script works well on my machine for many other models, including `bert-base-uncased` ~~and `distilbert-base-cased`~~.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
```
- `transformers` version: 2.11.0
- Platform: Linux-3.10.0-693.el7.x86_64-x86_64-with-centos-7.4.1708-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: [yes]
- Using distributed or parallel set-up in script?: [yes, parallel, but I received the same error when I ran the script just using the CPU]
```s

Comments:
update: this also happens with `distil-bert-cased` for `RTE` and `WNLI` tasks:

```
ds)  # (bs, seq_length, dim)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_distilbert.py", line 91, in forward
    word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 29236 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

update: a different error occurs with `roberta-base` on `STS-B`:

```
06/06/2020 23:49:26 - INFO - transformers.trainer -   ***** Running training *****
06/06/2020 23:49:26 - INFO - transformers.trainer -     Num examples = 5749
06/06/2020 23:49:26 - INFO - transformers.trainer -     Num Epochs = 3
06/06/2020 23:49:26 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
06/06/2020 23:49:26 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
06/06/2020 23:49:26 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/06/2020 23:49:26 - INFO - transformers.trainer -     Total optimization steps = 1080
Iteration:   0%|                                                                                                                                                                                                      | 0/360 [00:09<?, ?it/s]
Epoch:   0%|                                                                                                                                                                                                            | 0/3 [00:09<?, ?it/s]

wandb: Waiting for W&B process to finish, PID 7670
Traceback (most recent call last):
  File "./run_glue.py", line 246, in <module>
    main()
  File "./run_glue.py", line 173, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    wandb: Program failed with code 1. Press ctrl-c to abort syncing.
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bart.py", line 1103, in forward
    loss = F.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward
```

Hello! I tried to reproduce, but couldn't get any results to crash.
Do you mind showing me the exact command you use to launch the script?

Thanks for the response @LysandreJik -- here's an example of one command:

```
CUDA_VISIBLE_DEVICES="" python run_glue.py --model_name_or_path bert-base-cased --tokenizer_name bert-base-cased   --task_name MRPC   --do_train --do_eval --save_steps -1   --data_dir=./glue_data/MRPC/   --max_seq_length 256   --per_device_eval_batch_size=16   --per_device_train_batch_size=16   --learning_rate 2e-5   --num_train_epochs 3   --output_dir=./glue_models/bert-base-cased/MRPC/
```

Here's the full output:
```
06/11/2020 13:19:37 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt from cache at /u/.cache/torch/transformers/5e8a2b4893d13790ed4150ca1906be5f7a03d6c4ddf62296c383f6db42814db2.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1
Made tokenizer: <transformers.tokenization_bert.BertTokenizer object at 0x7fef6cd09550>
06/11/2020 13:19:38 - INFO - transformers.modeling_utils -   loading weights file https://cdn.huggingface.co/bert-base-cased-pytorch_model.bin from cache at /u/.cache/torch/transformers/d8f11f061e407be64c4d5d7867ee61d1465263e24085cfa26abf183fdc830569.3fadbea36527ae472139fe84cddaa65454d7429f12d543d80bfc3ad70de55ac2
06/11/2020 13:19:41 - INFO - transformers.modeling_utils -   Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
06/11/2020 13:19:41 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173336656 acquired on ./glue_data/MRPC/cached_train_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - transformers.data.datasets.glue -   Loading features from cached file ./glue_data/MRPC/cached_train_BertTokenizer_256_mrpc [took 0.110 s]
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173336656 released on ./glue_data/MRPC/cached_train_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173337552 acquired on ./glue_data/MRPC/cached_dev_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - transformers.data.datasets.glue -   Loading features from cached file ./glue_data/MRPC/cached_dev_BertTokenizer_256_mrpc [took 0.013 s]
06/11/2020 13:19:41 - INFO - filelock -   Lock 140666173337552 released on ./glue_data/MRPC/cached_dev_BertTokenizer_256_mrpc.lock
06/11/2020 13:19:41 - INFO - transformers.trainer -   Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: Tracking run with wandb version 0.8.36
wandb: Wandb version 0.9.1 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Run data is saved locally in wandb/run-20200611_171941-90an9vn0
wandb: Syncing run solar-sun-89
wandb: ⭐️ View project at https://app.wandb.ai/jxmorris12/huggingface
wandb: 🚀 View run at https://app.wandb.ai/jxmorris12/huggingface/runs/90an9vn0
wandb: Run `wandb off` to turn off syncing.

06/11/2020 13:19:43 - INFO - transformers.trainer -   ***** Running training *****
06/11/2020 13:19:43 - INFO - transformers.trainer -     Num examples = 3668
06/11/2020 13:19:43 - INFO - transformers.trainer -     Num Epochs = 3
06/11/2020 13:19:43 - INFO - transformers.trainer -     Instantaneous batch size per device = 16
06/11/2020 13:19:43 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
06/11/2020 13:19:43 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/11/2020 13:19:43 - INFO - transformers.trainer -     Total optimization steps = 690
Iteration:   1%|█▍                                                                                                                                                                          | 2/230 [00:16<31:42,  8.35s/it]
Epoch:   0%|                                                                                                                                                                                          | 0/3 [00:16<?, ?it/s]
Traceback (most recent call last):
  File "run_glue.py", line 247, in <module>
    main()
  File "run_glue.py", line 174, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1143, in forward
    inputs_embeds=inputs_embeds,
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 727, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to acc
wandb: Waiting for W&B process to finish, PID 46776
ess index 29597 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb: Process crashed early, not syncing files
```

But -- at least on my machine -- I think a host of combinations of model/task combinations (`--model_name_or_path` and `--data_dir`) fail, as I mentioned above.

I had the same problem, but on my own data set. Have you solved your problem?

Nope @SizhaoXu.

@LysandreJik -- have you had a chance to check this out again? Thanks.

> Nope @SizhaoXu.

For question: RuntimeError: index out of range: Tried to access index 29597 out of table with 28995 rows.
The reason for this question is that the maximum sequence length of the model is 512. 

> > Nope @SizhaoXu.
> 
> For question: RuntimeError: index out of range: Tried to access index 29597 out of table with 28995 rows.
> The reason for this question is that the maximum sequence length of the model is 512.

This solves my problem. I hope that will help you

@SizhaoXu how did you fix it then? by truncating the inputs?

> @SizhaoXu how did you fix it then? by truncating the inputs?

yes! you can try it. The maximum sequence length I set is 512 and I keep the first 200 words, the last 200 words and the middle 112 words

Thanks @SizhaoXu but my problems go beyond that specific one.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> update: a different error occurs with `roberta-base` on `STS-B`:
> 
> ```
> 06/06/2020 23:49:26 - INFO - transformers.trainer -   ***** Running training *****
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Num examples = 5749
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Num Epochs = 3
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
> 06/06/2020 23:49:26 - INFO - transformers.trainer -     Total optimization steps = 1080
> Iteration:   0%|                                                                                                                                                                                                      | 0/360 [00:09<?, ?it/s]
> Epoch:   0%|                                                                                                                                                                                                            | 0/3 [00:09<?, ?it/s]
> 
> wandb: Waiting for W&B process to finish, PID 7670
> Traceback (most recent call last):
>   File "./run_glue.py", line 246, in <module>
>     main()
>   File "./run_glue.py", line 173, in main
>     model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
>     tr_loss += self._training_step(model, inputs, optimizer)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
>     outputs = model(**inputs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
>     outputs = self.parallel_apply(replicas, inputs, kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
>     wandb: Program failed with code 1. Press ctrl-c to abort syncing.
> return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
>     output.reraise()
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
>     raise self.exc_type(msg)
> RuntimeError: Caught RuntimeError in replica 0 on device 0.
> Original Traceback (most recent call last):
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
>     output = module(*input, **kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bart.py", line 1103, in forward
>     loss = F.cross_entropy(logits.view(-1, self.config.num_labels), labels.view(-1))
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 2021, in cross_entropy
>     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
>   File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1838, in nll_loss
>     ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
> RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward
> ```

Hello ! I encountered the same problem when using the STSB dataset for fine-tuning BERT. How did you solve this problem?