Assertion error when training a new RoBERTa from scratch
Author: erip
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: <No.

### Who can help

Maybe @LysandreJik  ? :-) 


## Information

Model I am using RoBERTa:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

The dataset is a simple line-by-line dataset.

## To reproduce

Steps to reproduce the behavior:

1. Train a tokenizer according to [this](https://huggingface.co/blog/how-to-train#2-train-a-tokenizer)
2. Split line-by-line dataset into train and eval
3. Run below:

```sh
python run_language_modeling.py \
    --output_dir $MODEL_DIR/myBERT-small-v1 \
    --model_type roberta \
    --mlm \
    --config_name $MODEL_DIR/myBERT-small \
    --tokenizer_name $MODEL_DIR/myBERT-small \
    --do_train \
    --do_eval \
    --per_device_train_batch_size 8 \
    --learning_rate 1e-4 \
    --num_train_epochs 5 \
    --save_total_limit 2 \
    --save_steps 2000 \
    --per_gpu_train_batch_size 16 \
    --evaluate_during_training \
    --line_by_line \
    --train_data_file $HOME/myBERT/train.txt \
    --eval_data_file $HOME/myBERT/eval.txt \
    --seed 42
```

```log
08/13/2020 14:23:20 - INFO - transformers.configuration_utils -   loading configuration file /home/erippeth/myBERT/model/myBERT-small/config.json
08/13/2020 14:23:20 - INFO - transformers.configuration_utils -   Model config RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 52000
}

08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Model name '/home/erippeth/myBERT/model/myBERT-small' not found in model shortcut name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). Assuming '/home/erippeth/myBERT/model/myBERT-small' is a path, a model identifier, or url to a directory containing tokenizer files.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/added_tokens.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/special_tokens_map.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/tokenizer_config.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/tokenizer.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file /home/erippeth/myBERT/model/myBERT-small/vocab.json
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file /home/erippeth/myBERT/model/myBERT-small/merges.txt
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:21 - INFO - __main__ -   Training new model from scratch
/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_auto.py:709: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
08/13/2020 14:23:27 - INFO - transformers.data.datasets.language_modeling -   Creating features from dataset file at /home/erippeth/myBERT/train.txt
08/13/2020 17:40:20 - INFO - transformers.data.datasets.language_modeling -   Creating features from dataset file at /home/erippeth/myBERT/eval.txt
08/13/2020 18:56:31 - WARNING - transformers.trainer -   You are instantiating a Trainer but Tensorboard is not installed. You should consider installing it.
08/13/2020 18:56:31 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
08/13/2020 18:56:31 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
08/13/2020 18:56:31 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
08/13/2020 18:56:31 - INFO - transformers.trainer -   ***** Running training *****
08/13/2020 18:56:31 - INFO - transformers.trainer -     Num examples = 16661098
08/13/2020 18:56:31 - INFO - transformers.trainer -     Num Epochs = 5
08/13/2020 18:56:31 - INFO - transformers.trainer -     Instantaneous batch size per device = 8
08/13/2020 18:56:31 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
08/13/2020 18:56:31 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
08/13/2020 18:56:31 - INFO - transformers.trainer -     Total optimization steps = 5206595
^MEpoch:   0%|          | 0/5 [00:00<?, ?it/s]
^MIteration:   0%|          | 0/1041319 [00:00<?, ?it/s]ESC[A
^MIteration:   0%|          | 1/1041319 [00:01<508:20:24,  1.76s/it]ESC[A
^MIteration:   0%|          | 2/1041319 [00:02<395:24:33,  1.37s/it]ESC[A
^MIteration:   0%|          | 3/1041319 [00:02<306:50:22,  1.06s/it]ESC[A/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [229,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [229,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [275,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Iteration:   0%|          | 3/1041319 [00:03<332:03:04,  1.15s/it]
Epoch:   0%|          | 0/5 [00:03<?, ?it/s]
Traceback (most recent call last):
  File "run_language_modeling.py", line 281, in <module>
    main()
  File "run_language_modeling.py", line 245, in main
    trainer.train(model_path=model_path)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/trainer.py", line 622, in _training_step
    outputs = model(**inputs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 239, in forward
    output_hidden_states=output_hidden_states,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 762, in forward
    output_hidden_states=output_hidden_states,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 439, in forward
    output_attentions,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 371, in forward
    hidden_states, attention_mask, head_mask, output_attentions=output_attentions,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 315, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 258, in forward
    context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
RuntimeError: CUDA error: device-side assert triggered
```

## Expected behavior

Model should train without failure, but instead it fails with an assertion error. I believe this is related to an embedding dimension issue, but the config's `vocab_size` matches the length of the newly-trained tokenizer and this is the embedding dimension set in the training script.

Comments:
This may be due to an embedding dimension issue, but may also be due to a CUDA OOM error earlier that has been misreported in my experience. To verify that it is an embedding dimension issue, can you try using the `--no_cuda` flag?

Sure - let me give it a shot. The one issue is that the data is large so featurizing the inputs takes a long time (and isn't cached), so it may take several hours to report back.

@LysandreJik I confirmed it was indeed an embedding issue:

```log
Traceback (most recent call last):
  File "run_language_modeling.py", line 281, in <module>
    main()
  File "run_language_modeling.py", line 245, in main
    trainer.train(model_path=model_path)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/trainer.py", line 622, in _training_step
    outputs = model(**inputs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 239, in forward
    output_hidden_states=output_hidden_states,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 753, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 68, in forward
    input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 179, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

What's not immediately clear is _why_ it's happening. My understanding is the  process goes...

1. Load the tokenizer.
2. Encode each line (forcing the indices to necessarily fall in the range of |vocab|)
3. Train

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.