Error with run_language_modeling.py training from scratch
Author: mod-cpu
Pull request: no
Labels: name: wontfix

name: Ex: LM (Pretraining)

name: PyTorch

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Training from scratch

Language I am using the model on (English, Chinese ...): Training from scratch with Esperanto (per tutorial)

The problem arises when using:
* [ ] the official example scripts: (give details below) run_language_model.py
* [x] my own modified scripts: (give details below)

Added the class per the tutorial (https://huggingface.co/blog/how-to-train) and call it instead of TextDataset

```
class EsperantoDataset(Dataset):
    def __init__(self, evaluate: bool = false):
        tokenizer = ByteLevelBPETokenizer(
            "./models/EsperBERTo-small/vocab.json",
            "./models/EsperBERTo-small/merges.txt",
        )
        tokenizer._tokenizer.post_processor = BertProcessing(
            ("
", tokenizer.token_to_id("
")),
            ("
", tokenizer.token_to_id("
")),
        )
        tokenizer.enable_truncation(max_length=512)
        # or use the RobertaTokenizer from `transformers` directly.

        self.examples = []

        src_files = Path("./data/").glob("*-eval.txt") if evaluate else Path("./data/").glob("*-train.txt")
        for src_file in src_files:
            print("🔥", src_file)
        lines = src_file.read_text(encoding="utf-8").splitlines()
            self.examples += [x.ids for x in tokenizer.encode_batch(lines)]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        # We’ll pad at the batch level.
        return torch.tensor(self.examples[i])
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) Training from scratch
* [x] my own task or dataset: (give details below) 
 eo_dedup.txt.gz (https://traces1.inria.fr/oscar/)

## To reproduce

Steps to reproduce the behavior:

1. file structure:
project
│   run_language_modeling.py
│
└───models
│    │
│   └───EsperBERTo-small
│            │   merges.txt
│            │   vocab.json
|
└───datasets
             │   eo-dedup-train.txt

2.
```
python run_language_model.py  
	--output_dir ./models/EsperBERTo-small-v1    
	--model_type roberta     
	--mlm     
	--tokenizer_name ./models/EsperBERTo-small     
	--do_train     
	--learning_rate 1e-4     
	--num_train_epochs 5     
	--save_total_limit 2     
	--save_steps 2000     
	--per_gpu_train_batch_size 4     
	--evaluate_during_training     
	--seed 42 
	--train_data_file eo-dedup-train.txt
```
Results in this stack trace with CUDA_LAUNCH_BLOCKING=1:

```
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [53,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [53,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "test.py", line 832, in <module>
    main()
  File "test.py", line 782, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "test.py", line 386, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 243, in forward
    inputs_embeds=inputs_embeds,
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 799, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 64, in forward
    input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 193, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered
```

## Expected behavior


## Environment info
     
- `transformers` version: 2.4.1
- Platform: AWS p2.xlarge ubuntu
- Python version: 3.6.5
- PyTorch version (GPU?): 1.3.1
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
I ran into this with my own dataset. Following some discussion in #1538 I changed truncation to 256.

> tokenizer.enable_truncation(max_length=256)

I also had to make sure that the pad token had index 1, as that seems to be hardcoded in roberta.

This appears to work, though the previous error had only appeared fairly deep into training, and reducing context isn't great. So I'm not satisfied by this hack. 

If you want to see the error clearer, switch it to CPU, then it will print out the real error. I rant into this error in another project, and finally, I found out it is basically `index out of range` error. Fixed it by add some missing words to the vocabulary.txt and resize the model itself.

> I ran into this with my own dataset. Following some discussion in #1538 I changed truncation to 256.
> 
> > tokenizer.enable_truncation(max_length=256)
> 
> I also had to make sure that the pad token had index 1, as that seems to be hardcoded in roberta.
> 
> This appears to work, though the previous error had only appeared fairly deep into training, and reducing context isn't great. So I'm not satisfied by this hack.
@reidsanders were u able to train language model with ur own dataset eventually?
@binhna have u added missing words inplaceon unk or randomly added them in vocab, ?


Hello @reidsanders, @samreenkazi 

I encountered the same error and tried  `tokenizer.enable_truncation(max_length=256)` on some BERT models. But it seems that there is no such method:
`'PreTrainedTokenizer' object has no attribute 'enable_truncation'`

Could you give more details about how you solved the problem?

I am unable to solve this problem as  yet

On Thu, Mar 5, 2020 at 10:22 PM Jinan Zhou <notifications@github.com> wrote:

> Hello @reidsanders <https://github.com/reidsanders>, @samreenkazi
> <https://github.com/samreenkazi>
>
> I encountered the same error tried
> tokenizer.enable_truncation(max_length=256) on some BERT models. But it
> seems that there is no such method:
> 'PreTrainedTokenizer' object has no attribute 'enable_truncation'
>
> Could you give more details about how you solved the problem?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2877?email_source=notifications&email_token=ALPPGB4ZSN6BT5PXL6QRWWTRF7NU5A5CNFSM4KWFP7FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN6ETGY#issuecomment-595347867>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALPPGBYDQHHQTNNOESJOZ33RF7NU5ANCNFSM4KWFP7FA>
> .
>


> Hello @reidsanders, @samreenkazi
> 
> I encountered the same error and tried `tokenizer.enable_truncation(max_length=256)` on some BERT models. But it seems that there is no such method:
> `'PreTrainedTokenizer' object has no attribute 'enable_truncation'`
> 
> Could you give more details about how you solved the problem?

enable_truncation is not a method in PreTrainedTokenizers (we are training from scratch, not pretrained). I'm using ByteLevelBPETokenizer imported from tokenizers module as in the example blog post (and op).

Thanks for your comment @reidsanders @samreenkazi 

According to my observation, the error is indeed caused by the data samples longer than 512 tokens. A conservative solution is fixing `block_size=512` in `TextDataset` and `LineByLineTextDataset` class. Or if you worry about the truncation of data, you can go through `self.examples` in these two classes, check whether each sample is shorter than 512. If not, split it into multiple chunks with length 512. 
 

For me it helped to just specify the argument `--block_size=512` (it's -1 by default)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.