Problem fine-tuning BERTweet
Author: ioana-blue
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: no, single gpu


Maybe  @LysandreJik could help?


## Information

Model I am using BERTweet

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I get the following error: 
```
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "../models/jigsaw/tr-3.4//run_puppets.py", line 284, in <module>
    main()
  File "../models/jigsaw/tr-3.4//run_puppets.py", line 195, in main
    trainer.train(
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/trainer.py", line 888, in train
    tr_loss += self.training_step(model, inputs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/trainer.py", line 1250, in training_step
    loss = self.compute_loss(model, inputs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/trainer.py", line 1277, in compute_loss
    outputs = model(**inputs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1137, in forward
    outputs = self.roberta(
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 791, in forward
    embedding_output = self.embeddings(
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 121, in forward
    embeddings += position_embeddings
```

I just upgraded to the latest pytorch and transformers, I had same issue with different versions (transformers 3.4, torch 1.5.1). 

Some more info on how I got here: https://github.com/VinAIResearch/BERTweet/issues/26

I've used the same code with 10+ other models (e.g., bert, roberta, distillbert) with no issues. One difference that I noticed in the config files for these models compared to BERTweet: max_position_embeddings is 512 for the models I'm using with no issues, while it is set to 130 in the config file for BERTweet. 

One (related?) clarification question: what's the relation between `max_position_embeddings` and `max_seq_length`?

Any insights, more than welcome. Thanks!


Comments:
Hi @ioana-blue ,

`max_seq_length` is the naming "convention" when talking about the tokenization side, e.g. when you tokenize your tweets they will be converted to ids and normally padded to a `max_seq_length`.

`max_position_embeddings` is the naming convention when talking about pre-training the model, so e.g. BERT has seen 512 subtokens during pre-training phase.

So `max_seq_length` should be less or equal `max_position_embeddings`.

In your case it seems that the model has seen 130 subtokens during pre-training phase (which is ok, because tweets usually are much shorter than 512 subtokens). 

Could you check your tokenization part and the number of subtokens that you're later passing to the model :thinking: 

It would also help if you could paste the tokenization part (e.g. converting plain text/tweets into model ids) here, so we can have a look into it!

Ok, I thought that was one potential issue. So I ran with `max_seq_length` of 100 and I still get the same problem. I'll probably ask for a feature request to print an error when trying to run with a `max_seq_length` that is higher than `max_possition_embeddings` (I used to run it with 512 for `max_seq_length` and there is no complain). 

I'm using slightly modified version of the glue example in the code, so I didn't modify any of the tokenization part. The only thing that I added is the data processors/loaders. 

Let me know if I could provide any more info to help debug this issue. Greatly appreciate your help!


I just wanted to reproduce the error message with an example after my :pizza: , but it is working with the GLUE example:

```bash
python3 run_glue.py \
  --model_name_or_path vinai/bertweet-base \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --use_fast False \
  --output_dir /tmp/$TASK_NAME/
```

Important argument is to pass `--use_fast False` to avoid an error message. I set `TASK_NAME` to the `wnli` task.

Could you specify what version of Transformers you're using :thinking: I'm using a 4.3 version (d5888ef0ab949ec82ed4768556c2b2743e3ca1df).

Is it also possible that you paste the trainer output that shows e.g.:

```bash
02/04/2021 20:59:33 - INFO - __main__ -   Sample 281 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'idx': 281, 'input_ids': [0, 1717, 966, 9, 329, 2
125, 24, 6, 52562, 7, 42, 58, 8215, 29, 41118, 7939, 4, 2, 2, 2125, 8215, 29, 41118, 7939, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 1, 1], 'label': 0, 'sentence1': "Paul tried to call George on the phone, but he wasn't successful.", 'sentence2': "George wasn't su
ccessful.", 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0]}.
```



Oh, and could you also pass `--overwrite_cache` to the training script,  this is really helpful espec. when you experiment with different sequence lengths 😀

Thanks for your help! 

I tried the code with 3.4 and 4.2, similar behavior. 

The encoding look fine to me:

```
02/04/2021 16:15:57 - INFO - util_processors -   *** Example ***
02/04/2021 16:15:57 - INFO - util_processors -   guid: test-1178818409812746240_twitter
02/04/2021 16:15:57 - INFO - util_processors -   features: InputFeatures(input_ids=[0, 10584, 56843, 241, 66, 103, 6, 289, 1389, 32, 38, 97, 11, 23465, 72, 618, 27658, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
02/04/2021 16:15:57 - INFO - util_processors -   *** Example ***
02/04/2021 16:15:57 - INFO - util_processors -   guid: test-19346774_gab
02/04/2021 16:15:57 - INFO - util_processors -   features: InputFeatures(input_ids=[0, 52, 112, 52, 37, 1621, 11, 8812, 15, 634, 5230, 37, 116, 45, 96, 11, 3559, 25, 37, 56, 140, 28748, 701, 24, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=0)
02/04/2021 16:15:57 - INFO - util_processors -   *** Example ***
02/04/2021 16:15:57 - INFO - util_processors -   guid: test-1165819983701643266_twitter
02/04/2021 16:15:57 - INFO - util_processors -   features: InputFeatures(input_ids=[0, 557, 31, 39, 94, 11, 397, 31, 1844, 46154, 13, 1190, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], token_type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], label=1)
```

`--use_fast` doesn't work for me (no such param). 

I always use overwrite cache 👍 

This is my command line:

```
python ../models/jigsaw/tr-3.4//run_puppets.py --model_name_or_path vinai/bertweet-base --task_name binary_hatex --do_train --do_eval --do_logits --do_predict --data_dir /dccstor/redrug_ier/ioana/fairnlp/toxi-data/hatex/processed/ --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 1 --output_dir /dccstor/redrug_ier/ioana/fairnlp/toxi-data/hatex/run-results/runpred_e1_binary_hatex_bertweet_20210204_12_03_58 --cache_dir /dccstor/redrug_ier/ioana/fairnlp/toxi-data/hatex/run-results/runpred_e1_binary_hatex_bertweet_20210204_12_03_58/cache/ --overwrite_cache --logging_steps 10000 --save_steps 200000
```
I implemented some command line args for predicting and printing logits, etc., but it doesn't get there, the problem is in the training. 

If I feel adventurous, I will probably try to step through the training and see if I notice any issues. It looks like an out of bounds indexing somewhere. 

I'm going to try the run_glue on my side to see if I can reproduce your successful run. 


Ayayay. I got a successful run with 3.4 and the command line above (my own code, I mean). Strange. 

Yep, I can confirm running with different seq size and it works. I think what happened was the following:
- Initially I was running with a seq size that was too large. 
- I upgraded the transformers to 4.2 and also realized the seq size problem. I started using smaller seq sizes, but there was a problem. I'm guessing the problem comes from some backward-compatibility (my code was inspired by the sample code from version 3.4; I'm guessing something changed that breaks the code with 4.2)
- Once I went back to 3.4 AND small seq size, it worked. I'll open a feature request. I don't think runs should be allowed with `max_seq_size > max_position_embeddings`

Thanks for your help, much appreciated. I'm closing this one.