RuntimeError: expected dtype Float but got dtype Long - run_lm_finetuning.py
Author: paulthemagno
Pull request: no

# 🐛 Bug

## Information

I'm using my Camembert-based language model on Italian language (built from scratch).

I'm trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) to fine-tune my language model on a dataset. 

@julien-c suggested my to add  `--line_by_line`  to my launch script, beacuse without that flag, the program blocked on the tokenization of the training set. That advide let the program work. But after some hours, the program crashes with a strange Runtime Error: in the assignment of 10 % of random words to masks at [line 218](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L218) in the _mask_tokens()_ function:
```python
# 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random] #it crashes here
```

The error was this:
`RuntimeError: expected dtype Float but got dtype Long
`
It's a strange error, beacuse it crashes after several minutes or hours after the launch. The time of the proper functioning of program seems random: sometimes the script completes 3/4 epochs and then it crashes, sometimes it crashes before the end of the first epoch.
 
## To reproduce

I launched this:
```bash
python3 run_lm_finetuning.py \
--train_data_file /path/to/train.txt \
--eval_data_file /path/to/eval.txt \
--output_dir /path/to/output \
--mlm \
--do_train \
--do_eval \
--model_type camembert \
--model_name_or_path /path/to/my/model \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--overwrite_output_dir \
--overwrite_cache \
--max_steps 500000 \
--block_size 128 \
--save_steps 50000 \
--eval_all_checkpoints \
--line_by_line
```

I got this error in the middle of 6th epoch:
```
File "run_lm_finetuning.py", line 801, in <module>51:22,  4.94it/s]
    main()
  File "run_lm_finetuning.py", line 750, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 342, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 222, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  55%|█████▍    | 6/11 [20:26:33<17:02:07, 12265.60s/it]
Iteration:  69%|██████▊   | 33378/48603 [1:47:45<49:09,  5.16it/s]
```

I'm managing to run the code anyway, restarting the program using the flag `--model_name_or_path` and giving the last saved checkpoint rather then the original language model every time it crashes.

I printed `inputs[indices_random]` and `random_words[indices_random]` beacause are the two variables in the line in which the program crashes:

- The code crashes with this 2 variables:
```
inputs[indices_random] = tensor([1173.])
Random_words[indices_random] = tensor([4220])

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 797, in <module>
    main()
  File "run_lm_finetuning.py", line 747, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 349, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 229, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  60%|██████    | 3/5 [14:31:21<9:40:54, 17427.18s/it]
```
- while before the crash the code enters the _mask_tokens()_ function corretly and prints lines like these:
```
inputs[indices_random] = tensor([19807,    78,    51,  1204])
Random_words[indices_random] = tensor([14538, 15381, 30255,  3778])
```

In my opinion the only difference is that **tensor([1173.])** in the crash example, contains a not integer value (there is the '.' at the end of the number, while all the other times not. Maybe with a cast of `inputs[indices_random]` it would work.

## Environment info

- `transformers` version: 2.4.1
- Platform: Linux-4.4.0-108-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Comments:
@paulthemagno thanks for creating this. My environment is exactly the same, except I am running the lm fine tuner on a Python 3.7.3 environment.  @LysandreJik  asked for more information. A couple of more inputs although I am not sure if this is going to help. This problem is not happening when I subset my dataset and run the code. Neither on training nor evaluation.  So this tells me there is a data problem somewhere.

So I caught the runtime error and output the results of those data objects for my dataset on line 120-122 that is posted above. Only unusual thing I see is that one of the examples for this batch where the code errors out is all zeros for the input tensor. 
```

tensor([[    0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.],
[ 2004.,  2019.,  5587., 10497.,  2819.,  2000.,  1996.,  2194.,  2015.,
          2128.,  8569., 28200., 10896.,  1010.,  3531.,  2421.,  1037.,  2862.,
          1997.,  2169.,  8875.,  2073.,  1996.,  2194.,   103.,  2015.,  2007.,
          1996.,   103.,  1997.,  1996.,  2060.,  4243.,  2000.,   103.,  8946.,
          3388.,  1012.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.],
...
```
My data is split as 1) each sentence in one line, 2) and the documents are split by an empty line as recommended in the documentation. I looked at that particular document but I do not see anything unusual but maybe some of this provide some clues.  My hunch was that there is a non-ascii character which I am cleaning up, or maybe a dash or underscore repeating many many times for that particular example but if my eyes are not failing me, I can't find that in the dataset for that batch.

Thanks you all... 



Hi, I'm looking into this problem right now, thank you for providing so much helpful information!
I could set up an experiment where I would get the same error, and I patched it with a cast as @paulthemagno recommended.

It is visible in 1ebfeb7. This should hopefully patch your issue, but as I don't have your particular dataset I can't verify first hand. Do you mind letting me know if it fixes it?

> Hi, I'm looking into this problem right now, thank you for providing so much helpful information!
> I could set up an experiment where I would get the same error, and I patched it with a cast as @paulthemagno recommended.
> 
> It is visible in [9c67196](https://github.com/huggingface/transformers/commit/9c67196b83a824df577742d32d38e9121d8a9285). This should hopefully patch your issue, but as I don't have your particular dataset I can't verify first hand. Do you mind letting me know if it fixes it?


Thanks, that patch or code block does not reflect the change, maybe a typo on the commit hash? 

Indeed, sorry, edited.

@LysandreJik happy to confirm that it worked. I patched my own script with the added line for casting the inputs and it ran through the whole corpus 160K records and outputted 5.6 perplexity score. I am assuming it worked.  Thank you very much... 

Ozan

Fantastic! Thank you for checking!

> Fantastic! Thank you for checking!

You're welcome.  I am glad I could help.  By the way, out of topic, could you shed some light on why my input tensors are truncated to length = 51 as you can see in my original post above. I don't see where I set that to 51 nor a hard code somewhere. Here are my script arguments:

```
python run_lm_finetuning.py \
 --train_data_file /path/to/data \
  --eval_data_file /path/to/eval_file \
   --output_dir /path/fine_tuned/bert_uncased_lm \
    --mlm \
    --do_train \
     --do_eval \
     --cache_dir  /cache_dir \
     --model_type bert \
     --model_name_or_path bert-base-uncased \
      --per_gpu_train_batch_size 16 \
      --gradient_accumulation_steps 2 \
      --per_gpu_eval_batch_size 16 \
      --block_size 256 \
      --eval_all_checkpoints \
       --line_by_line \
       --fp16
```
As far as I understand Block size is the after tokenization, max seq length is that, where is 51 coming from? This might be a stupid question but I am just trying to avoid making a gross error and get a little bit more of an understanding of the code. 

Ozan


That seems weird, indeed, but it's hard for me to debug without having more information about your dataset. Since you're using the `--line_by_line` flag, it should be building tensors according to the line returns in your dataset. Is it possible 51 is the maximum length of a sequence for that specific batch, so it pads up to 51 for the rest of the batch?

Yes, that must be it, I checked some random batches and the length for the input tensors varies from batch to batch.  I apologize for sidetracking this thread. Seemed like while I had you and the data above, I would get a quick answer. thank you again. 

No worries, glad I could help.

> Hi, I'm looking into this problem right now, thank you for providing so much helpful information!
> I could set up an experiment where I would get the same error, and I patched it with a cast as @paulthemagno recommended.
> 
> It is visible in [1ebfeb7](https://github.com/huggingface/transformers/commit/1ebfeb79469d544a2bd817aa32c77e0514485ff9). This should hopefully patch your issue, but as I don't have your particular dataset I can't verify first hand. Do you mind letting me know if it fixes it?

Thanks to all. I had already launched the code before you wrote this message, with the additional line `inputs = inputs.type(dtype=torch.long)` without the _clone_ method. It has worked, but I think it is better to restart from 0. Also beacuse re-launching the code from the last saved checkpoint (before the crash), I have noticed that the first new checkpoint had a peek on the perplexity and after that it return to decrease, so better restarting.

Anyway the code worked with my change, so I think also with yours, which is more correct :)