Trouble running fine tuned language model script
Author: Khev
Pull request: no

## 🐛 Bug

I'm having trouble running the finetuning script. When I run

```

TRAIN_FILE='/home/khev/research/transformers/data/wikitext-2-raw/wiki.train.raw'
TEST_FILE='/home/khev/research/transformers/data/wikitext-2-raw/wiki.test.raw'

%run run_lm_finetuning.py \
    --output_dir=output \
    --model_type=bert \
    --model_name_or_path=bert-base-uncased\
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm


```

I get the following error:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/research/transformers/examples/run_lm_finetuning.py in <module>
    545 
    546 if __name__ == "__main__":
--> 547     main()

~/research/transformers/examples/run_lm_finetuning.py in main()
    497             torch.distributed.barrier()
    498 
--> 499         global_step, tr_loss = train(args, train_dataset, model, tokenizer)
    500         logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
    501 

~/research/transformers/examples/run_lm_finetuning.py in train(args, train_dataset, model, tokenizer)
    222         epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
    223         for step, batch in enumerate(epoch_iterator):
--> 224             inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
    225             inputs = inputs.to(args.device)
    226             labels = labels.to(args.device)

~/research/transformers/examples/run_lm_finetuning.py in mask_tokens(inputs, tokenizer, args)
    147     probability_matrix = torch.full(labels.shape, args.mlm_probability)
    148     special_tokens_mask = [tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()]
--> 149     probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
    150     masked_indices = torch.bernoulli(probability_matrix).bool()
    151     labels[~masked_indices] = -1  # We only compute loss on masked tokens

RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'
```

Info:

Python 3.7.3
Torch 1.1.0
Ubuntu 18.04.3 LTS


Comments:
Hi, could you specify the versions on which you are running? (especially PyTorch).

`bool` was introduced in PyTorch 1.2 so, unfortunately, this example will crash if using an anterior version.

Sorry, forgot to give that info. As you preempted, I'm running torch 1.1.0. Guess I should update?

The core library should run on PyTorch 1.0.1+, but examples usually require torch 1.2+. It would be better to upgrade if you want to try out the examples, indeed!

Excellent, will do, thanks! And Kudos on the fast response time BTW :-)

No problem, glad to help!

Ah, now I'm getting the following error. Any insight?

FYI: I'm now at pytorch 1.3.1 -- think I should downgrade to 1.2.0?

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/research/transformers/examples/run_lm_finetuning.py in <module>
    546 
    547 if __name__ == "__main__":
--> 548     main()

~/research/transformers/examples/run_lm_finetuning.py in main()
    498             torch.distributed.barrier()
    499 
--> 500         global_step, tr_loss = train(args, train_dataset, model, tokenizer)
    501         logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
    502 

~/research/transformers/examples/run_lm_finetuning.py in train(args, train_dataset, model, tokenizer)
    227             labels = labels.to(args.device)
    228             model.train()
--> 229             outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
    230             loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
    231 

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, masked_lm_labels)
    768                             token_type_ids=token_type_ids,
    769                             position_ids=position_ids,
--> 770                             head_mask=head_mask)
    771 
    772         sequence_output = outputs[0]

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask)
    622             head_mask = [None] * self.config.num_hidden_layers
    623 
--> 624         embedding_output = self.embeddings(input_ids, position_ids=position_ids, token_type_ids=token_type_ids)
    625         encoder_outputs = self.encoder(embedding_output,
    626                                        extended_attention_mask,

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids)
    165             token_type_ids = torch.zeros_like(input_ids)
    166 
--> 167         words_embeddings = self.word_embeddings(input_ids)
    168         position_embeddings = self.position_embeddings(position_ids)
    169         token_type_embeddings = self.token_type_embeddings(token_type_ids)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115 
    116     def extra_repr(self):

~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485 
   1486 

RuntimeError: index out of range: Tried to access index 38889 out of table with 30521 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418

```

Hmm that error should not be linked to a version problem. Did you use the same command:

```
run_lm_finetuning.py \
    --output_dir=output \
    --model_type=bert \
    --model_name_or_path=bert-base-uncased\
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
```

or did you use a different checkpoint?

I used the same command...

I also have same issue while finetuning distilgpt2:

```
  File "train_docker/train", line 229, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 533, in forward
    head_mask=head_mask)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 420, in forward
    inputs_embeds = self.wte(input_ids)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/functional.py", line 1467, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 50257 out of table with 50256 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:237
```

Update: the following two commands, for GTP-2 and roberto, are working. So I will stick to these for now. Thanks for your help!

```
%run run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE
```

```
TRAIN_FILE='/home/khev/research/transformers/data/wikitext-2-raw/wiki.train.raw'
TEST_FILE='/home/khev/research/transformers/data/wikitext-2-raw/wiki.test.raw'

%run run_lm_finetuning.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
```

Thanks for reporting those bugs @Khev and @iedmrc, I'll check what's going on for BERT and DistilGPT-2.

I found that It occurs when we have a "cached feature file".

I have one last question. I want to use the fine tuning script on a custom dataset: the titles of scientific papers. So my dataset has form: 

[title1, title2, ..., ]

where title_i = 'this is a paper title'

I'm wondering how this dataset should be structured. I assume it should mimic the wikitext dataset, but I'm not entirely sure of their structure. It looks like sections are delimited by " ==". For exampl, in the wiki.test.raw file, the first few lines are:

"""
= Robert Boulter =

TEXT

== Career ==
"""

Should I put my dataset in this form? So like:

== Paper1 ==
title 1

== Paper2
title 2?


To me, you should use 
 as a delimiter in the gpt2 case. Check page 4 of the original gpt paper openai published. 
https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf

As @iedmrc said, you should probably mimic the way the individual models were pre-trained instead. 

For example, for GPT-2 you might want to add the suffix `
` (available via the `tokenizer.eos_token` attribute) to indicate to the model you're joining different texts. 

We're not doing that in the `run_lm_finetuning.py` script as we want to keep the example simple and the texts are long enough in the WikiText-2 dataset for it not to affect performance too much.

I understand. Thanks!

I just pushed a fix on `master` that should correct the second issue you faced. Feel free to re-open if it did not solve your issue so that I may investigate further.