Trouble fine tuning BERT language model 
Author: Buguemar
Pull request: no

## 🐛 Bug

Hello, I'm having trouble running **run_lm_finetuning.py** script. I'm using pytorch 1.2, python 3.5, CUDA 9.2, Ubuntu 18.04.

When I run

```
$python run_lm_finetuning.py     
--output_dir= my_output_dir/  
--model_type=bert     
--model_name_or_path=bert-base-uncased    
--do_train     
--train_data_file=$TRAIN_FILE     
--do_eval     
--eval_data_file=$TEST_FILE     
--mlm
```

I obtain: 

```
***** Running training *****
Num examples = 4517
Num Epochs = 1
Instantaneous batch size per GPU = 4
Total train batch size (w. parallel, distributed & accumulation) = 4
Gradient Accumulation steps = 1
Total optimization steps = 1130
Epoch:   0%|                                                                                        | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):                                                               | 0/1130 [00:00<?, ?it/s]
  File "lm_fine_backup.py", line 712, in <module>
    main()
  File "lm_fine_backup.py", line 662, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "lm_fine_backup.py", line 298, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mbugueno/.local/lib/python3.5/site-packages/transformers/modeling_bert.py", line 899, in forward
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/functional.py", line 2009, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/casapanshop/anaconda2/envs/py3/lib/python3.5/site-packages/torch/nn/functional.py", line 1838, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97
```

I'm using the official example script in WikiText-2 dataset.
Any insights?


Comments:
I have the same question but no answer. In my case, I ran it in google colab and used easydict to deal with arg parser. 

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97

I faced this problem when I checkout the latest code, but it is worked when I checkout v2.3.0 version.

facing the same problem

works fine when i build from source 

I assume you installed the transformers with pip install, there is a bug in roberta, you can manually fix it by editing transformers/modeling_roberta.py file in Line 291 - https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py#L291

Change:
`loss_fct = CrossEntropyLoss(-1)`
to 
`loss_fct = CrossEntropyLoss()`

Thank you guys! Specially to @orena1 ! 
Change the definition of the loss function (CrossEntropyLoss(-1)) worked for me. 

Sorry for the late! but I'm very happy, I can finally fine-tune the model! hahaha