Repository: huggingface/transformers

Issue:
GLUE benchmark crashes with MNLI and STSB
## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: no, running only on CPU or single GPU

### Who can help
Tags: @patrickvonplaten  @LysandreJik 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: GLUE
* [ ] my own task or dataset:

## To reproduce

Steps to reproduce the behaviour:

Create model with a very recent version of transformers:
```
>>> from transformers import RobertaForMaskedLM, RobertaConfig, RobertaTokenizer
>>> tok = RobertaTokenizer.from_pretrained('roberta-base')
>>> config = RobertaConfig.from_pretrained('roberta-base')
>>> model = RobertaForMaskedLM(config=config)
>>> tok.save_pretrained('tmp_model')
('tmp_model/tokenizer_config.json', 'tmp_model/special_tokens_map.json', 'tmp_model/vocab.json', 'tmp_model/merges.txt', 'tmp_model/added_tokens.json')
>>> model.save_pretrained('tmp_model')
```

Run GLUE benchmark on MNLI:
```
python ./examples/text-classification/run_glue.py \
            --model_name_or_path tmp_model \
            --task_name MNLI \
            --do_train \
            --do_eval \
            --max_seq_length 128 \
            --per_device_train_batch_size 32 \
            --per_device_eval_batch_size 32 \
            --learning_rate 1e-5 \
            --num_train_epochs 1 \
            --output_dir TMP/
```
or on STSB:
```
python ./examples/text-classification/run_glue.py \
            --model_name_or_path tmp_model \
            --task_name STSB \
            --do_train \
            --do_eval \
            --max_seq_length 128 \
            --per_device_train_batch_size 32 \
            --per_device_eval_batch_size 32 \
            --learning_rate 1e-5 \
            --num_train_epochs 1 \
            --output_dir TMP/
```

### Error logs
MNLI:
> Traceback (most recent call last):
>   File "./examples/text-classification/run_glue.py", line 480, in <module>
>     main()
>   File "./examples/text-classification/run_glue.py", line 415, in main
>     train_result = trainer.train(resume_from_checkpoint=checkpoint)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1048, in train
>     tr_loss += self.training_step(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1432, in training_step
>     loss = self.compute_loss(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1464, in compute_loss
>     outputs = model(**inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1168, in forward
>     loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
>     return F.cross_entropy(input, target, weight=self.weight,
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
>     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
>     ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
> IndexError: Target 2 is out of bounds.

The problem seems to be related to the incorrect initialisation of the classification head. However, with some prints I noticed that the shape of the classification head was `hidden_size x 3`, so I do not really understand where the problem comes from...

STSB:
> Traceback (most recent call last):
>   File "./examples/text-classification/run_glue.py", line 480, in <module>
>     main()
>   File "./examples/text-classification/run_glue.py", line 415, in main
>     train_result = trainer.train(resume_from_checkpoint=checkpoint)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1048, in train
>     tr_loss += self.training_step(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1432, in training_step
>     loss = self.compute_loss(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1464, in compute_loss
>     outputs = model(**inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1168, in forward
>     loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
>     return F.cross_entropy(input, target, weight=self.weight,
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
>     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
>     ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
> RuntimeError: expected scalar type Long but found Float

Here the problem seems to be related to the `dtype` of the targets.


Interestingly, loading an old model like `bert-base-cased` or `roberta-base` does not raise errors....



Discussion:
Hi! Actually I think you touched the base of the issue for MNLI, I would guess it has to do with the number of labels. When you do the following:
```py
>>> from transformers import RobertaForMaskedLM, RobertaConfig, RobertaTokenizer
>>> tok = RobertaTokenizer.from_pretrained('roberta-base')
>>> config = RobertaConfig.from_pretrained('roberta-base')
>>> model = RobertaForMaskedLM(config=config)
>>> model.save_pretrained('tmp_model')
>>> from transformers import RobertaForSequenceClassification
>>> model = RobertaForSequenceClassification.from_pretrained("tmp_model")
```

You'll see that the sequence classification model you've loaded has 2 labels, whereas MNLI is a 3-way classification:
```py
>>> model.classifier.out_proj
Linear(in_features=768, out_features=2, bias=True)
```

I don't get any such errors when changing the configuration initialization to the following:
```py
>>> config = RobertaConfig.from_pretrained('roberta-base', num_labels=3)
```

Regarding the STS-B issue, I think this is an issue that was recently solved on `master`. Can you try pulling the `master` branch once again and letting me know if it fixes your issue? If not, I'll try and check what's happening.
> Hi! Actually I think you touched the base of the issue for MNLI, I would guess it has to do with the number of labels. When you do the following:
> 
> ```python
> >>> from transformers import RobertaForMaskedLM, RobertaConfig, RobertaTokenizer
> >>> tok = RobertaTokenizer.from_pretrained('roberta-base')
> >>> config = RobertaConfig.from_pretrained('roberta-base')
> >>> model = RobertaForMaskedLM(config=config)
> >>> model.save_pretrained('tmp_model')
> >>> from transformers import RobertaForSequenceClassification
> >>> model = RobertaForSequenceClassification.from_pretrained("tmp_model")
> ```
> 
> You'll see that the sequence classification model you've loaded has 2 labels, whereas MNLI is a 3-way classification:
> 
> ```python
> >>> model.classifier.out_proj
> Linear(in_features=768, out_features=2, bias=True)
> ```
> 
> I don't get any such errors when changing the configuration initialization to the following:
> 
> ```python
> >>> config = RobertaConfig.from_pretrained('roberta-base', num_labels=3)
> ```
I didn't have this problem on branch `v4.0.1-release` because the `num_labels` parameter was set automatically. I didn't have to set the exact number of label for each GLUE task. In fact, the script computes them automatically, but for some reason the error still happens.

> Regarding the STS-B issue, I think this is an issue that was recently solved on `master`. Can you try pulling the `master` branch once again and letting me know if it fixes your issue? If not, I'll try and check what's happening.

Error still present on master (`4.4.0.dev0`).
cc @sgugger for knowledge regarding the MNLI issue.

Resolving patch:
diff --git a/examples/text-classification/run_glue.py b/examples/text-classification/run_glue.py
--- a/examples/text-classification/run_glue.py
+++ b/examples/text-classification/run_glue.py
@@ -406,12 +406,15 @@ def compute_metrics(p: EvalPrediction):
 
     # Training
     if training_args.do_train:
+        checkpoint = None
         if last_checkpoint is not None:
             checkpoint = last_checkpoint
         elif os.path.isdir(model_args.model_name_or_path):
-            checkpoint = model_args.model_name_or_path
-        else:
-            checkpoint = None
+            # Check the config from that potential checkpoint has the right number of labels before using it as a
+            # checkpoint.
+            if AutoConfig.from_pretrained(model_args.model_name_or_path).num_labels == num_labels:
+                checkpoint = model_args.model_name_or_path
+
         train_result = trainer.train(resume_from_checkpoint=checkpoint)
         metrics = train_result.metrics
 


Test patch: