run_glue.py on stsb works on bert-base-uncased but fails on a finetuned BERT ckpt
Author: BiEchi
Pull request: no

### System Info

-  `transformers` version: 4.32.0.dev0
- Platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.0
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

 @ArthurZucker  and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Working example:

```
transformers/examples/pytorch/text-classification/run_glue.py \
        --model_name_or_path "bert-base-uncased" \
        --task_name "stsb" \
        --save_strategy no \
        --do_train \
        --do_eval \
        --fp16 \
        --ddp_timeout 180000 \
        --max_seq_length 512 \
        --per_device_train_batch_size 32 \
        --per_device_eval_batch_size 32 \
        --learning_rate 2e-5 \
        --num_train_epochs 3 \
        --output_dir $OUTPUT_DONTCARE_DIR \
        --ignore_mismatched_sizes \
        --overwrite_output_dir
```

Then I finetune this on MNLI and save it to $OUTPUT_MNLI_DIR. Then it fails using:

```
transformers/examples/pytorch/text-classification/run_glue.py \
        --model_name_or_path $OUTPUT_MNLI_DIR \
        --task_name "stsb" \
        --save_strategy no \
        --do_train \
        --do_eval \
        --fp16 \
        --ddp_timeout 180000 \
        --max_seq_length 512 \
        --per_device_train_batch_size 32 \
        --per_device_eval_batch_size 32 \
        --learning_rate 2e-5 \
        --num_train_epochs 3 \
        --output_dir $OUTPUT_DONTCARE_DIR \
        --ignore_mismatched_sizes \
        --overwrite_output_dir
```

With error:

```
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/examples/pytorch/text-classification/run_glue.py", line 629, in <module>
    main()
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/examples/pytorch/text-classification/run_glue.py", line 537, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/src/transformers/trainer.py", line 1532, in train
    return inner_training_loop(
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/src/transformers/trainer.py", line 1805, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/src/transformers/trainer.py", line 2648, in training_step
    loss = self.compute_loss(model, inputs)
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/src/transformers/trainer.py", line 2673, in compute_loss
    outputs = model(**inputs)
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/accelerate/utils/operations.py", line 581, in forward
    return model_forward(*args, **kwargs)
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/accelerate/utils/operations.py", line 569, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/global/scratch/users/asap7772/projects/crate4text/transformers/src/transformers/models/bert/modeling_bert.py", line 1597, in forward
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/modules/loss.py", line 1174, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/global/scratch/users/asap7772/miniconda3/envs/crate/lib/python3.9/site-packages/torch/nn/functional.py", line 3029, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Float'
```

### Expected behavior

It should run without an error.

Comments:
I'm closing this issue as adding `problem_type='regression'` solves this issue.