`Gradient clipping` function is not compatible with upgrade
Author: Baibaifan
Pull request: no

### System Info

`Gradient clipping` function is not compatible with upgrade.

transformers4.28.1：
![image](https://github.com/huggingface/transformers/assets/39549453/52744be1-ebaf-4259-aacd-438e495bf9d2)

Transformers4.30.2:
![image](https://github.com/huggingface/transformers/assets/39549453/b8b9fe7a-ca3e-4b85-8b6d-0598391e9663)

**The old and new versions do not support fp16 uniformly.**


### Who can help?

@sgugger，@pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

you can run with transformers.

```shell
torchrun \
  --nproc_per_node $NUM_GPU \
  --master_port $PORT_ID \
  run_bloom.py \
  --model_name_or_path facebook/opt-125m \
  --use_fast_tokenizer False \
  --train_file $TRAIN \
  --validation_file $VAL \
  --test_file $TEST \
  --max_seq_length 512 \
  --output_dir $OUTPUT \
  --do_train True\
  --do_eval False \
  --do_predict False \
  --evaluation_strategy no \
  --eval_steps 1000000000 \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 16 \
  --learning_rate 1e-5 \
  --optim adamw_torch \
  --adam_beta2 0.95 \
  --weight_decay 0.1 \
  --num_train_epochs 1 \
  --lr_scheduler_type constant_with_warmup \
  --warmup_ratio 0.1 \
  --logging_first_step True \
  --logging_steps 10 \
  --logging_nan_inf_filter False \
  --save_strategy steps \
  --save_steps 10000 \
  --save_total_limit 3 \
  --fp16 True \
  --disable_tqdm False \
  --log_on_each_node False \
  --report_to tensorboard \
```

  **important configuration**: --fp16 True 

### Expected behavior

Will not report bugs.

Comments:
The `unscale_gradients` is done in the first screenshot a couple of lines above. I'm not sure what it is you are reporting as a bug.

> unscale_gradients

I run the same code in the same environment, the difference is the version of transformers.

transformers4.28.1 running normally
![Pasted Graphic 3](https://github.com/huggingface/transformers/assets/39549453/1132d543-e653-457b-8b58-2afbb9599488)

There is a problem with transformers4.31.0
![Pasted Graphic 4](https://github.com/huggingface/transformers/assets/39549453/05271b59-0d82-4dea-8fc3-2fb3b5fc5864)

I found that the problem lies in the  `clip_grad_norm_` function of accelerate. When performing `self.unscale_gradients()` calculation, it seems to support `amp` training. It seems that there is no support for `pure fp16` calculation, but there is no limit. It's weird here, the before and after versions are not compatible. I don't know if my understanding is correct.
![image](https://github.com/huggingface/transformers/assets/39549453/1fde1d6d-603a-4a05-9e55-c32e394d44e0)

@sgugger 


Hello @Baibaifan, you shouldn't explicitly call model.half() or model.to(torch.float16) when using amp. See this PyTorch forum message: https://discuss.pytorch.org/t/valueerror-attemting-to-unscale-fp16-gradients/81372/14

Could you please make sure to remove such lines and rerun and see if that resolves the issue?

> https://discuss.pytorch.org/t/valueerror-attemting-to-unscale-fp16-gradients/81372/14

If I don't want to run under `amp`, I just want to run under `pure fp16`, the 4.28.1 version is fine, but 4.31.0 will report an error, I want to know why? @pacman100 

@Baibaifan we don't support pure fp16 training in the `Trainer` as it doesn't converge. You can use pure `fp16` evaluation with the `--fp16_full_eval` flag.

> @Baibaifan we don't support pure fp16 training in the `Trainer` as it doesn't converge. You can use pure `fp16` evaluation with the `--fp16_full_eval` flag.

OK, thanks. @sgugger