fp16_full_eval argument/flag in training arguments does not increase runtime or decrease memory footprint
Author: garg-aayush
Pull request: no

### System Info

I am running a simple benchmarking test to see the speedup and accuracy changes that we see when we switch to `float16` (full float16, not mixed precision). For this purpose, I am using the example [semantic segmentation example script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/semantic-segmentation/run_semantic_segmentation.py)

This is how I am running code 
for float32
```bash
python run_semantic_segmentation.py  --model_name_or_path nvidia/mit-b0 
 --dataset_name segments/sidewalk-semantic  --output_dir ./segformer_outputs/  
 --remove_unused_columns False     --do_train     --evaluation_strategy steps 
 --push_to_hub  --push_to_hub_model_id segformer-finetuned-sidewalk-10k-steps  
 --max_steps 10000  --learning_rate 0.00006  --lr_scheduler_type polynomial  
 --per_device_train_batch_size 8  --per_device_eval_batch_size 8  --logging_strategy steps 
 --logging_steps 100 --evaluation_strategy epoch  --save_strategy epoch  --seed 1337
```

for full float16
```bash
python run_semantic_segmentation.py  --model_name_or_path nvidia/mit-b0 
 --dataset_name segments/sidewalk-semantic  --output_dir ./segformer_outputs/  
 --remove_unused_columns False     --do_train     --evaluation_strategy steps 
 --push_to_hub  --push_to_hub_model_id segformer-finetuned-sidewalk-10k-steps  
 --max_steps 10000  --learning_rate 0.00006  --lr_scheduler_type polynomial  
 --per_device_train_batch_size 8  --per_device_eval_batch_size 8  --logging_strategy steps 
 --logging_steps 100 --evaluation_strategy epoch  --save_strategy epoch  --seed 1337
 --fp16_full_eval
```

For both I am getting, `~6.9 GB` gpu memory and `~6it/s`

I don't think that should be case. For `fp16_full_eval`, there should be some speedup. 


System info:
```
transformers version: 4.31.0.dev0
python: 3.8.16
GPU: RTX4090
```

### Who can help?

@sgugger @amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

for float16 full run

```bash
python run_semantic_segmentation.py  --model_name_or_path nvidia/mit-b0 
 --dataset_name segments/sidewalk-semantic  --output_dir ./segformer_outputs/  
 --remove_unused_columns False     --do_train     --evaluation_strategy steps 
 --push_to_hub  --push_to_hub_model_id segformer-finetuned-sidewalk-10k-steps  
 --max_steps 10000  --learning_rate 0.00006  --lr_scheduler_type polynomial  
 --per_device_train_batch_size 8  --per_device_eval_batch_size 8  --logging_strategy steps 
 --logging_steps 100 --evaluation_strategy epoch  --save_strategy epoch  --seed 1337
 --fp16_full_eval
```

### Expected behavior

I would have expected some speed up in terms of more iterations/s for `fp16_full_eval`

Comments:
From the command you are pasting, you are not doing any evaluation (`--do_eval` is not set to True). `--fp16_full_eval` is ignored during training, so your model will still take the same place and the same speed during training. It's just for evaluation.

@sgugger Thanks for replying. Is there any way to run training in full `float16` rather than mixed-precision `float16`? 
Something similar to lightning `Fabric` where you have two separate options of running training, for fp16 use `precision="16"` and for mixed-fp16 use `precision="mixed-16"`

No there is none in the `Trainer`, since training in float16 does not converge in most cases.

Thanks