[s2s] Can't mix --fp16 and --device cpu
Author: stas00
Pull request: yes

This PR fixes this user-side error:

```
RuntimeError: "threshold_cpu" not implemented for 'Half'
```
reported at https://github.com/huggingface/transformers/issues/10040

This combination `--fp16 --device cpu` is not possible, as explained here: https://github.com/pytorch/pytorch/issues/48245#issuecomment-730714723
and it's not really usable anyway - it takes minutes to run fp16 on cpu while it takes a split second on the gpu.

full trace:
```
export DATA_DIR=wmt_en_ro; PYTHONPATH=../../src ./run_eval.py t5-base     $DATA_DIR/val.source t5_val_generations.txt     --reference_path $DATA_DIR/val.target     --score_path enro_bleu.json     --task translation_en_to_ro     --n_obs 100     --device cpu     --fp16     --bs 32

Traceback (most recent call last):
  File "./run_eval.py", line 176, in <module>
    run_generate(verbose=True)
  File "./run_eval.py", line 137, in run_generate
    runtime_metrics = generate_summaries_or_translations(
  File "./run_eval.py", line 67, in generate_summaries_or_translations
    summaries = model.generate(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/generation_utils.py", line 847, in generate
    model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/generation_utils.py", line 379, in _prepare_encoder_decoder_kwargs_for_generation
    model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 946, in forward
    layer_outputs = layer_module(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 683, in forward
    hidden_states = self.layer[-1](hidden_states)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 299, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-cpu-no-fp16/src/transformers/models/t5/modeling_t5.py", line 258, in forward
    hidden_states = F.relu(hidden_states)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/functional.py", line 1206, in relu
    result = torch.relu(input)
RuntimeError: "threshold_cpu" not implemented for 'Half'
```

@sgugger, @patil-suraj 

Fixes: https://github.com/huggingface/transformers/issues/10040

Comments:
I'm not following,  `finetune_trainer.py` is replaced by `run_seq2seq.py`.

What replaces these 3?
```
run_distributed_eval.py
run_eval.py
run_eval_search.py
```

Not familiar with `run_eval_search` but the new `run_seq2seq` is supposed to run the right evaluation (so no need for `run_eval`) and can do it in a distributed fashion (so no need for `run_eval_distributed`). But I may have missed something.

`run_eval_search` uses `run_eval` to find the best hparams. It's not an example, but a real tool to pick the best initial model config hparams when a new model is ported. Written by yours truly.

So it probably needs to be ported to use `run_seq2seq` then.