[T5] enable T5 fp16
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR enables fp16 for T5 models, by clamping hidden states to the max value of the current data type.

As detailed in #9295, T5 produces large (`inf`) activations at 3 places
1. Output of `T5LayerFF`
2. Output of `T5LayerSelfAttention`
3. Output of `T5LayerCrossAttention`

To avoid these `inf` activations this PR clamps the `hidden_states` after above 3 outputs




Comments:
This is great!

Dear @patil-suraj 
Your PR works well for t5 model, thank you for your work.
But now I tried new t5 model version released recently by Google: google/t5-v1_1-xl
The same code after loading google/t5-v1_1-xl instead of t5-3b is going to return a lot "overflow" errors.

Can you tell me, should your code fix fp16 on google/t5-v1_1-xl model? 
Here is training code:
https://github.com/exelents/try_t5_qa
Run ./run-qa-3b.sh

Upd: I run my code on Transformers's branch from your current PR #9487 merged with PR #9211 needed for deepspeed integration.
Can you confirm a problem, or it's just mine?

> Dear @patil-suraj
> Your PR works well for t5 model, thank you for your work.
> But now I tried new t5 model version released recently by Google: google/t5-v1_1-xl
> The same code after loading google/t5-v1_1-xl instead of t5-3b is going to return a lot "overflow" errors.
> 
> Can you tell me, should your code fix fp16 on google/t5-v1_1-xl model?
> Here is training code:
> https://github.com/exelents/try_t5_qa
> Run ./run-qa-3b.sh
> 
> Upd: I run my code on Transformers's branch from your current PR #9487 merged with PR #9211 needed for deepspeed integration.
> Can you confirm a problem, or it's just mine?

Hey @exelents, can you include a code snippet to reproduce your error as well as the full stack trace of your error?

@patrickvonplaten , @exelents 

as stated in #9432 

This fix works for following models and versions, with apex `01` and `native amp`
- T5v1: t5-small, t5-base, t5-large
- T5v1_1: google/t5-v1_1-small, google/t5-v1_1-base
- MT5: google/mt5-small, google/mt5-base

Just did a small experiment with `t5-v1_1-large` and it still gives `nan` loss after 200 steps, so might not work for `xl`, 

also, @exelents  by overflow error do you mean the gradient overflow warning thrown by `apex` ?

> @patrickvonplaten , @exelents
> 
> as stated in #9432
> 
> This fix works for following models and versions, with apex `01` and `native amp`
> 
> * T5v1: t5-small, t5-base, t5-large
> * T5v1_1: google/t5-v1_1-small, google/t5-v1_1-base
> * MT5: google/mt5-small, google/mt5-base
> 
> Just did a small experiment with `t5-v1_1-large` and it still gives `nan` loss after 200 steps, so might not work for `xl`,
> 
> also, @exelents by overflow error do you mean the gradient overflow warning thrown by `apex` ?

Ah ok, we still see `nan's` with `t5-v1_1-large` then :-/ Do you think this could be fixed by adding one more clamp statement? @patil-suraj 

> Hey @exelents, can you include a code snippet to reproduce your error as well as the full stack trace of your error?
My code is here:
https://github.com/exelents/try_t5_qa
It requires deepspeed to run, as well as code from #9211 PR (deepspeed integration) be merged. Use run-qa-3b.sh to test.

Here is error stack:
https://gist.github.com/exelents/10f1d03e61059ddf2dfba7068114c93a
Look at the end - we have a message after every step:
`[2021-01-11 16:58:18,163] [INFO] [stage2.py:1361:step] [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 256.0, reducing to 128.0`
Wait a second, I'll try to check loss value tensor.

> Do you think this could be fixed by adding one more clamp statement?

I'm again trying to locate where exactly in the model this happen. In case it's the same as above (first `inf` then `nan` ) then we could fix it by adding one more clamp

I have checked a loss value, and it seems in is not NaN. It got values like "48.7500" or "40.9688" but there are vaild values. Despite that I see messages like "OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1024.0, reducing to 512.0", that it seems means that something bad happened with model's loss.

> Attempted loss scale: 1024.0, reducing to 512.0", that it seems means that something bad happened with model's loss.

Those warnings don't mean anything went wrong, it's logical with dynamic loss scaling that some loss scale values are too big at the beginning of training.