Clamping hidden state values to allow FP16
Author: SSamDav
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
Following the discussion in [#9295](https://github.com/huggingface/transformers/issues/9295) and the solution proposed by  [#9487](https://github.com/huggingface/transformers/pull/9487). 

Implement the solution that enables the FP16 for LongT5 models.

Who can review:
@patrickvonplaten, @patil-suraj

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi, it seems that I have a test error, however, I didn't change the code that is falling. 
Does anyone know how I can pass these tests?

Thanks @SSamDav.

And sorry, I forgot to mention in order to avoid the test failure you saw yesterday, you can rebase the working branch on `main`. (If it still appears in the new run)

Ok for me if we know that it helps for inference. In T5 it didn't really help for training at all in the end, so if this was implemented to enable training in fp16, I'm not sure it's a good idea (I think @patil-suraj won't have time to look into it btw).

Also cc @ArthurZucker here just FYI

Since @patrickvonplaten prefers to have an issue where this PR will solve, I think we are not going to merge this PR at this moment. Let's see if there will be such issues reported for `LongT5` in the future. We can make an investigation and decide if to re-open/merge this PR (or with a different fix). WDYT? cc @younesbelkada @ArthurZucker .

----

And regarding `nan`: My though is that it's most likely coming from the sequences with all `-inf` after adding the mask to the attention scores, and `nan` after `softmax`, like what we observed in OPT or  Bloom recently, where we provided a fix `as close as to` where `nan` happens. (However, in the case of `T5`, a clamp is done `T5LayerFF` which is not attention-related).

I my tests when I run a finetuned version of the `google/long-t5-tglobal-base` in FP16 I got `Nan`  in the forward step, I could check if the values com from the `LongT5LayerFF`. 


> I my tests when I run a finetuned version of the `google/long-t5-tglobal-base` in FP16 I got `Nan` in the forward step, I could check if the values com from the `LongT5LayerFF`.

Hi, so it is running the inference, right? Is that finetuned checkpoint uploaded to Hub?

> Hi, so it is running the inference, right?
 
Yes

>  Is that finetuned checkpoint uploaded to Hub?

No, it was trained in confidential data.

> > Is that finetuned checkpoint uploaded to Hub?
> 
> No, it was trained in confidential data.

Got it. However, it would be really nice if we have a public available checkpoint (on another dataset) that can show the issue and the effect of the fix. I understand that it may not easy to obtain another such checkpoint - and potentially time consuming.

@patrickvonplaten Any further comment?

Hi!
I second what @ydshieh said, probably the root cause of this is happening inside the attention score computation as it has been observed for BLOOM and OPT - maybe it's worth investigating a bit before merging! 
As a simple test, we could try to reproduce what has been done in #18057 & #17437 and see if this fixes the initial issue

Hey @SSamDav,

If the PR as is now solves your problem for **inference** - it's good for me to merge! I don't think it'll fix problems with fine-tuning though

Hey @patrickvonplaten, good thanks for the help!