Fix t5 failures
Author: Abdennacer-Badaoui
Pull request: yes

Fix all the remaining T5 failures. 

- The test `test_export_t5_summarization` was failing because `eos_token_id` was not being passed to GenerationConfig in `Seq2SeqLMExportableModule`, causing the generate loop to never stop at the end-of-sequence token and produce extra tokens. 
- The `test_fp16_fp32_conversion` test was failing because T5's `_keep_in_fp32_modules = ["wo"]` was unconditionally forcing the `wo` layer to FP32 for both FP16 and BF16 loading, so I created a centralized `_build_dtype_plan_for_loading()` method that encapsulates all the `dtype_plan` building logic in one place, taking the target dtype as input and properly applying `_keep_in_fp32_modules` (FP16 only) and `_keep_in_fp32_modules_strict` (FP16 and BF16) based on the actual loading `dtype`.
- Fix some expectations (tested for both AMD and Nvidia)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_43374). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is weird because we explicitly have `_keep_in_fp32_modules` and `_keep_in_fp32_modules_strict` - the later being for bf16 as well, the former only for fp16 

Something must have changed for loading models

That's true. I think I found the problem. 
So the `dtype_plan` initialization in `PreTrainedModel.__init__` was unconditionally adding `_keep_in_fp32_modules` entries (intended for FP16 only) even when loading with BF16, so I will fix the core loading logic in `modeling_utils.py` to only add `_keep_in_fp32_modules` to `dtype_plan` when `dtype == torch.float16`, respecting the intended distinction between `_keep_in_fp32_modules` (FP16 only) and `_keep_in_fp32_modules_strict` (FP16 and BF16).

I created a centralized `_build_dtype_plan_for_loading()` method that encapsulates all the` dtype_plan` building logic in one place, taking the target `dtype` as input and        properly applying `_keep_in_fp32_modules` (FP16 only) and `_keep_in_fp32_modules_strict` (FP16 and BF16) based on the actual loading `dtype`, replacing my previous fragmented logic.


> The test_fp16_fp32_conversion test was failing because T5's _keep_in_fp32_modules = ["wo"] was unconditionally forcing the wo layer to FP32 for both FP16 and BF16 loading (even though BF16 is numerically stable enough and doesn't need FP32), so we added a post_init() override in T5PreTrainedModel that checks torch.get_default_dtype() and removes wo from dtype_plan when loading with bfloat16, allowing wo to stay in FP32 only for FP16 loading while using BF16 for BF16 loading.

What kind of failures we have here? I mean could we just adjust the test logic (if that makes sense)? To change the logic in `modeling_utils.py`, it's best to make good and precise definitions of what those flags should do. So my question is: do the currently logic (on `main`) around what you changed in `modeling_utils.py` is wrong itself, or the change is mainly to pass the test (which has a flaw testing logic and to be adjusted)?




> > The test_fp16_fp32_conversion test was failing because T5's _keep_in_fp32_modules = ["wo"] was unconditionally forcing the wo layer to FP32 for both FP16 and BF16 loading (even though BF16 is numerically stable enough and doesn't need FP32), so we added a post_init() override in T5PreTrainedModel that checks torch.get_default_dtype() and removes wo from dtype_plan when loading with bfloat16, allowing wo to stay in FP32 only for FP16 loading while using BF16 for BF16 loading.
> 
> What kind of failures we have here? I mean could we just adjust the test logic (if that makes sense)? To change the logic in `modeling_utils.py`, it's best to make good and precise definitions of what those flags should do. So my question is: do the currently logic (on `main`) around what you changed in `modeling_utils.py` is wrong itself, or the change is mainly to pass the test (which has a flaw testing logic and to be adjusted)?



The current logic in `modeling_utils.py` on main seems to be wrong, it doesn't respect the documented distinction between `_keep_in_fp32_modules` and `_keep_in_fp32_modules_strict`.
   According to the documentation, `_keep_in_fp32_modules` should "avoid casting to anything other than float32, except bfloat16" (only prevent FP16 casting), while `_keep_in_fp32_modules_strict` is meant to prevent both FP16 and BF16 casting. However, the current implementation in `__init__` unconditionally adds both  
  to `dtype_plan` during initialization, before knowing the target dtype, which means `_keep_in_fp32_modules` incorrectly forces modules to FP32 even for BF16 loading. This causes T5's `wo` layer to stay in FP32 when loading with BF16 (when using `_keep_in_fp32_modules = ["wo"]`). The test ( `tests/models/t5/test_modeling_t5.py::T5ModelFp16Tests::test_fp16_fp32_conversion` ) is correctly catching this bug by expecting wo to be BF16 when loading with dtype=torch.bfloat16.    
                                                                                                                                                                            

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43374&sha=030d5e

Re: keeping modules in fp32, see my comment https://github.com/huggingface/transformers/pull/43374#issuecomment-3774271277

I'm pretty sure this behavior is wrong and was only recently changed - iirc @eustlb added the strict flag, maybe cc @Cyrilvallez for model loading?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: t5

Force merging as failures are completely unrelated!