All Flan-T5 models configs use the incorrect activation function
Author: michaelroyzen
Pull request: no

### System Info

The [configs](https://huggingface.co/google/flan-t5-xxl/blob/main/config.json) for all of the Flan-T5 says that the activation function is 'gelu' and yet 'is_gated_act' is set to true. This is an inherent contradiction.

Doing more digging, I realized that per [Google's original Flan-T5 checkpoints ](https://github.com/google-research/t5x/blob/main/docs/models.md#t5-11-lm-adapted-checkpoints), Flan-T5 is directly instantiated from T5v1.1 LM-adapt, which all use [gated-gelu](https://huggingface.co/google/t5-xxl-lm-adapt/blob/main/config.json)

### Who can help?

@younesbelkada @arthur

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Compare the T5v1.1+LM configs to the Flan-T5 configs.

### Expected behavior

"feed_forward_proj" should be "gated-gelu" and "dense_act_fn" is redundant and should be removed entirely from the config.

Comments:
Hi @michaelroyzen 
Thanks for raising this.
You are right, one should use `gated-gelu` as it is done in  t5 LM-adapt checkpoints. We have updated with @ArthurZucker the config files of flan-T5 models. 
Note that forcing `is_gated_act` to `True` leads to using gated activation function too. The only difference between these 2 approaches is that using `gated-gelu` forces the model to use `gelu-new` activation function. See [this line](https://github.com/huggingface/transformers/blob/a00b7e85ea4b3e3185440f1f82a6b58e3660b01d/src/transformers/models/t5/configuration_t5.py#L132). `gelu-new` gives slightly different results than `gelu` but does not affect the overall performance of the model. 
This also is not a breaking change from what I can see, as it affects only inference of the model. If someone has fine-tuned flan-T5 with `gelu` should not be affected by this change.
Closing this issue as the config files have been fixed. Thanks for your help!

cc @sgugger

Thank you @younesbelkada. May I ask how this only affects inference? If a flan-t5 model was fine-tuned with the old checkpoint, wouldn't the wrong activations be used?

> Note that forcing `is_gated_act` to `True` leads to using gated activation function too.

@younesbelkada From this line of code, it seems that is_gated_act will be set to false without gated-relu? https://github.com/huggingface/transformers/blob/a00b7e85ea4b3e3185440f1f82a6b58e3660b01d/src/transformers/models/t5/configuration_t5.py#L122

@LiJunnan1992 
No, it gets overriden by the kwargs [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/configuration_t5.py#L135), check this snippet:
```
from transformers import T5Config

config_gated = T5Config(is_gated_act=True, hidden_act="gelu")
print(config_gated.is_gated_act)
>>> True

config_gated = T5Config(hidden_act="gelu")
print(config_gated.is_gated_act)
>>> False

config_gated = T5Config(feed_forward_proj="gated-gelu")
print(config_gated.is_gated_act)
>>> True
```

@younesbelkada I see. Thanks so much for the explanation!