Add Gated-SiLU to T5
Author: DanielHesslow
Pull request: yes

# What does this PR do?

Adds gated SiLU to the t5 model in order to support recently the released UL2 model: https://github.com/google-research/google-research/tree/master/ul2

@patrickvonplaten, @patil-suraj


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hmm, yeah so I think I understand what you want to do, but I really don't understand why.

for back compatibility we want:
- non-gated activation function if you specify `feed_forward_proj='relu'`
- gated activation function if you specify `feed_forward_proj='gated-gelu'`
Both my current version and with your modifications this is handled correctly.

For new activation functions:
my version:
- you get the activation function you specify in `feed_forward_proj`
- you get a gated activation function if you specify `is_gated=True`

your version:
- you always get a gated activation function if you specify `dense_act_fn`

I don't see why it's better to 1. change the parameter name where you specify the activation function 2. not support new non-gated activation functions. 


```
- you always get a gated activation function if you specify dense_act_fn
```
no that was not the logic. The logic was to only get a "gated" feed forward when you specify `feed_forward_proj="gated-gelu"`, but maybe that's too complicated then here actually. 

New (better) idea maybe:
How about we just add a new `feed_forward_proj="gated-silu"` and then you extract if the model should be gated or not with:
```py
is_gated = feed_forward_proj.split("-")[0] == "gated"
```
and the activation function with:
```py
act_fn = feed_forward_proj.split("-")[-1]
```

maybe that's the cleanest actually

This way no dup code, no need for an additional config attribute and it's fairly clean

Alright, should be mostly in order now. I do agree that it's a bit cleaner to not introduce more new parameters.

Getting a few (as far as I can tell) unrelated tests erroring out, with protobuff problems:

```
E     TypeError: Descriptors cannot not be created directly.
E     If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E     If you cannot immediately regenerate your protos, some other possible workarounds are:
E      1. Downgrade the protobuf package to 3.20.x or lower.
E      2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
```

But otherwise I guess things are ok. 

@DanielHesslow - great the solution works well for me - thanks for making the changes. Left 1 suggestion to improve the error message a bit, but besides that all good to me.

@DanielHesslow,

The failing CI tests are because of TF releasing a new protobuffer version which broke our CI. You could solve this by rebasing your branch to main (or just pull main into your branch).

Once your branch is up to date, the CI tests should work again :-)

```
git pull origin main
git push
```

Thanks a lot for your work here!

Okay, fixed the error message and rebased onto main, so all should be good now I believe. 

Merging now