non-negligible difference between GPT2 and TFGP2
Author: ydshieh
Pull request: no

## Environment info
- `transformers` version: 4.11.0.dev0
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.9.5
- PyTorch version (GPU?): 1.9.0+cpu (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Models:
- gpt2: @patrickvonplaten, @LysandreJik

## Information
(I provide a potential fix at the end)

Model I am using: TFGPT2Model

In the current TFGPT2 model file, the activation used in `class TFMLP` is fixed to be `gelu`, as shown in the following line

https://github.com/huggingface/transformers/blob/ea92136597c49a20c5e2c31ef20ccec1693a8858/src/transformers/models/gpt2/modeling_tf_gpt2.py#L177

However, in PyTorch version GPT2, it is set to `ACT2FN[config.activation_function]`, see

https://github.com/huggingface/transformers/blob/ea92136597c49a20c5e2c31ef20ccec1693a8858/src/transformers/models/gpt2/modeling_gpt2.py#L278

I checked the git history, and found in the earlier versions of `modeling_tf_gpt2.py`, the function `gelu` is defined within the model file itself, but it is in fact the current version's `gelu_new`. 

[Early version gelu]
https://github.com/huggingface/transformers/blob/1487b840d3457bf8b0f1fcacd02d3a2fae407fe5/src/transformers/modeling_tf_gpt2.py#L46

[Early version gelu being used]
https://github.com/huggingface/transformers/blob/1487b840d3457bf8b0f1fcacd02d3a2fae407fe5/src/transformers/modeling_tf_gpt2.py#L164

[Current gelu_new]
https://github.com/huggingface/transformers/blob/ea92136597c49a20c5e2c31ef20ccec1693a8858/src/transformers/activations_tf.py#L34

This causes larger difference between `TFGPT2` and `GPT2` (I observed this when worked on #13222).

## To reproduce

```
import numpy as np
import torch
import tensorflow as tf
from transformers import GPT2Model, TFGPT2Model


pt_gpt2 = GPT2Model.from_pretrained('gpt2').to('cpu')
tf_gpt2 = TFGPT2Model.from_pretrained('gpt2')

input_ids = [list(range(13))]
pt_input_ids = torch.tensor(input_ids, dtype=torch.int32).to('cpu')
tf_input_ids = tf.constant(input_ids, dtype=tf.int32)

pt_out = pt_gpt2(pt_input_ids, output_hidden_states=True)
tf_out = tf_gpt2(tf_input_ids, output_hidden_states=True)

for idx, (pt_h, tf_h) in enumerate(zip(pt_out.hidden_states, tf_out.hidden_states)):

    layer_name = 'embedding layer' if idx == 0 else f'hidden layer {idx}'

    pt_h = pt_h.detach().numpy()
    tf_h = tf_h.numpy()

    print(f"difference within {layer_name}: {np.max(np.abs(pt_h - tf_h))}")

```
gives the outputs
```
difference within embedding layer: 0.0
difference within hidden layer 1: 0.01474761962890625
difference within hidden layer 2: 0.0545654296875
difference within hidden layer 3: 0.09014892578125
difference within hidden layer 4: 0.109375
difference within hidden layer 5: 0.09521484375
difference within hidden layer 6: 0.093017578125
difference within hidden layer 7: 0.0931396484375
difference within hidden layer 8: 0.09295654296875
difference within hidden layer 9: 0.09246826171875
difference within hidden layer 10: 0.09307861328125
difference within hidden layer 11: 0.09442138671875
difference within hidden layer 12: 0.10050582885742188
```

## Expected behavior

The difference should be even smaller (< 1e-3).


## Potential fix

Change 

https://github.com/huggingface/transformers/blob/ea92136597c49a20c5e2c31ef20ccec1693a8858/src/transformers/models/gpt2/modeling_tf_gpt2.py#L177

to `self.act = get_tf_activation(config.activation_function)`.

Rerun the above debugging script gives:

```
difference within embedding layer: 0.0
difference within hidden layer 1: 3.0517578125e-05
difference within hidden layer 2: 6.103515625e-05
difference within hidden layer 3: 0.000732421875
difference within hidden layer 4: 0.00048828125
difference within hidden layer 5: 0.000244140625
difference within hidden layer 6: 0.000244140625
difference within hidden layer 7: 0.0001220703125
difference within hidden layer 8: 0.0001220703125
difference within hidden layer 9: 6.103515625e-05
difference within hidden layer 10: 6.103515625e-05
difference within hidden layer 11: 7.62939453125e-05
difference within hidden layer 12: 7.62939453125e-05
```

But this probably will break users' fine-tuned models?

**[Extra question]**: Currently, is there some enforced check for each model between its different versions' (PyTorch/TF/Flax) results?

Comments: