Loading T5ForConditionalGeneration model by TFT5ForConditionalGeneration using from_pt=True
Author: FrozenWolf-Cyber
Pull request: no

### System Info

Ran the codes in Colab [here](https://colab.research.google.com/drive/1CJlQ5rVf5b3BysAnP6Y43I2LWl5ExcT2?usp=sharing)


- `transformers` version: 4.26.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1+cu116 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

 I have ran these code in Colab notebook [here](https://colab.research.google.com/drive/1CJlQ5rVf5b3BysAnP6Y43I2LWl5ExcT2?usp=sharing)


### Who can help?

@Rocketknight1 @ArthurZucker @gante 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


 **Code:**
 
 ```python
from transformers import T5Config, T5ForConditionalGeneration, TFT5ForConditionalGeneration
distill_config = T5Config(d_model=256, d_kv = 32, d_ff=512, num_heads=4, decoder_start_token_id=0)
model = T5ForConditionalGeneration(config=distill_config)
model.save_pretrained("T5-pt")

distill_config = T5Config(d_model=256, d_kv = 32, d_ff=512, num_heads=4, decoder_start_token_id=0)
model = TFT5ForConditionalGeneration(config=distill_config)
model.from_pretrained("T5-pt", from_pt=True)
```

**Output:**
_The following warnings were observed for` t5-small `model too_
```
Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFT5ForConditionalGeneration: ['lm_head.weight', 'encoder.embed_tokens.weight', 'decoder.embed_tokens.weight']
- This IS expected if you are initializing TFT5ForConditionalGeneration from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFT5ForConditionalGeneration from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFT5ForConditionalGeneration were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.
<transformers.models.t5.modeling_tf_t5.TFT5ForConditionalGeneration at 0x7f96b8372dc0>
```

_If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training._

To check in case if the outputs were similar I used the following code to check a sample output:

**Code:**
Pytorch:
```python
tokenizer = AutoTokenizer.from_pretrained('t5-small')

def test(text, model, tokenizer):
    tokenized = tokenizer(text, return_tensors='pt', padding='max_length', truncation=True)

    print(tokenizer.batch_decode(model.generate(tokenized["input_ids"]).tolist(), skip_special_tokens=True))


test("summarize: i got permission to begin a start up company by my own..
", model, tokenizer)
```
TF2.0:
```python
from tensorflow.python.ops.numpy_ops import np_config
np_config.enable_numpy_behavior()

def test(text, model, tokenizer):
    tokenized = tokenizer(text, return_tensors='tf', padding='max_length', truncation=True)

    print(tokenizer.batch_decode(model.generate(tokenized["input_ids"]).tolist(), skip_special_tokens=True))


test("summarize: i got permission to begin a start up company by my own..
", model, tokenizer)
```

Output:
Pytorch: ``` ['cod cod cod cod cod cod cod cod cod cod cod cod cod cod cod cod cod cod cod']```
Tensorflow: ``` ['allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance allowance'] ```

 I have ran these above code in Colab notebook [here](https://colab.research.google.com/drive/1CJlQ5rVf5b3BysAnP6Y43I2LWl5ExcT2?usp=sharing)

### Expected behavior

I have trained a T5ForConditionalGeneration model with a custom config using Pytorch and now I am trying to load it into Tensorflow but some weights of the PyTorch model were not used when initializing the TF 2.0 model T5ForConditionalGeneration. I got the same warning when trying with `t5-small` I have ran these following code in Colab notebook [here](https://colab.research.google.com/drive/1CJlQ5rVf5b3BysAnP6Y43I2LWl5ExcT2?usp=sharing)

I would like to know how to "properly" import a `T5ForConditionalGeneration` model that was trained in Pytorch to `TFT5ForConditionalGeneration`? 


 I have ran these above code in Colab notebook [here](https://colab.research.google.com/drive/1CJlQ5rVf5b3BysAnP6Y43I2LWl5ExcT2?usp=sharing)

Comments:
Hey! Thanks for submitting this issue. 
In T5, the encoder and decoder's embedding tokens are shared, and tied. 
As you can see here in the pytorch modeling code : 
```python 
    _keys_to_ignore_on_load_missing = [
        r"encoder.embed_tokens.weight",
        r"decoder.embed_tokens.weight",
        r"lm_head.weight",
    ]
```
This is because the values stored in `shared.weight` are used for these 3 layers. 
The issue is that these layers get filled after initialisation and are then saved. The warning can be safely ignored as the `shared` layer was properly initialized. 



The outputs of the model should be the same however. I can't reproduce your output, and when I run an inference on my side, the generated tokens are the same. Here is a snippet: 
```python 
>>> from transformers import AutoTokenizer, T5Config, T5ForConditionalGeneration, TFT5ForConditionalGeneration, set_seed
>>> set_seed(0) #set seed for reproducibility 
>>> model = T5ForConditionalGeneration.from_pretrained("t5-small")
>>> model.save_pretrained("Arthur/T5-pt")
>>> tf_model = TFT5ForConditionalGeneration.from_pretrained("Arthur/T5-pt", from_pt=True)
>>> tokenizer = AutoTokenizer.from_pretrained("t5-small", padding='max_length', truncation=True)
>>> inputs = tokenizer("this is a random input", return_tensors="pt")
>>> model.generate(**inputs)
# tensor([[0, 3, 5, 1]])
```

```python 
>>> inputs = tokenizer("this is a random input", return_tensors="tf")
>>> tf_model.generate(**inputs)
# <tf.Tensor: shape=(1, 4), dtype=int32, numpy=array([[0, 3, 5, 1]], dtype=int32)>
```

However it seems that if you use the a default config like the following: 
```python 
>>> from transformers import AutoTokenizer, T5Config, T5ForConditionalGeneration, TFT5ForConditionalGeneration, set_seed
>>> set_seed(0) #set seed for reproducibility 
>>> distill_config = T5Config(d_model=256, d_kv = 32, d_ff=512, num_heads=4, decoder_start_token_id=0)
>>> model = T5ForConditionalGeneration(config=distill_config)
>>> model.save_pretrained("Arthur/T5-pt")
>>> tf_model = TFT5ForConditionalGeneration.from_pretrained("Arthur/T5-pt", from_pt=True)
>>> tokenizer = AutoTokenizer.from_pretrained("t5-small", padding='max_length', truncation=True)
>>> inputs = tokenizer("this is a random input", return_tensors="pt")
>>> model.generate(**inputs)
```
the outputs do not match. This should not be expected

@ArthurZucker  Is there any temporary quick fix to this problem? 

The quickest fix is the following (tested locally) :  `transformers-cli pt-to-tf --model-name "ArthurZ/T5-pt"`. This will make sure the conversion and the hidden states match. Will help you debug if there are any issues. In my case, conversion went well and logits match. 
Use `transformers-cli pt-to-tf --model-name <path_to_checkpoint_on_hub>`. Your model ( and the tokenizer) need to be updated and you need to be logged in using `huggingface-cli login `.
See [here](https://huggingface.co/ArthurZ/T5-pt/discussions/1) for an example of the PR that will be automatically created to your repo

OKay, the issue stems from the fact that the model is `training`. If you make sure that `model.eval()` is done for both, everything is fixed! The following works. 
```python 
>>> from transformers import AutoTokenizer, T5Config, T5ForConditionalGeneration, TFT5ForConditionalGeneration, set_seed
>>> set_seed(0) #set seed for reproducibility 
>>> distill_config = T5Config(d_model=256, d_kv = 32, d_ff=512, num_heads=4, decoder_start_token_id=0)
>>> model = T5ForConditionalGeneration(config=distill_config).eval()
>>> model.save_pretrained("Arthur/T5-pt")
>>> tf_model = TFT5ForConditionalGeneration.from_pretrained("Arthur/T5-pt", from_pt=True)
>>> tokenizer = AutoTokenizer.from_pretrained("t5-small", padding='max_length', truncation=True)
>>> inputs = tokenizer("this is a random input", return_tensors="pt")
>>> model.generate(**inputs)
```

This is an expected behaviour, I think I will just update the documentation to make sure it is clearly stated that this can be a discrepancy. Closing this as fixed, unless you still have problems! 😉 

Thank you for the solution @ArthurZucker :)