unable to completely load T5 pretrained model; missing/unexpected keys
Author: dhecloud
Pull request: no

# 🐛 Bug

## Information

Model I am using: T5

## To reproduce

```
model, info = T5ForConditionalGeneration.from_pretrained('t5-small',output_loading_info=True)
```
info is 
`
{'missing_keys': ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight'], 'unexpected_keys': ['encoder.block.0.layer.0.layer_norm.bias', 'encoder.block.0.layer.1.layer_norm.bias', 'encoder.block.1.layer.0.layer_norm.bias', 'encoder.block.1.layer.1.layer_norm.bias', 'encoder.block.2.layer.0.layer_norm.bias', 'encoder.block.2.layer.1.layer_norm.bias', 'encoder.block.3.layer.0.layer_norm.bias', 'encoder.block.3.layer.1.layer_norm.bias', 'encoder.block.4.layer.0.layer_norm.bias', 'encoder.block.4.layer.1.layer_norm.bias', 'encoder.block.5.layer.0.layer_norm.bias', 'encoder.block.5.layer.1.layer_norm.bias', 'encoder.final_layer_norm.bias', 'decoder.block.0.layer.0.layer_norm.bias', 'decoder.block.0.layer.1.layer_norm.bias', 'decoder.block.0.layer.2.layer_norm.bias', 'decoder.block.1.layer.0.layer_norm.bias', 'decoder.block.1.layer.1.layer_norm.bias', 'decoder.block.1.layer.2.layer_norm.bias', 'decoder.block.2.layer.0.layer_norm.bias', 'decoder.block.2.layer.1.layer_norm.bias', 'decoder.block.2.layer.2.layer_norm.bias', 'decoder.block.3.layer.0.layer_norm.bias', 'decoder.block.3.layer.1.layer_norm.bias', 'decoder.block.3.layer.2.layer_norm.bias', 'decoder.block.4.layer.0.layer_norm.bias', 'decoder.block.4.layer.1.layer_norm.bias', 'decoder.block.4.layer.2.layer_norm.bias', 'decoder.block.5.layer.0.layer_norm.bias', 'decoder.block.5.layer.1.layer_norm.bias', 'decoder.block.5.layer.2.layer_norm.bias', 'decoder.final_layer_norm.bias'], 'error_msgs': []}
`
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

No keys should be missing or unexpected
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.7.0
- Platform: Ubuntu
- Python version: 3.6
- PyTorch version (GPU?): 1.2.0 (yes)
- Tensorflow version (GPU?): nope
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: nope


Comments:
Hi @dhecloud, 

Thanks for you issue :-) 
Does the model still work fine? 

> Hi @dhecloud,
> 
> Thanks for you issue :-)
> Does the model still work fine?

Hi, thanks for your reply.
Using the examples provided in the doc, the model works fine. 
Before i used `T5WithLMHeadModel` in version `2.5.1` which did not raise this missing keys warning.  After i moved to `T5ForConditionalGeneration` in `2.7.0` there was this warning and my training loss diverged so i thought i might raise this issue in case there was some sort of change in naming in the checkpoint

I'm gonna take a look :-) 

Hi guys, 
Any news on this?
When I try to load t5-base I receive this:


INFO:transformers.modeling_utils:Weights of T5ForConditionalGeneration not initialized from pretrained model: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']

> Hi guys,
> Any news on this?
> When I try to load t5-base I receive this:
> 
> INFO:transformers.modeling_utils:Weights of T5ForConditionalGeneration not initialized from pretrained model: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']

i think it's mostly a harmless misplaced error. The model should still work fine. You can test it by trying out the examples 

Yeah this should not be a problem, all these weights are `[encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']` are weights tied to the input embedding matrix and therefore don't need to be initialized.

How can we silence the error?

It should be enough to lower the cli-logger

@sgugger @patrickvonplaten this seems like a very old post, but I just got this warning:

```
[WARNING|trainer.py:2231] 2023-11-23 23:20:29,636 >> There were missing keys in the checkpoint model loaded: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight'].                                                                    
```

 on `4.35.2` trying to restart my training from ckpt after my training run crashed. I'm working with FLAN-T5 base. I restarted by passing `resume_from_checkpoint=True` to the `Trainer`.
 
 I think there was recently an attempt to migrate to `safetensors` so maybe that's why the warning crept in there? Also, rather strangely, the progress training bar started from `0` steps ... I do remember that the `Trainer` used to forward the data loader to the point where you left off when you stopped training and such tricks, but nothing really indicates that this is happening or that indeed I am restarting from my checkpoint ...
 
 UPDATE: Just reading through the `ignore_data_skip` docs, I'm pretty sure that I haven't started training from the checkpoint, training started rather quickly so I betting the farm on no batch being skipped...

UPDATE 2: I should expect that to skip batches the trainer state would have to be saved (right?). So given that my code failed at `L2407` (so during `_save_checkpoint`) because I mistakenly passed a metric that I was not calculating as the metric for best model, I presume this elucidates the mystery of not skipping batches. The question remains, has the training started from the checkpoint or not?


 

> Yeah this should not be a problem, all these weights are `[encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']` are weights tied to the input embedding matrix and therefore don't need to be initialized.

Hello, what do you mean are "tied to the input"?  
  
I am trying to convert a flax finetuned T5 model into pytorch format using these commands:
    - `from_flax_model = T5ForConditionalGeneration.from_pretrained(model_path, from_flax=True)`
    - `from_flax_model.save_pretrained(folder)`
 
and I get the same message: `Some weights of T5ForConditionalGeneration were not initialized from the Flax model and are newly initialized: ['decoder.embed_tokens.weight', 'lm_head.weight', 'encoder.embed_tokens.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.`