Training using accelerate and deepspeed with ZeRO results in model weights mismatch
Author: JohnnyRacer
Pull request: no

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am currently trying to use deepspeed to finetune a AutoModelForCausalLM model (facebook/opt1.3b) on a multi-GPU instance with ZeRO optimization with the unmodified `run_clm_no_trainer.py` script from [this blog post on HF](https://huggingface.co/blog/pytorch-fsdp). The model trains correctly but when loading the model using the code snippet below.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b", use_fast=True)
model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b")
model.load_state_dict(torch.load("./opt-1.3b-wikitext/pytorch_model.bin"))

```

It results in an error with the following message below.
```
RuntimeError: Error(s) in loading state_dict for OPTForCausalLM:
        size mismatch for model.decoder.embed_tokens.weight: copying a param with shape torch.Size([50265, 2048]) 
from checkpoint, the shape in current model is torch.Size([50272, 2048]).
        size mismatch for lm_head.weight: copying a param with shape torch.Size([50265, 2048]) from checkpoint, the
shape in current model is torch.Size([50272, 2048]).
```

Which is very confusing since the model does not raise any errors about loading weights during training, even over multiple epochs. I have tried to use a different optimizer as well as trying to disable mixed and half precision but the error still persists. I am unsure if this is a bug or I have something misconfigured, any help would be greatly appreciated.

My ds_config :

```python
{'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'gradient_accumulation_steps': 1, 'zero_optimization': {'stage': 2, 'offload_optimizer': {'device': 'none'}, 'offload_param': {'device': 'none'}, 'stage3_gather_16bit_weights_on_model_save': False}, 'steps_per_print': 'inf', 'fp16': {'enabled': True, 'auto_cast': True}}
```

My training command:
```
accelerate launch run_clm_no_trainer.py \                                                                
--model_name_or_path facebook/opt-1.3b \
--dataset_name  wikitext \
--num_train_epochs 6 \
--block_size 128 \
--output_dir ./opt-1.3b-wikitext
```

### Expected behavior

Models trained using accelerate should be loadable using  `model.load_state_dict`. 

Comments:
Thanks for the report!

Cc @ArthurZucker
The OPT tokenizer does not have the same length (50265) as the model embeddings (50272), which causes problems with all our language modeling fine-tuning scripts where there is an automatic resize of the model embeddings to the tokenizer length.
I'm guessing this is to get to a multiple of 8, but there should be fake tokens in the tokenizer to accommodate that maybe.

@JohnnyRacer If you remove the line [here](https://github.com/huggingface/transformers/blob/c87ae86a8f9f56ae193461fa3db6dc20f80eabe4/examples/pytorch/language-modeling/run_clm_no_trainer.py#L381) in the example you're using, you won't have any problem.

I see. We should probably have all tokenizers and models have the same embed dim? Seems like people often ask the question and it's a bit confusing + could be good for zero shot learning if we have extract fake tokens. 
WDYT? 

In those cases, I think we add fake tokens to the tokenizer. cc @LysandreJik to make sure I'm not saying something wrong.

**Edit:** Actually talked to him and we can fix the example instead. Will make a PR later today.

Should be now fixed by the above PR!