RuntimeError: Error(s) in loading state_dict for the converted MarianMTModel due to the source vocab not different with target vocab
Author: Kitty-Bert-Mom
Pull request: no

### System Info

WARNING:tensorflow:From d:\git\transformers\src\transformers\commands\env.py:100: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-09-22 17:41:23.252507: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical opera
tions:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.34.0.dev0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.9.10
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 1.8.1+cpu (False)
- Tensorflow version (GPU?): 2.9.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker  @younesbelkada @gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### **Errors I faced:**
RuntimeError: Error(s) in loading state_dict for MarianMTModel:
	size mismatch for final_logits_bias: copying a param with shape torch.Size([1, 52]) from checkpoint, the shape in current model is torch.Size([1, 36]).
	size mismatch for model.decoder.embed_tokens.weight: copying a param with shape torch.Size([52, 256]) from checkpoint, the shape in current model is torch.Size([36, 256]).
	You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.


### **How to repro the issue:**
run https://github.com/huggingface/transformers/blob/b880508440f43f80e35a78ccd2a32f3bde91cb23/src/transformers/models/marian/convert_marian_to_pytorch.py

the model I want to convert is the one I trained myself ( tied-embeddings-src is set as false which indicates **source vocab and target vocab is not shared** )




### **Root cause:**

for my model, source vocab has 36 words. the target vocab has 52 words.  Current python code treats the source vocab as the target vocab. That's why the errors said copying a param with shape torch.Size(**[52,** 256]) from checkpoint, the shape in current model is torch.Size(**[36,** 256]).

I saw the comment in the marianMTModel, there is an assumption for the model, that  the source and target vocab should be shared. But the assumption is not suitable for all marian models. at least the model I trained is not shared. (I can not retrain the model from scratch due to some reason)


### Expected behavior

update the MarianMTModel to support the source vocab and target vocab could be different.

Comments:
Hey! I'll have a look 😉 

> Hey! I'll have a look 😉

Hi, Arthur, I create a PR to fix the issue. Could you please help me review it? 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I reviewed the PR which is stale, if there is a strong need from the community I can take it over 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.