How can I convert fairseq checkpoint to huggingface for `XLMProphetModel`?
Author: Doragd
Pull request: no

Recently, I spent a lot of time reproducing the New Title Generation (NTG) experiment result in the original paper with `microsoft/xprophetnet-large-wiki100-cased-xglue-ntg`. However, it seems cannot be reproduced.

Finally, I find the pre-trained model named `microsoft/xprophetnet-large-wiki100-cased` seems to be not the Prophet-X, i.e. Prophet-Multi in the original paper.  Actually, it is a baseline named `Unicoder-FNP` in the original paper.

From the readme from the official repo: https://github.com/microsoft/ProphetNet/tree/master/ProphetNet_Multi
> For ProphetNet-Multi-Wiki100(Baseline Model Unicoder-FNP for XGLUE), it is pretrained with 100 languages Wikipedia data Wiki-100 described in [XGLUE](https://arxiv.org/abs/2004.01401).

![image](https://user-images.githubusercontent.com/26213546/131821740-08028e68-823a-4577-a748-e4dc72ebe46f.png)

**Fortunately, I run the code in the official repo with `fairseq` and reproduced the results.**

**I would like to know if there is a chance to offer a script to convert `fairseq` checkpoint to `huggingface` for `XLMProphetModel`** at your convenience. 

Thanks a lot. @patrickvonplaten @LysandreJik @sgugger 



Comments:
@patrickvonplaten Could you share the script to convert fairseq checkpoint to huggingface for XLMProphetModel? 

Thanks very much!

Hey @ericwtlin,

yes we do have such a script here: https://github.com/huggingface/transformers/blob/master/src/transformers/models/prophetnet/convert_prophetnet_original_pytorch_checkpoint_to_pytorch.py

> Hey @ericwtlin,
> 
> yes we do have such a script here: https://github.com/huggingface/transformers/blob/master/src/transformers/models/prophetnet/convert_prophetnet_original_pytorch_checkpoint_to_pytorch.py

@patrickvonplaten I found this script before. But where is the "transformers_old" dependency? Moreover, from the code, I guess the prophetnet_checkpoint_path should be huggingface style checkpoint, rather than official checkpoint.

`prophet_old = XLMProphetNetForConditionalGenerationOld.from_pretrained(prophetnet_checkpoint_path)`


I tried to import  src/transformers of branch save_old_prophetnet_model_structure as transformers_old, and use official dump (e.g. https://msraprophetnet.blob.core.windows.net/prophetnet/release_checkpoints/prophetnet_multi_wiki100.pt), the code doesn't work: 


Let me clarify, I can run the code, but it says most of the weights of official dump are not loaded.

@patrickvonplaten , could you share us the script to convert fairseq official checkpoint (e.g.  https://msraprophetnet.blob.core.windows.net/prophetnet/release_checkpoints/prophetnet_multi_wiki100.pt) to patrickvonplaten/xprophetnet-large-wiki100-cased-xglue-ntg_old?

Many thanks!

Hey @ericwtlin,

Exactly `transformers_old` actually corresponds to this version of transformers: https://github.com/huggingface/transformers/tree/save_old_prophetnet_model_structure

(very ugly code that I've added there, sorry!)

In general, it's very difficult to ensure that checkpoint conversion scripts stay correct as the original codebase is often very likely to change or model parameter names change, etc...

We could try to debug this together in a google colab - I sadly don't have the time to dig deep into the current checkpoint of microsoft. 

It would be great if you could provide a google colab where you get the error message mentioned above

`https://github.com/huggingface/transformers/tree/save_old_prophetnet_model_structure`
not available now