ProphetNet
Author: qiweizhen
Pull request: yes
Labels: name: model card

# Add [ProphetNet](https://arxiv.org/abs/2001.04063).

This PR implements both ProphetNet and XLM-ProphetNet. The model architectures are identical, but each model uses a different tokenizer.

## Description:

ProphetNet is a new pre-trained language model for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. ProphetNet is able to predict more future tokens with an n-stream decoder. The original implementation is Fairseq version at [github repo](https://github.com/microsoft/ProphetNet).   
xProphetNet has the same model structure but is pretrained with wikipedia 100 languages dataset as described in [xGLUE](https://arxiv.org/abs/2004.01401). xGLUE is a benchmark for cross-lingual NLU and NLG tasks. xProphetNet is also served as a baseline model for cross-lingual generation tasks in xGLUE NTG and QG.

## Usage:

Take xGLUE NTG task as an example:  
The cross-lingual pretrained model is finetuned with English news title generation data, but inference with both English and other zero-shot language data.  
A quick usage is like: 
```
from transformers import ProphetNetTokenizer, ProphetNetForConditionalGeneration, ProphetNetConfig

model = ProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-ntg')
tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-ntg')

EN_SENTENCE_TO_QUESTION = "Microsoft Corporation intends to officially end free support for the Windows 7 operating system after January 14, 2020, according to the official portal of the organization. From that day, users of this system will not be able to receive security updates, which could make their computers vulnerable to cyber attacks."
RU_SENTENCE_TO_QUESTION = "орпорация Microsoft намерена официально прекратить бесплатную поддержку операционной системы Windows 7 после 14 января 2020 года, сообщается на официальном портале организации . С указанного дня пользователи этой системы не смогут получать обновления безопасности, из-за чего их компьютеры могут стать уязвимыми к кибератакам."
ZH_SENTENCE_TO_QUESTION = "根据该组织的官方门户网站，微软公司打算在2020年1月14日之后正式终止对Windows 7操作系统的免费支持。从那时起，该系统的用户将无法接收安全更新，这可能会使他们的计算机容易受到网络攻击。"
inputs = tokenizer([EN_SENTENCE_TO_QUESTION, RU_SENTENCE_TO_QUESTION, ZH_SENTENCE_TO_QUESTION], padding=True, max_length=256, return_tensors='pt')

summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=100, early_stopping=True)
print([tokenizer.decode(g) for g in summary_ids])  
```
Model will generate news titles like:
```
['[SEP] Microsoft to end Windows 7 free support after January 14, 2020[SEP][PAD][PAD][PAD][PAD]',
 '[SEP] Microsoft намерена прекратить бесплатную поддержку Windows 7 после 14 января 2020 года[SEP]',
 '[SEP]微软打算终止对Windows 7操作系统的免费支持[SEP][PAD][PAD][PAD][PAD][PAD][PAD]']
```

## Released checkpoints:

pretrained:
```
microsoft/prophetnet-large-uncased
microsoft/xprophetnet-large-wiki100-cased
```
fine-tuned:
```
microsoft/prophetnet-large-uncased-cnndm
microsoft/xprophetnet-large-wiki100-cased-xglue-ntg
microsoft/xprophetnet-large-wiki100-cased-xglue-qg
```

## Notes

According to the outputs of original fairseq outputs, integration tests for prophetnet include:

1. encoder hidden states, decoder hidden states, model hidden states of pretrained Prophetnet, xProphetnet checkpoints
2. model hidden states of xProphetnet NTG finetuned model
3. Cross-lingual outputs of xProphetNet NTG finetuned model with different beam sizes
4. CNN/DM outputs of ProphetNet CNN/DM finetuned model with different input lengths

The model was implemented so all of its parts can be used separately. This means that `ProphetNetEncoder` and `ProphetNetEncoder` can be used as stand-alone models. `ProphetNetForCausalLM` can be instantiated easily from pretrained checkpoints and can be used within the EncoderDecoderModel framework. 

Comments:
I opened a wrong PR yesterday, please help me check this version, thanks!
@JetRunner @patrickvonplaten 

@qiweizhen - this looks great! Is this the complete PR? Can we close the "old" PR: https://github.com/huggingface/transformers/pull/6187 in favor of this one? 

@qiweizhen the Integration tests look great! @JetRunner, I think we can take it from here :-) 

I saw that there are models, such "xprophetnet-large-wiki100-cased-xglue-ntg" that are both under microsoft and under weizhen - @qiweizhen are these models identical?  

This PR is complete version, as I rebased this branch to the latest huggingface version with directions of @JetRunner .

Models under Microsoft are what we actually used. Those under qiweizhen were used to debug.  I will delete the models under qiweizhen.

Thank you for your helps @patrickvonplaten  @JetRunner 

> This PR is complete version, as I rebased this branch to the latest huggingface version with directions of @JetRunner .
> 
> Models under Microsoft are what we actually used. Those under qiweizhen were used to debug. I will delete the models under qiweizhen.
> 
> Thank you for your helps @patrickvonplaten @JetRunner

Awesome! Thanks a million for your work! We will take it from here :-) 

@patrickvonplaten Hi, may I ask when could ProphetNet be added into Transformers? Are there any jobs I can co-work to help it be integrated?

Hey @qiweizhen ,

Sorry for the delay on this. Prophetnet is my no 1 priority next week. It should be merged by the end of next week. You have done your part - I might ping you for some further questions

@qiweizhen - the integration tests are awesome! Thanks to that it should be quite straightforward to integrate the model

@qiweizhen - would it be ok for you if we add a `ProphetNetModel` and a `XLMProphetNetModel`, each with their  respective tokenizers. I think this would be cleaner and is also more in line with `Roberta` and `XLMRoberta` for example. I should be quite easy to do this. I can take care of it - would just be great to have your approval on it :-) 

> @qiweizhen - would it be ok for you if we add a `ProphetNetModel` and a `XLMProphetNetModel`, each with their respective tokenizers. I think this would be cleaner and is also more in line with `Roberta` and `XLMRoberta` for example. I should be quite easy to do this. I can take care of it - would just be great to have your approval on it :-)

Sure! Thank you!