XLMProphetNet returning different results when using padding
Author: alexayalamcs
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (False)

### Who can help?

@ArthurZucker @patrickvonplaten 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```

from transformers import XLMProphetNetTokenizer, XLMProphetNetForConditionalGeneration
import torch

tokenizer = XLMProphetNetTokenizer.from_pretrained("microsoft/xprophetnet-large-wiki100-cased-xglue-ntg")
model = XLMProphetNetForConditionalGeneration.from_pretrained("microsoft/xprophetnet-large-wiki100-cased-xglue-ntg").eval()

enc_input = tokenizer("test", return_tensors="pt")
input_ids = enc_input.input_ids
attention_mask = enc_input.attention_mask

dec_input_ids = torch.tensor([[model.config.decoder_start_token_id]], dtype=torch.int64)
dec_attention_mask = torch.tensor([[1]], dtype=torch.int64)

dec_input_ids_pad = torch.tensor([[model.config.decoder_start_token_id, model.config.pad_token_id]], dtype=torch.int64)
dec_attention_mask_pad = torch.tensor([[1, 0]], dtype=torch.int64)

out1 = model(
    input_ids=input_ids, attention_mask=attention_mask,
    decoder_input_ids=dec_input_ids, decoder_attention_mask=dec_attention_mask
)

out2 = model(
    input_ids=input_ids, attention_mask=attention_mask,
    decoder_input_ids=dec_input_ids_pad, decoder_attention_mask=dec_attention_mask_pad
)

torch.isclose(out1.logits, out2.logits[:, 0], atol=1e-1).all() # false
```

### Expected behavior

XLMProphetNet is not returning the same output when the decoder input ids are padded. While the logits are quite similar (high cosine similarity), they are not the same which results in different losses and, in some cases, different predictions.

The expected behavior is that the padded and unpadded version produce the same output.

Comments:
Hey! Thanks for reporting! I think this is probably inherent to the model itself, but will see if theres a bug! Our integration tests don't cover this case, and seems like we don't have our common test for this model! 

No updates yet! @Rocketknight1 if you have time to look at this

Sorry for the delay - I'm going to try to take this one this week!

I spent a while at this - I couldn't track down the exact cause, but I noted the following:
- This issue affects `ProphetNet` as well as `XLMProphetNet`
- The discrepancy occurs in the `decoder` module. There is a very small change in `decoder_hidden_states`, but this is probably a numerical issue, and disappears depending on the precision used. It is not the cause of the problem.
- The real issue is in `decoder_ngram_hidden_states` which is initially identical but diverges wildly after the first layer.
- `decoder_ngram_hidden_states` is unique to `ProphetNet` and gets mixed in unusual ways at each step.

The problem is that ProphetNet's n-gram decoder is so odd I'm not even sure there's an error here! I feel like an actual bug that caused cross-timestep mixing would have been extremely apparent when the model was first added to the Hub, and all equivalence tests would have failed. It's possible that because of the way it operates on n-grams, `ProphetNet` can't really guarantee that padding will have no effect?

Maybe we open this issue to the community and see if anyone has time to read the original papers and step through the code to figure out if the behaviour here is intentional?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.