DeBERTa for MaskedLM appears to be producing random results
Author: alexdauenhauer
Pull request: no
Labels: name: bug

### System Info

@LysandreJik when I run the sample code in the docs for `DeBERTaV2ForMaskedLM` it appears to be returning random predictions

```python
from transformers import DebertaV2Tokenizer, DebertaV2ForMaskedLM
import torch

tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")
model = DebertaV2ForMaskedLM.from_pretrained("microsoft/deberta-v3-base")

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

# retrieve index of [MASK]
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
tokenizer.decode(predicted_token_id)
```
If I run this code block multiple times in a row, I get a different response every time, none of which are the correct response. If I run this code block with `BERT` or `RoBERTa` I get the correct answer every time. I also tried different models of DeBERTa such as `"microsoft/deberta-v2-xlarge"` and I get the same thing, random responses

transformers version = 4.15
python version = 3.8.12
os = macOS 12.4
running on cpu

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

run the code block in the description over and over again. The result will be different everytime.

```python
from transformers import DebertaV2Tokenizer, DebertaV2ForMaskedLM
import torch

tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")
model = DebertaV2ForMaskedLM.from_pretrained("microsoft/deberta-v3-base")

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

# retrieve index of [MASK]
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
tokenizer.decode(predicted_token_id)
```
example output
```python
'Independence'
```

### Expected behavior

I expect the result to at least be the same every time, but I also expect it to be correct.

Comments:
Hi @alexdauenhauer ,

I think there's also at least one issue in the upstream repo:

https://github.com/microsoft/DeBERTa/issues/74

However - this is not a rant - but the DeBERTa guys are not really responsive and e.g. pretraining code of v3 is still not available after months (but that's another story).

@stefan-it ah... ok thanks for the info, sounds like I'll just have to use a different model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.