Outputs are not consistent when using DeBERTa for inference
Author: polarispw
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.8.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)


### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am running a simple mask-filling task on DeBERTa and find that the output logits of the same input sentence vary every run (reloading the model from the disk).
I have set `eval()` `manual_seed()`, but they do not work. And the outputs are very different and do not look like they are caused by random seeds.
Even the [official discript](https://huggingface.co/docs/transformers/model_doc/deberta#transformers.DebertaForMaskedLM) shows the same problem. By the way, it works well when holding the model in memory and feed the same input twice to it.
```
from transformers import AutoTokenizer, DebertaForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-base', cache_dir='model_cache')
model = DebertaForMaskedLM.from_pretrained('microsoft/deberta-base', cache_dir='model_cache').eval()

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

# retrieve index of [MASK]
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
tokenizer.decode(predicted_token_id)

labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
# mask labels of non-[MASK] tokens
labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)

outputs = model(**inputs, labels=labels)
print(round(outputs.loss.item(), 2))
# Run twice without reloading models work fine
# with torch.no_grad():
#     logits = model(**inputs).logits
# 
# # retrieve index of [MASK]
# mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]
# 
# predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
# tokenizer.decode(predicted_token_id)
# 
# labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
# # mask labels of non-[MASK] tokens
# labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)
# 
# outputs = model(**inputs, labels=labels)
# print(round(outputs.loss.item(), 2))

```
### Expected behavior

By feeding the same input, I should get the same outputs.

Comments:
Hi, the mismatch seems to be related to the classification layer, namely `cls.predictions.transform.dense.weight`, whose weights are randomly initialized at each loading from the pretrained model. I assume it is expected since it is a pretrained model, hence this layer should depend on fine-tuning.

#22790 It seems that there is a problem with the connection between Microsoft and Huggingface?
Is there any alternative methods like downgrading the version of transformers?

When running your code snippet, the `_load_pretrained_model()` method expects (but doesn't find)
`{'lm_predictions.lm_head.dense.weight', 'lm_predictions.lm_head.bias', 'lm_predictions.lm_head.LayerNorm.bias', 'lm_predictions.lm_head.LayerNorm.weight', 'deberta.embeddings.position_embeddings.weight', 'lm_predictions.lm_head.dense.bias'}` weights in the state dict, whereas `{'deberta.embeddings.position_embeddings.weight', 'lm_predictions.lm_head.dense.weight', 'lm_predictions.lm_head.bias', 'lm_predictions.lm_head.dense.bias', 'lm_predictions.lm_head.LayerNorm.weight', 'lm_predictions.lm_head.LayerNorm.bias'}` are in the state dict but never used. I think this is the cause of the issue, and it might as well for [#22790](https://github.com/huggingface/transformers/issues/22790).

I'd still need to investigate, but at least, it explains why those weights are randomly initialized even if we start from a LM pretrained model.

Yep, the porting of the model probably went wrong at some point and we have to refactor that part of the code. It's long due and I don't have quick fix for you appart from just using other more recent and better models! 😅 

Is this issue open for contribution?

If you have time you can take over #22105, but it should be a 3rd contribution or more, needs to be BC and properly done. I should be able to tackle this this month! 

@ArthurZucker this would be my first contribution to hf. However, I am familiar with the project and down to help out with #22105. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.