Fix bug in DebertaForMaskedLM
Author: YeDeming
Pull request: no

# 🚀 Feature request

## Motivation

Authors of DeBERTa have released their pre-training code. 

https://github.com/microsoft/DeBERTa/blob/771f5822798da4bef5147edfe2a4d0e82dd39bac/DeBERTa/deberta/bert.py#L269-L289

Could you please update the MLM head classifier in DebertaForMaskedLM?

## Your contribution

I refine the code according to the code of DeBERTa. But the MLM loss of the example code only decreases from 11+ to 3.86, which is still large:
```
from transformers import RobertaTokenizer, RobertaForMaskedLM
import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaForMaskedLM.from_pretrained('roberta-base')

inputs = tokenizer("The capital of France is <mask>.", return_tensors="pt")
labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]

outputs = model(**inputs, labels=labels)
loss = outputs.loss   # about 3.86 after I refine the code
```

Here is my modification:

```
class DebertaLMPredictionHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        # self.transform = DebertaPredictionHeadTransform(config)
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        if isinstance(config.hidden_act, str):
            self.transform_act_fn = ACT2FN[config.hidden_act]
        else:
            self.transform_act_fn = config.hidden_act
        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

        self.bias = nn.Parameter(torch.zeros(config.vocab_size))

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.transform_act_fn(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)

        hidden_states = self.decoder(hidden_states)
        return hidden_states
....
class DebertaForMaskedLM(DebertaPreTrainedModel):
    _keys_to_ignore_on_load_unexpected = [r"pooler"]
    _keys_to_ignore_on_load_missing = [r"position_ids", r"predictions.decoder.bias"]

    def __init__(self, config):
        super().__init__(config)

        self.deberta = DebertaModel(config)
        self.lm_predictions = DebertaOnlyMLMHead(config)

        self.init_weights()

    def get_output_embeddings(self):
        return self.lm_predictions.lm_head.decoder

    def set_output_embeddings(self, new_embeddings):
        self.lm_predictions.lm_head.decoder = new_embeddings
```


Best,
Deming

Comments:
Pinging @BigBird01 for advice!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.