Deberta's Enhanced Masked Decoder
Author: avacaondata
Pull request: no

# 🚀 Feature request

I don't know if this is a feature request or the feature is already included but I wasn't able to find it. Reviewing Deberta's code, I tried to look for the Enhanced Masked Decoder, which is supposed to take into account absolute positional embeddings. However, when looking both DebertaModel and DebertaForMLM (and its submodules) I don't see where this is done... Could someone please tell me, if the Enhanced Masked Decoder is applied somewhere? Reading the Deberta paper it is clear this is one of the key improvements with respect to BERT and RoBERTa, and I think the Transformers' implementation of Deberta was carried out by the original author of the model, therefore I am sure this part is not missing and it's me that I don't find it... If someone can help me with this I'd really appreciate it :) 

@patrickvonplaten @sgugger @BigBird01 

## Motivation


## Your contribution




Comments:
The implementation of Enhanced Mask Decoder was here, https://github.com/microsoft/DeBERTa/blob/master/DeBERTa/apps/models/masked_language_model.py
We will port the implementation to transformers.

Hi, Alex, thanks for your interest to our work and sorry for the inconvenience.  The missing EMD part will not affect the convergence of pre-training. It will slightly affect the perplexity of MLM training. The inconsistent issue between transformers and the official implementation is a bug due to history reason. We will fix it and improve our future practice to maintain higher quality bars at both side. You are welcome to help us to port the official EMD and the data loading/preparation part from our official repo to transformers. 
If you have other issue with pre-training, please feel free to reach me by mail directly. Thanks! 
  

Thank you so much for the understanding and for the kind reply. Of course I had interest in your work from the moment I first read the paper, it's so original and your results are impressive. I really appreciate you taking the time to fix this, I'll be looking forward to try the changes. From what you say, do you think that we could safely continue training from the last checkpoint with the new model updates without trouble? Or would we need to make an adjustment to make things work? 
Again, thanks for the kind explanation, and for sharing your work with the community. 

Thanks for you understanding and kind words, Alex:) 
I think you can just continue train with your current settings as long as the training loss didn't go wild. 
Basically, besides of DA and EMD described in our paper, there are a few other things important for pre-training.

1. Batch size, learning rate, warmup. For base model, our best local settings are: 2k batch size, 1M steps with learning rate 3e-4 and warmup for 10k steps. You may also need to set beta2 of Adam to 0.98 to stabilize the training with large batch size.
2. Model size
3. Vocabulary
4. Data volume and quality


We just added a [document](https://github.com/microsoft/DeBERTa/tree/master/experiments/language_model) on pre-training in our official repo, you make take it as a reference for your settings. And you are welcome to leave comments on it.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.