Before revision:  In this paper wepropose a new model architecture DeBERTa (Decoding-enhanced BERT with dis-entangled attention) that improves the BERT and RoBERTa models using two novel techniques.
After revision:  In this paper wepropose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques.