Before revision:  The DeBERTa code and pre-trained models will be made publicly available at URL
After revision:  Notably, we scale up DeBERTa to 1.5 billion parameters and it substantially outperforms Google's T5 with 11 billionparameters on the SuperGLUE benchmark (Wang et al., 2019a) and, for the first time, surpasses the human performance (89.9 vs. 89.8).