Before revision:  We show that these two techniques significantly improve the efficiency of model pre-training and performance of downstream tasks.
After revision:  We show that these two techniques significantly improve the efficiency of model pre-training and the performance of both natural languageunderstand (NLU) and natural langauge generation (NLG) tasks.