Before revision:  We demonstrate that the lighter state-of-the-art models with nearly 80\% of self-attention layers pruned, may achieve even better resultson multiple tasks, including natural language understanding, document classification, named entity recognition and POS tagging, with nearly twice faster inference .
After revision:  We demonstrate that the lighter state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks .