Before revision:  We demonstrate that the lighter state-of-the-art models with nearly 80\% of self-attention layers pruned, may achieve even better resultson multiple tasks, including natural language understanding, document classification, named entity recognition and POS tagging, with nearly twice faster inference .
After revision:  We demonstrate that state-of-the-art models with nearly 80\% of self-attention layers pruned, may achieve even better resultson multiple tasks, including natural language understanding, document classification, named entity recognition and POS tagging, with nearly twice faster inference .