Before revision:  Our results suggest that layer normalization plays a much more important role than usually assumed .
After revision:  This effect is observed across several BERT-family models and other popular pre-trained Transformer architectures, including BART, XLNet and ELECTRA; we also show a similar effect in GPT-2 .