Before revision:  These are high-magnitude normalization parameters that emerge early in pre-training and show up consistently in the same dimensional position throughout the model.
After revision:  In case of BERT and other pre-trained encoder Transformers, the affected component is the scaling factors and biases in the LayerNorm. The outliers are high-magnitude normalization parameters that emerge early in pre-training and show up consistently in the same dimensional position throughout the model.