Before revision:  Transformer and BERT modelsoperate in their first layers preferably in the global averaging regime, while they operate in higher layers in metastable states.
After revision:  The new update rule is equivalent to the attention mechanism used in transformers. This equivalence enables a characterization of the heads of transformer models. These heads perform in the first layers preferably in the global averaging regime, while they operate in higher layers in metastable states.