a phase transition right here this is the blue line by the way the blue training line a face transition where all of a sudden these just these attention heads they somehow decide okay this is the thing i want to specialize in this is the type of task i want like a sub task of linguistic subtask i want to specialize in and then they concentrate on one particular pattern per input so they are really specializing whereas in the last layer you see here that even during training they are sort of continuously learning