Before revision:  This method gives consistent performance gains  on the Wall Street Journal, AISHELL, Switchboard and AMI ASR tasks.
After revision:  The proposed method also forces each head independently learn the most useful patterns. We apply the proposed method to train Transformer based and Convolution-augmented Transformer (Conformer) based ASR models. Our method gives consistent performance gains on the Wall Street Journal, AISHELL, Switchboard and AMI ASR tasks.