Before revision:  However, we observed in trained models, the diagonal attention matrices indicating the redundancy of the corresponding attention heads.
After revision:  However, we observed in trained models, we have previously observed that many attention matrices are close to diagonal, indicating the redundancy of the corresponding attention heads.