Before revision:  We find that the proposed approach, which only uses hand-crafted and learnable attentions, is comparable with the full self-attention  .
After revision:  With the assistance of t-SNE, PCA and some observation, the attention weights in self-supervised audio transformers can be categorized into four general cases. Based on these cases and some analyses, we are able to use a specific set of attention weights to initialize the model. Our approach shows comparable performance to the typical self-attention .