## 1 Introduction  
Automatic speech recognition (ASR) is capable of automatically converting speech signals into corresponding text sequences. With the continuous iterative development of deep learning techniques, end-to-end speech recognition models are gradually replacing the traditional Hidden Markov Model (HMM)-based speech recognition models. End-to-end speech recognition models are favored by more and more researchers because they simplify the overall model structure and training steps, and reduce the reliance on domain knowledge compared to traditional HMM-based speech recognition models. The most typical representative of end-to-end speech recognition models is Transformer [1], which uses encoders and decoders with a multi-head attention mechanism to simplify the training and recognition process, accelerate parallel computation, and improve the accuracy of model recognition. Recently, [PERSON] et al. [2] proposed the Conformer model, which uses a convolutional module to capture local content dependencies. Its shows better results on English datasets [3] compared to other models such as Transformer and Transformer-XL [4]. Although the Conformer end-to-end models have shown better results in the field of speech recognition and are widely used, they have also caused their own number of parameters to become larger and larger compared to Transformer. Some Conformer-based end-to-end speech recognition models have reached tens of millions of parameters, which also brings about problems such as higher computational complexity of the model, more resource consumption and large storage space. Therefore, it is important to build a lightweight end-to-end speech recognition model. Many works have been carried out to study this. [PERSON] et al. [5] constructed the low-rank Transformer (LRT) using low-rank multi-head attention and low-rank feed-forward. [PERSON] et al. [6] used sparse processing for the computation of self-attention in the Conformer model to reduce the computational complexity on attention. In addition, MHA was used to carry out down sampling to improve the performance of the Conformer model [7]. [PERSON] et al. constructed a low-rank feed-forward network module and a multi-head linear attention module to reduce the model parameters for the Conformer model [8]. Ref. [9] uses two approaches based on singular value decomposition (SVD) to solve the adaptation and personalization problems of DNN. Low-rank compression can compress the parameters of the model, but at the same time, it has an impact on the recognition accuracy of the model; moreover, to our best knowledge, few research works have investigated compression for Conformer's multi-headed attention module. There is a long-tail problem in most data sets. The long-tail problem has been extensively studied in computer vision [10; 11]; for example, [PERSON] et al. [12] used causality to construct a balanced classifier to solve the long-tail problem in vision. Additionally, we found that there is a long-tail problem in the open-source Kazakh language dataset [13], that is, high-frequency words occupy a small part of the dataset, while low-frequency words occupy the majority of the dataset. The uneven distribution of high-frequency words and low-frequency words makes the final recognition accuracy higher for high-frequency words that are adequately trained and lower for low-frequency words that are not adequately trained. At the same time, some low-frequency words may be incorrectly classified and recognized because of speech similarity, resulting in the overall low recognition accuracy of the final model [14; 15]. To the best of our knowledge, few studies have explored the long tail in agglutinative languages.