## 3 Methods
### Low Rank Approximation Decomposition
#### 3.1.1 Low-Rank Encoder  
Unlike the Conformer encoder, our encoder uses a low-rank decomposition multi-head self-attention module to replace the multi-head self-attention module in the original model. Each encoding layer in the encoder consists of a stack of four modules: two feedforward modules, a low-rank decomposition multi-head self-attention module, and a convolution module. The detailed structure is shown in Figure 1.  
When the \(i\)-th encoder layer input \(X_{i}\), the output \(Y_{i}\) can be expressed as follows:  
\[\widetilde{x}_{i}=x_{i}+\frac{1}{2}FFN(x_{i}) \tag{1}\]  
\[x_{i}^{\prime}=\widetilde{x}_{i}+LMHSA\left(\widetilde{x}_{i}\right) \tag{2}\]  
\[x_{i}^{\prime\prime}=x_{i}^{\prime}+Conv(x_{i}^{\prime}) \tag{3}\]  
\[y_{i}=LayerNorm\left(x_{i}^{\prime\prime}+\frac{1}{2}FFN\left(x_{i}^{\prime \prime}\right)\right) \tag{4}\]  
where \(FFN\)(-) denotes the feed-forward module, \(LMHSA\)(-) denotes the low-rank multi-head self-attention module, \(Conv\)(-) denotes the convolution module, and \(LayerNorm\)(-) denotes the layer normalization. Next, we focus on the low-rank multi-head self-attention module.  
The multi-head self-attention mechanism maps \(Q\), \(K\), \(V\), by linear transformations and then stitches the outputs together. The original multi-head self-attention module of Conformer's encoder is denoted as follows.  
\[MultiHead(Q,K,V)=Concat(head_{1},\ head_{2},\dots,\ head_{n}) \tag{5}\]  
\[head_{i}=Attention(Q_{i},K_{i},V_{i}) \tag{6}\]  
\[Q_{i}=QW_{i}^{Q},K_{i}=KW_{i}^{K},V_{i}=VW_{i}^{V} \tag{7}\]  
To reduce the number of parameters in the model, we attempt to construct a low-rank multi-head self-attention module using a low-rank decomposition. We try to decompose the weight matrix of the multi-head self-attention module into the product of two matrices using low-rank approximation decomposition. The weight matrix \(W\) can be decomposed by a low-rank approximation as \(\hat{\Omega}D\hat{V}^{T}\), where \(W\in R^{m\times n}\), \(\hat{\Omega}\in R^{m\times m}\), \(D\in R^{m\times n}\), \(\hat{V}\in R^{n\times n}\). Then, we proceed to decompose the matrix D into two identical  
Figure 1: Low-rank encoder structure diagram.  
matrices \(E\), i.e., \(D=E^{2}\). If the parameters of the matrix are \(m\times m\), the decomposed parameters are \(m\times r+m\times r=2 mr\). If the rank r is less than m, the number of parameters after compression is less than the number of parameters of the original matrix. We use low-rank compression in the training and inference phases. The final decomposition result is shown in the following Equation (8):  
\[W=UV^{T} \tag{8}\]  
where \(U=\hat{U}E\), and \(V=E\hat{V}\). After applying the low-rank decomposition multi-head self-attended module, it can be expressed as:  
\[LMultiHead(Q,K,V)=Concat\big{(}head_{1}^{\prime},head_{2}^{\prime},\ldots, head_{n}^{\prime}\big{)} \tag{9}\]  
\[head_{i}^{\prime}=Attention\Big{(}Q_{i}^{\prime},K_{i}^{\prime},V_{i}^{i}\Big{)} \tag{10}\]  
\[Q_{i}^{\prime}=QW_{i}^{Q^{\prime}},K_{i}^{\prime}=KW_{i}^{K^{\prime}},V_{i}^{ i}=VW_{i}^{V^{\prime}} \tag{11}\]  
where \(W_{i}^{Q^{\prime}}\), \(W_{i}^{K^{\prime}}\), and \(W_{i}^{V^{\prime}}\) are the decomposed matrix of \(W_{i}^{Q}\), \(W_{i}^{K}\) and \(W_{i}^{V}\), respectively.