## 3 Methodology
### Convolutional Self-Attention  
A vanilla convolutional self-attention mechanism is proposed in [16] for learning short-range dependencies. Self-attention [36] is a mechanism that allows a model to focus on different parts of an input sequence when processing each element, enabling it to capture long-range dependencies by assigning attention weights across all elements. However, the self-attention mechanism distributes attention across all elements, which can sometimes lead to the neglect of relationships between adjacent elements and phase-level patterns. To address this, convolutional self-attention (CSA) [16] introduces a convolutional layer within the self-attention framework. CSA applies localized filters that focus on neighboring elements, enabling the model to capture short-range dependencies and fine-grained features between adjacent utterances. This makes CSA particularly effective for extracting local patterns in speech data. As illustrated in Figure 5a, the multi-head convolutional self-attention mechanism operates with each color representing a different attention head. The darker regions indicate the areas that convolutional self-attention focuses on. From the figure, it is evident that CSA restricts its attention to local regions, concentrating on the contextual relationships between adjacent elements. This localized attention allows CSA to capture finer details and better extract local features, making it especially suited for modeling dependencies between neighboring elements.  
Figure 4: Backward fusion structure, where \(\phi\) represents the attention score calculation function as shown in Equation (1), and \(F_{i}\) denotes the feature of the \(i\)-th layer.  
However, CSA with a fixed kernel size has limitations in fusing multi-scale features. Specifically, when CSA projects multi-head spaces of the same dimension, the restricted regions of interest and the high number of heads can cause the regions projected by different heads to converge into similar feature spaces, leading to a lack of diversity in the feature representations learned by the different heads.  
To overcome this limitation, we propose a multi-scale feature extraction approach to enhance the feature space projection. This allows self-attention to compute temporal dependencies across different regions of information while mitigating the overfitting problem associated with the multi-head mechanism in CSA. In this paper, we introduce an improved convolutional self-attention layer that incorporates multi-scale feature extraction. As shown in Figure 5b, we use different kernel sizes across the attention heads, enabling the extraction of fine-grained features at multiple scales. Given an input sequence \(X\), CSA focuses on local regions for each query \(q_{i}\) and restricts its attention region to a local scope of fixed size \(m+1\) (\(m\in 3,7,11,15\)), centered at position \(i\). To maintain the same output dimension as the input, we reduce the output dimension for each kernel width by a factor of four.  
\[\hat{K_{m}}=\{k_{i-\frac{m}{2}},\cdots,k_{i},\cdots,k_{i+\frac{m}{2}}\}. \tag{3}\]  
\[\hat{V_{m}}=\{v_{i-\frac{m}{2}},\cdots,v_{i},\cdots,v_{i+\frac{m}{2}}\}. \tag{4}\]  
Each head's linear mapping query, key, and value with dimensions of \(dk\), \(dk\), and \(dv\). In practice, self-attention computes the attention function on a set of queries simultaneously. The query, key, and value are packed together into a matrix \(Q\), \(\hat{K_{m}}\), and \(\hat{V_{m}}\). The attention output is calculated as  
\[Attention(Q,\hat{K_{m}},\hat{V_{m}})=softmax(\frac{Q\hat{K_{m}}^{T}}{\sqrt{d_{ k}}})\hat{V_{m}}. \tag{5}\]  
\[Head_{i,m}=Attention(QW_{i}^{Q},\hat{K_{m}}W_{i}^{\hat{K_{m}}},\hat{V_{m}}W_{i }^{\hat{V_{m}}}) \tag{6}\]  
\[MultiHead(Q,\hat{K_{m}},\hat{V_{m}})=Concat(head_{1,3},\cdots,head_{l,m}) \tag{7}\]  
where \(W_{i}^{Q}\), \(W_{i}^{\hat{K_{m}}}\), \(W_{i}^{\hat{V_{m}}}\) are the weight matrices in multi-head attention with dimensions \(d_{k}/l\), \(d_{k}/l\), \(d_{v}/l\), respectively.