## 2 Background and Proposed Method
### Proposed Method Overview
#### 2.4.4 Multi-Head Attentive Statistics Pooling  
Analysis of MFCC features extracted from the original speech reveals that noise is predominantly distributed in the low-frequency and mid-frequency bands, while speech is predominantly found in the low-frequency band. Upon substituting the MFCC extractor with a pre-trained model, the noise in the newly extracted features is also characterized by a full-channel distribution. Attentive statistics pooling in ECAPA-TDNN is designed to capture global channel information, which, however, may lead to heightened noise interference in the pooling outcomes. To address this issue, this work introduces a modification involving the replacement of the attentive statistics pooling with a multi-head attention-based pooling mechanism [55]. This approach divides the entire channel space into several autonomous sub-channel spaces, aiming to reduce mutual interference.  
Specifically, the mean and standard deviation of the output features from the final TDNN layer are computed initially. As shown in Figure 4d \"Catenate\", the resulting feature, after aggregating the above three, is defined as \(o(d,T)\). Here, \(d\) can be understood as the frequency band and \(T\) as the time. \(o\) is separated into \(k\) non-overlapping subspaces \((x(d/k,T))\) along the channel dimension \(d\), as shown in Figure 4(d1), \"Chunk\". Among these subspaces, \(x_{t}\) for a \(T\) channel dimension in \(x\) is defined as \([x_{1}^{i},x_{1}^{i},\ldots,x_{t}^{i}]\) (\(1\leq t\leq T,1\leq i\leq k,x_{t}^{i}\in R^{d/k}\)). Similarly, \(x^{i}\) for a \(d\) channel dimension in \(x\) is defined as \([x_{1}^{i},x_{2}^{i},\ldots,x_{T}^{i}]\). Next, unlike the channel-independent self-attention scoring mechanism in [20], the channel-isolated self-attention scoring mechanism is used here. The corresponding scoring factor \(e\) for each sub-channel space (in \(d\) channel) is calculated by using Equation (13).  
\[e=W_{6}\cdot\tanh(W_{5}\cdot x+b_{5})+b_{6} \tag{13}\]  
\(W_{5}\in R^{F(d/k)}\) and \(W_{6}\in R^{F\times F}\) are linear unit operations, \(b_{5}\) and \(b_{6}\) are their corresponding bias variables, as shown in Figure 4(d1) \"Fully Connected\". Then, the _Softmax_ function (Equation (14)) is utilized to compute the importance percentage scores across various \(T\) channels. For each dimension of the \(d\) channel, the score is denoted as \(\beta^{i}\), which represents the importance distribution of data frames across different \(T\) channels within each \(d\) channel dimension.  
\[\beta^{i}=\frac{\exp\left(e_{j}^{i}\right)}{\sum_{j=1}^{T}\exp\left(e_{j}^{i} \right)} \tag{14}\]  
\(\beta\) is concatenated along the channel dimension d from the \(\beta^{i}\) components. Subsequently, \(\gamma\) is constructed by concatenating the \(\beta\) across the \(k\) subspaces, as depicted in Figure 4(d1) \"Catenate\". \(\gamma_{t}\), representing the \(T\) channel dimension within \(\gamma\), is defined as the concatenation of \([\gamma_{1}^{1},\gamma_{1}^{2},\ldots,\gamma_{t}^{k}]\), where each \(\gamma_{t}^{i}\) is a component of the respective subspace. Due to the specificity of different \(d\) channel dimensions, the mean \(\widetilde{\mu}\) and variance \(\widetilde{\sigma}\), obtained through multi-head attentive statistics pooling, are formulated in Equations (15) and (16), also as shown in Figure 4d, \"Hadamard Product\", \"mean\", and \"std\". The pooling output is derived from the concatenation of \(\widetilde{\mu}\) and \(\widetilde{\sigma}\).  
\[\tilde{\mu}=\sum_{t}^{T}\gamma_{t}\cdot x_{t} \tag{15}\]  
\[\widetilde{\sigma}=\sqrt{\sum_{t}^{T}\gamma_{t}\cdot x_{t}{}^{2}-\beta} \tag{16}\]