## 2 Background and Proposed Method
### Speech Model of SSL  
HuBERT is derived from the integration of BERT [29] and wav2 vec [35]. Its architecture is characterized by a multilayer CNN encoder, Transformer [49] encoders, and a projection layer. The parameter details for the base HuBERT are presented in Table 1. \begin{table}
\begin{tabular}{c c c} \hline \hline
**CNN Encoder** & & \\
**(Seven Convolutional Layers)** & **Transformer Encoders** & **Projection Layer** \\ \hline strides [5,2,2,2,2,2,2,2] & layers [12] & dimension [256] \\ kernels [10,3,3,3,3,3,2,2] & attention heads [12] & dimension [256] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters of base HuBERT. Figure 1: Various spectrograms for actual sound field situations: (**a**) clean spectrum; (**b**) with weak noise; (**c**) with strong noise; (**d**) with reverberation; (**e**) with weak noise and reverberation; (**f**) with strong noise and reverberation; (**g**) noise spectrum at low frequency; (**h**) speaker-like noise spectrum; (**i**) noise spectrum across frequencies. The network architecture is shown in Figure 2. An input speech of length \(L\) is processed by the CNN encoder, which consists of seven convolutional layers, resulting in \(T\) (\(T=L/320\)) feature vectors with 512 channels. The \(T\) feature vectors are categorized into two types: masked feature vectors \(\tilde{x}\), where the masking process involves randomly selecting the starting point for the mask and setting 10 consecutive features to a value of 0, and unmasked feature vectors, \(x\). The masked and unmasked features are processed by the Transformer encoders, resulting in the output feature sequences [\(\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{ \mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{\mathbf{ \mathbf{  }}}}}}}}}}}}}\) \(\mathbf{\mathbf{\mathbf{\mathbf  
The projection matrix \(P\) is back-propagable; \(e_{c}\) denotes the codebook corresponding to label \(z\), and \(e_{c^{\prime}}\) represents the standard codebook. The variable \(c\) indicates a specific classification. The scale parameter \(\tau\) is utilized for adjustment. The function _sim(-,-)_ calculates the cosine similarity between two tensors. \(p_{m}\) represents the probability density function derived from the masked \(\vec{x}\), while \(p_{um}\) denotes the probability density function from the unmasked \(x\). Cross-entropy loss function is employed. The loss functions \(L_{m}\) and \(L_{um}\), derived from \(x\) with the corresponding pseudo-label, are defined by Equation (3) and Equation (4), respectively. The final loss function, as shown in Equation (5), comprises \(L_{m}\) and \(L_{um}\), where \(\lambda\) represents an adjustment parameter. \[L_{m}=-\sum_{t=1}^{T_{1}}\log\ p_{m}(z,c|\vec{x},t) \tag{3}\]  
\[L_{um}=-\sum_{t=1}^{T_{2}}\log\ p_{um}(z,c|x,t) \tag{4}\]  
\[L=\lambda\cdot L_{m}+(1-\lambda)\cdot L_{um} \tag{5}\]  
Notably, HuBERT employs an ensemble of \(K\) clustering methods, leading to a multinomial adjustment of the loss function, as demonstrated in Equations (6) and (7). \[L_{m}=-\sum_{k=1}^{K}\sum_{t=1}^{T_{1}}\log\ p_{m}^{k}(z,c|\vec{x},t) \tag{6}\]  
\[L_{um}=-\sum_{k=1}^{K}\sum_{t=1}^{T_{2}}\log\ p_{um}^{k}(z,c|x,t) \tag{7}\]  
During the subsequent fine-tuning phase of the ASR task, the projection matrix \(P\) is not updated, and the parameters of the CNN encoder are frozen. Only the Transformer encoders are subjected to parameterization, utilizing connectionist temporal classification (CTC) as the loss function. In the HuBERT strategy, MFCC is initially employed for clustering to generate pseudo-labels, which serve as the target for the initial training epoch. Subsequently, the output from the 6 th layer of the Transformer encoders in HuBERT is utilized for clustering purposes, yielding pseudo-labels that facilitate the second phase of training. In contrast to HuBERT, WavLM enhances the training process by initially introducing additive noise and additional speaker speech to pure speech.