# A Dual-Branch Speech Enhancement Model with Harmonic Repair
## 2 Proposed Methods
### The Harmonic Repair Large Frame Enhancement Model HRLF-Net  
Under normal circumstances, the noise in noisy speech is non-additive, and the noisy speech \(y\in\mathbb{R}^{p\times l}\) can be represented as \(y=s+n\), where \(s\) is the pure speech signal desired in the task, \(n\) is the noise, and \(p\) is the total number of samples in the signal. The task of speech enhancement is to separate \(s\) from \(y\) as much as possible while suppressing the noise as much as possible.  
The input to the proposed model is the complex spectrum after STFT, represented as \(Y=S+N\), where Y, S and N respectively represent the complex spectra of the noisy speech, pure speech, and noise signal, with \(\{Y,S,N\}\in\mathbb{R}^{2T\times F}\). T and F represent the number of bins in the time and frequency dimensions in the real and imaginary parts, respectively. Figure 5 shows the overall architecture of our proposed HRLF-Net.  
Figure 4: LSTM with skip connections.  
Figure 5: The overall structure of HRLF-Net.  
Several harmonic repair modules are used to extract and refine features, and harmonic fading-out modules, represented as HFO in Figure 5, are applied after each processing to prune the processed features to a certain extent. Harmonic fading-out modules are used concurrently with each harmonic repair module for optimal filtering.  
Real and Imag branches correspond to the real and imaginary parts of the CIRM mask structure and are processed using two branch networks. In the real branch, convolutional layers with causal 2D convolution, batch normalization [27], and PReLU [28] preliminarily filter the input signal and separate different channels, which is an autoencoder design [29], followed by cascaded FFC modules forming residual connections. The resulting output is up sampled to obtain the predicted CIRM spectrum's real part output \(M_{r}\). To avoid issues of non-structural and wrapping phase jumps [30], the Imag branch employs a phase decoder with dilated DenseNet [31]. After the deconvolution block, parallel dual 2D convolution layers output pseudo-components, and a dual-parameter arctangent function activates these two components to obtain the predicted CIRM imaginary spectrum output \(M_{i}\), where instance normalization layers are connected to standardize the network's intermediate features. This structure is based on the definition of the complex ideal ratio mask.  
\[M_{r}=\frac{Y_{r}S_{r}+Y_{i}S_{i}}{Y_{r}^{2}+Y_{i}^{2}} \tag{4}\]  
and  
\[M_{i}=\frac{Y_{r}S_{i}+Y_{i}S_{i}}{Y_{r}^{2}+Y_{i}^{2}}, \tag{5}\]  
where \(Y_{r}\) and \(Y_{i}\) respectively represent the real and imaginary parts of the noisy complex spectrum. \(S_{r}\) and \(S_{i}\) represent the real and imaginary parts of the pure speech complex spectrum. The masks calculated using \(M_{r}\) and \(M_{i}\), estimated from the noisy speech, can be represented using the following formula:  
\[M=\frac{Y_{r}S_{r}+Y_{i}S_{i}}{Y_{r}^{2}+Y_{i}^{2}}+i\frac{Y_{r}S_{i}+Y_{i}S_{ i}}{Y_{r}^{2}+Y_{i}^{2}}, \tag{6}\]  
Multiplying the noisy speech spectrum \(Y=Y_{r}+Y_{i}\) with the mask \(M=M_{\gamma}+jM_{i}\) results in the enhanced speech spectrum, which is then transformed back into the final waveform (W) using the Inverse Short-Time Fourier Transform (ISTFT):  
\[\widetilde{w}=Y_{\gamma}M_{r}-Y_{i}M_{i}+\mathrm{i}(Y_{r}M_{i}+X_{i}M_{r}). \tag{7}\]