# Speech Enhancement Based on Unidirectional Interactive Noise Modeling Assistance
###### Abstract  
It has been demonstrated that interactive speech and noise modeling outperforms traditional speech modeling-only methods for speech enhancement (SE). With a dual-branch topology that simultaneously predicts target speech and noise signals and employs bidirectional information communication between the two branches, the quality of the enhanced speech is significantly improved. However, the dual-branch topology greatly increases the model complexity and deployment cost, thus limiting its practicality. In this paper, we propose UniInterNet, a unidirectional information interaction-based dual-branch network to achieve noise modeling-assisted SE without any increase in complexity. Specifically, the noise branch still receives information from the speech branch to achieve more accurate noise modeling. Subsequently, the noise modeling results are utilized to assist the learning of the speech branch during backpropagation, while the speech branch no longer receives the auxiliary information from the noise branch, so only the speech branch is required during model deployment. Experimental results demonstrate that under the causal inference condition, the performance of UniInterNet only marginally decreases compared to the corresponding bidirectional information interaction scheme, while the model inference complexity is reduced by about 75%. With comparable overall performance, UniInterNet also outperforms previous interactive speech and noise modeling-based benchmarks in terms of causal inference and model complexity. Furthermore, UniInterNet surpasses other existing competitive methods. s +
Footnote †: 2020: _Applied sciences_  
layers to capture both local and global features of the signal, achieving direct mapping from noisy to clean speech. Nevertheless, more methods realize SE in the time-frequency (TF) domain [4; 5; 6; 7; 8; 9; 10]. The TF domain methods employ a DNN model to operate the noisy spectrum, suppress the noise components, and estimate the enhanced spectrum. In the past few years, a plethora of DNN models of different structures have been studied, such as convolutional neural network (CNN) [11], recurrent neural network (RNN) [12], and more recently, Transformer [13; 14]. The CNN excels at extracting high-level features but primarily focuses on local TF patterns. RNN is able to capture long-distance contexts, but it performs poorly in the analysis of local feature information. As for Transformer, although it has demonstrated outstanding performance in SE, it simultaneously results in significant computational costs and memory consumption. This limits its practicality, particularly in real-time processing applications. In recent years, the convolutional recurrent network (CRN) has been introduced to SE, which effectively combines the strengths of both CNN and RNN. By incorporating a symmetric convolutional encoder-decoder (CED) and a recurrent module, a CRN [4] is proposed to receive a noisy magnitude spectrum and predict the enhanced one. To further correct the noisy phase and improve SE performance, the CRN is subsequently extended to its complex-valued version, known as DCCRN [7]. In addition, the gated linear unit (GLU) is employed to replace the convolutional and deconvolutional layers in the CED, resulting in the GCRN [15]. The GCRN utilizes two parallel decoders, respectively, to estimate the real and imaginary parts of the enhanced complex-valued speech spectrum. DPCRN [16], on the other hand, employs intra-chunk long short-term memory (LSTM) and inter-chunk LSTM layers to achieve more comprehensive sequence modeling. More recently, FRCRN [17] proposes a convolutional recurrent encoder-decoder (CRED) structure to boost the feature representation capability of the CED. ICCRN [18] abandons the frequency downsampling operation of the convolutional encoder and implements SE in the cepstral space. SICRN [19] enhances CRN by combining a state space model and the inplace convolution operation.