FIRING-Net: A filtered feature recycling network for speech enhancement
Authors: Xinmeng Xu
Yiqun Zhang
Jizhen Li
Yuhong Yang
Yong Luo
Weiping Tu
Venue: ICLR 2025 Poster (2025)
Keywords: speech enhancement
target and non-target features
self-attention
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Current deep neural networks for speech enhancement (SE) aim to minimize the distance between the output signal and the clean target by filtering out noise features from input features. However, when noise and speech components are highly similar, SE models struggle to learn effective discrimination patterns. To address this challenge, we propose a Filter-Recycle-Interguide framework termed Filter-Recycle-INterGuide NETwork (FIRING-Net) for SE, which filters the input features to extract target features and recycles the filtered-out features as non-target features. These two feature sets then guide each other to refine the features, leading to the aggregation of speech information within the target features and noise information within the non-target features. The proposed FIRING-Net mainly consists of a Local Module (LM) and a Global Module (GM). The LM uses outputs of the speech extraction network as target features and the residual between input and output as non-target features. The GM leverages the energy distribution of self-attention map to extract target and non-target features guided by highest and lowest energy regions. Both LM and GM include interaction modules to leverage the two feature sets in an inter-guided manner for collecting speech from non-target features and filtering out noise from target features. Experiments confirm the effectiveness of the Filter-Recycle-Interguide framework, with FIRING-Net achieving a strong balance between SE performance and computational efficiency, surpassing comparable models across various SNR levels and noise environments.

Published as a conference paper at ICLR 2025
FIRING-NET: A FILTERED FEATURE RECYCLING NET- WORK FOR SPEECH ENHANCEMENT
Xinmeng Xu', Jizhen Li', Yiqun Zhang', Yong Luo', Yuhong Yang" 22 & Weiping Tu
NERCMS,, School of Computer Science, Wuhan University, China Hubei Key Laboratory of Multimedia and Network Communication Engineering. Wuhan University, China (xuxinmeng, yangyuhong, tuweiping] @whu. edu. cn
ABSTRACT
Current deep neural networks for speech enhancement (SE) aim to minimize the distance between the output signal and the clean target by filtering out noise fea- tures from input features. However, when noise and speech components are highly similar, SE models struggle to learn effective discrimination patterns. To address this challenge, we propose a Filter- Reee--ecccc--egguuccie framework termed Fllter- Cecycle-NNeeruidee NETwork (FIRING-Net) for SE, which filters the input fea- tures to extract target features and recycles the filtered-out features as non-target features. These two feature sets then guide each other refine the features, lead- ing to the aggregation of speech information within the target features and noise information within the non-target features. The proposed FIRING-Net mainly consists of Local Module (LM) and Global Module (GM). The LM uses out- puts of the speech extraction network as target features and the residual between input and output non-target features. The GM leverages the energy distribution of the self-attention map to extract target and non-target features guided by the highest and lowest energy regions. Both LM and GM include interaction modules to leverage the two feature sets in an inter-guided manner for collecting speech from non-target features and filtering out noise from target features. Experiments framework. Addition-
confirm the effectiveness of the ally, FIRING-Net achieves good balance between SE performance and compu- tational efficiency, outperforming other comparable models across various signal- to-noise ratio levels and noise environments.
INTRODUCTION
Speech enhancement (SE) aims to separate speech from background interference signals (Xu et al.) 2013]. It is a fundamental speech processing problem and has been frequently used as the pre- processor of several acoustic tasks, such speech recognition (Peng et al. 2022), speaker verifica- tion (Rao et al., 2019).. speaker diarization (Sell et al.) 2018), etc. Traditional SE approaches, such as Wiener filtering (Chen et al. (2066), spectral subtraction (Vaseghil 2008). and principle compo- nent analysis (Srinivasarao & Ghanekar 2020), assume that the noises belong to stationary signals, which are significantly different from the speech signal. However, this assumption usually cannot be satisfied in practice and thus these approaches often fail in real-world applications.
Recently, deep networks have shown their promising performance SE, even under highly non- stationary noise environments. Deep learning- SE methods train SE networks on extensive noisy-clean pairs to learn distinguish between speech and noise, aiming effectively remove noise components (Sell et al. 2018 Xu et al.) 2013). For example, we can utilize the convolutional operations to extract local speech features PPar & Lee| 20177] Pandey & Wang 2019). the recurrent neural networks or transformers to capture global speech features (Sun et 2011 Kim Kim et al., 2020). and integrating both techniques can enhance internal feature differences within speech (Abdulatif et al. 2024| Lu et al. 2023 Xu al. (023).. Despite significant improvements, issues of speech distortion and residual noise persist (Jo & Yoo. 2010 Xia et al., 2020 Wakabayashi et al. 2018).
"Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Speech feature component
Target information flow
Network for speech extraction
x Noise feature component
Interference information flow
Network for noise extraction
Stage Feature Encoding
Stage 2: Interference Feature Filtering
Stage 3: Target Feature Extraction
(a) Conventional feature processing framework
Stage Feature Recycling
Stage 1: Feature Encoding
Stage 2: Interference Peature Filtering
Target Feature Extraction
Strrpe
Torgel P7AN
(b) Proposed Filter- Recydee-eteereeeeeeeeeee framework
Figure 1: Comparison of (a) the conventional SE framework and (b) the proposed Filter- Recycle- Interguide framework. (a) The conventional framework filters input features to remove noise but risks overestimating or underestimating noise. (b) The proposed framework recycles filtered features and processes them interactively with output features. This approach leverages the speech-dominant output and noise-dominant filtered features for mutual guidance, enhancing feature separation.
That is, they remain ineffective against noise containing highly similar acoustic events, such as speech from non-target speakers or echoes reflected from walls (Hu et al.) 2023 Zheng et al. (02TT. The primary reason is the often indistinct differences between speech and noise components. As shown in Figure []), SE networks struggle to learn discriminative patterns between highly similar speech and noise components, leading to the overestimation or underestimation of noise information.
Some recent solutions use noise as part of the supervision to enable SE networks to establish a dual mapping from mixed signals to both speech and noise signals. A representative work is the dual-branch network structure, where the two branches aim to model the speech and noise signal, respectively, and an interacting operation is designed explore the correlation between speech and noise features in noisy mixture (Zheng et al. [202 Zhao et al., 2022 Li et al. 2024). This method interactively leverages the features of speech and noise signals and using both as supervision. The goal is to ensure the relative independence of speech and noise signals in the feature space, and thereby enabling the network to extract discriminative patterns for each signal. However, due to the dynamic and highly random nature of noise, completely predicting noise signal is infeasi- ble (Ortega-Gan García González- Rodriguzzz 1996). Consequently, its performance still heavily relies on the accuracy of the speech signal prediction branch, thereby diminishing the primary purpose of predicting noise signals.
To remedy this drawback, design Metter-Recycll-nntrrge (FRT) strategy extract dis- criminative patterns between speech and noise within the mixture input. This achieved by: (1) avoiding overemphasis on exploring feature differences within speech and (2) achieving more ro- bust noise feature extraction for interaction with speech features without directly predicting the noise signal. As shown in Figure bb), method consists of three main steps: (1) Processing input fea- tures through network modules to extract target features, similar to conventional SE models; (2) Re-collecting and redefining filtered-out features as non- features; (3) Guiding and interacting target and non-target features, where target features, mainly composed of speech components, inte- grate speech from non-target features and remove noise, while non-target features, predominantly containing noise, assist in refining the target features.
By incorporating the designed FRI strategy into deep neural networks, we obtain the proposed Fllter- Recycce--cceeerruuiide NETwork (FIRING-Net). IRING-Net integrates local and global per- spectives through the Local Module (LM) and Global Module (GM) to perform Filtering and Recy- cling operations noisy speech features. The LM refines local features using residual calculations between input and output features, inspired by projecrroottion (Afouras et al.] 2018). to extract non-target features. The GM addresses global dependencies by leveraging energy distributions in
2

---

Published as a conference paper at ICLR 2025
self-attention maps and applying Top-K mechanism (Chen al. 2023b) for non-target feature refinement. To address the overlap and similarity between speech and noise, both modules adopt two-stage coarse-to-fine framework, enabling precise separation of target and non-target features, ensuring speech-dominant and noise-dominant characteristics are accurately distinguished. Addi- tionally, FIRING-Net considers the distinct roles of encoder and decoder features: the encoder pri- marily contains noise, while the decoder focuses on speech reconstruction. During the interguide stage, tailored interaction modules are employed for each, facilitating effective noise removal in the encoder and enhancing the completeness of reconstructed speech in the decoder. This compre- hensive design ensures robust speech separation and reconstruction, even under challenging noise conditions.
Our model trained on DNS Challenge 2021 (Reddy et al. 2021 and is evaluated on two public datasets (WSJO-SI 84 + NOISX-92 (Paul & Baker 1992. Varga & Steeneken 1993) and AVSpeech AudioSet (Ephrat et al., 2018, Gemmeke et al. 2017)). The extensive results demonstrate the efficacy of the proposed method. Specifically, under the babble noise environment, it achieves 3.03% relative improvement in terms of PESQ and a 6% dB gain in terms of SI-SNRi compared to the most competitive MP-SENet (Lu et al. 2023).
To summarize, the main contributions of this paper are:
SE strategy, which enables mutual refinement be- tween retained and filtered-out features, i.e., effectively eliminating noise from the retained features while recovering speech information from the filtered-out features.
We design Dilte-Recyycl-nnee
We propose FIRING-Net for SE, comprising two key components: LM and GM. Both modules employ two-stage coarse-to-fine framework, effectively separating target and non-target features, ensuring that speech-dominant and noise-dominant characteristics are accurately captured.
We propose two distinct interaction modules employed for the encoder and decoder, abling comprehensive noise removal in the encoder and enhancing the completeness of reconstructed speech in the decoder.
RELATED WORKS
The integration of deep learning has significantly enhanced SE performance. Recently, convolu- tional neural networks (CNNs) have been employed in SE tasks with notable success, as they ef- fectively capture implicit information within the speech signal and manage small shifts in the time- frequency (TF) domain of speech features, thus adapting to varying speaker identities and acoustic environments (Park & Lee|
ing broader dependencies | 2017 Fu et al. (017). However, CNNs are somewhat limited in model- lowelevel features (O O'shea& Nash 2015). To address this issue, many approaches have turned to transformers (Vaswani et al. 2017 to replace CNNs, enabling the capture of global information in waveforms or spectrograms (Kim et al.) 2020 Dang et al.) 2022a). Further- more, several studies have sought to integrate CNNs and transformers, thereby harnessing both local and global information ( (Abdulatif et al. 2024 Lu et al., (003). However, deep learning-basee SE methods primarily focus on extracting speech features based on speech characteristics alone, leading to speech distortion or residual noise in the enhanced output when speech and noise components in the deep latent space are highly similar.
Rather than solely focusing on estimating target speech, some SE methods improve the performance by building noise models account for noise prior. For instance, certain approaches (Odelowo & Anderson 2017: Liu et al.) (021)) implement spectral subtraction using deep neural networks
(DNNs) through two-stage process: noise signal estimation followed by speech signal recovery. Although outperform traditional signal processing techniques in modeling structured noise (Ortega-
García & González- RRdriiuuz 1996). DNNs' generalization capability remains limited. To enhance
noise modeling accuracy, advanced methods (Zheng et al.) 2021 Zhao et al.) 2022 Li et al. 2024) introduce a two-branch framework to predict both speech and noise simultaneously. incorporating
interaction modules various layers based on the correlations between predicted speech and the residual signal. However, this approach still struggles with the unpredictability of noise, as its
success largely depends the accuracy of the speech prediction branch.
3

---

Published as a conference paper at ICLR 2025
Partnute Opersdion
Roshape Operation
Elemest-wise Prosluction
Demeettwwee Substraction
Recoder
www.form
Target Feature Flow
Festure Flow
Gacal Eander (33-))
Madele Beenker (La6)
13cc
Figure 2: Overview of our FIRING-Net, which is an encoder-decoder architecture that leverages the Reecycle-Interguide strategy for SE by dividing input features into target (speech) and non- target (noise) components. FIRING-Net integrates the LM and GM to implement the FRI strategy through distinct target and non-target feature separation methods. The LM extracts non-target fea- tures by calculating the error between input and output, i.e., target features, while the GM leverages self-attention energy distributions to separate target and non-target features. the encoder, where non-target features are noise- diisn-ddmmnnnt, the IM-E facilitates reciprocal extraction of speech and noise from target and non-targ con-target features. the decoder, where non-target features are speech ppeech-dominant, the IM-D ensures comprehensive integration of target and non-target features.
METHOD
Figure 2 is overall illustration of our method. The noisy signal y E RIxL undergoes short-time Fourier transform (STFT) to produce a complex spectrogram Y E 2x We extract the mag- nitude spectrogram Y M glxxxF and the wrapped phase spectrogram Yp R1xT where T and F denote time and frequency. Before processing with FIRING-Net a power-law compres- sion (Wisdom et al. 2019) applied Yм using a compression factor of 0.3 (Braun Tashev] 2021). to enhance alignment with human auditory perception (Lee et al. 2018 Wilson et al. (018)) The resulting compressed magnitude spectrogram Yi, is then concatenated with Yp and used as the input for FIRING-Net.
Encoder. Given the input feature Y,,, E 2 the encoder includes two convolution blocks, to- gether with three densely connected Local Modules for encoder (LM-Es) positioned between the two convolutional blocks. The first convolution block maps Y.,. to an intermediate space with C chan- nels. Three LM-Es aim to extract more completed speech information and accurately suppress noise by using the Filter- Mitter-Reyyll--toe Recycle-Interguide framework (see See (.11) while dense connections combine features from all layers to capture different levels of detail. The final convolution block decreases the frequency dimension to F' to reduce complexity. Subsequently, the encoder output is then pro- cessed by N Global Modules (GMs), with N inspired by TF-Conformer (Abdulatif et al. 2024 Lu et al.) (002)), alternately captures time and frequency dependencies within the Filter-Recylle-niiee Interguide framework.
4, described in See 3.2 The architecture,
Magnitude and Phase Decoders. The decoder processes the output from the N GMs in a decou- pled manner, involving two branches: the magnitude decoder and the phase decoder. The former is designed to predict a mask, which is then element-wiss multiplied by the input magnitude spec- trogram to enhance the signal. In contrast, the phase decoder predicts residual that refines the noisy phase to improve phase accuracy (Afouras et al.) 2018). Both decoders contain three densely connected LMs for decoder (LM-Ds). both branches, a deconvolutional block is employed to upsample the frequency dimension back to F, while reducing the channel number to 1. For the magnitude decoder, a PReLU activation function is used to predict the final mask, allowing it to leam different slopes for each frequency band (Abdulatif et al.) 2024). In the phase decoder, the predicted phase residual is added to the noisy phase and then normalized to produce a clean phase prediction (Afouras al. 2018).
4

---

Published as a conference paper at ICLR 2025
Neney Imput
(b) Course Target Foature IIM
(r) Gramen Nem- Target Preturm
(d) Refined Target Peature (IM
(c) Refined Non: Tua Fouruere
(196-1))
(IM Fl
10 Clean Torges
b Soarsw Targst Nettrr (M4-0)
D0. Courn Kon- Targe Peaturu (14-0)
@ Redned Tugget Feaure [I
Reftaed Non: Featur (IN D)
Figure 3: Visualizations of coarse and refined target and non-target features, i.c., f,, f,s f,, and f, extracted by LM-E and LM-D, along with their corresponding spectrograms of noisy mixtures and clean target speech.
3.1 LOCAL MODULE
We replace the conventional convolutional modules in the Encoder and Magnitude and Phase De- coders with Local Module for Encoder (LM-F) and Local Module for Decoder (LM-D). As shown in Figure LM-E and LM-D have identical structures, with the only difference being that LM-E tains an Interaction Module specifically designed for the Encoder part (IM-E), while LM-D includes an Interaction Module tailored for the Decoder part (IM-D). The LM coarsely produce target fea- ture f, processed from the input feature fin. by applying a convolutional layer with normalization and filter the non-target feature fn.
f; Heone (,
f, = fm f..
(1)
where Hoonv (+) denotes the convolutional operation.
A simple convolutional layer, due to its limited capacity, only provides a coarse separation of target and non-target features within the f.n- Consequently, further refine both the target and non-target features separately. For the non-target features, we use dilated DenseNet (Pandey & Wang 2020) further extract the ignored target features fn.: and calculate the residual between f,,: and f, to obtain more purified non-target feature fn..- Similarly, we apply the same process to the target features, resulting in more purified target feature fr. as well as contributing to the refinement of non-target feature fr
Lr.t Hiense (fn); f.. = Heense (ft).
fn.n = f, En,t frn = f. fr,t>
where (-ensel-) denotes the dilated DenseNet that contains three convolution blocks with dense connections, the dilation factors of each block are set to 2, 4]. We aggregate all target and non- target features through addition operation obtain the refined target features f. = =f.+ fn,t and non-target feature f. = fr.n + fn,n, respectively.
The purpose of the Interaction Module is to enable mutual guidance between target and non-target features, allowing for more refined extraction of speech information from non-target features to supplement the target features, while filtering out noise information from the target features. How- ever, since the information contained in non-target features differs between the Encoder and De- coder, we designed IM-E and IM-D to specifically handle the target and non-target features in LM-E and LM-D, respectively.
LM-E: In the Encoder, input features typically contain more noise components. As shown in Fig- ure c), the non-target features extracted by LM-E are predominantly noise. The role of IM-E is extract speech components from these non-target features and filter out noise from the target fea- tures. As depicted in Figure 4ta). IM-E concatenates f, and f, and feeds them into convolutional module to generate a mask M,,- This mask identifies the preserved areas of non-target features to extract speech information, which is then added to f, to produce more speech information com- pleted target feature f, Additionally, Sigmoid((n.) M,, isolates purer noise information from the non-target features (The reason for using "iiimmid M,,"" instead of "1 M,," " is explained in
5

---

Published as a conference paper at ICLR 2025
M
Jun
[Sgeood
(a) Interaction Module for Encoder Part
(b) Interaction Module for Decoder Part
Figure 4: Network architecture of the Interaction Module (IM): (a) The IM in the encoder extracts speech from non-target features guided by target features and removes noise from target features guided by non-target features. (b) The IM in the decoder, where noise is largely suppressed, refines discarded speech information from non-target features using target feature guidance for a more complete speech reconstruction.
Toр 25%
Gany Ix
Softmas
Softmar
Top AELSOON
MHSA
Softmax
Softmax
Copy 1xl
2ELEON
Hottorn
Softeek
Soitman
Bottomn 20%
fz
Softraax
Softmax
Xorm
(a) FRI-Conformer
(b) FRI-MHSA
Figure 5: Network architecture of the
Conformer (FRI-Conformer)) in Multi- Head Self-Attenionn (FRI-
the global module, featuring the crucial Recy-Re--etteettteeee MHSA). FRI-MHSA classifies input features into four parts based on attention map energy distri- bution, using the highest and lowest energy regions to guide speech extraction from mid-low energy features and noise extraction from mid-high energy features.
the Appendix B.3), which is combined with f to guide noise extraction from f.. before feeding into convolutional module to generate a mask M, that is subtracted by "Sigmoid(f.)". and the absolute value is taken afterward, i.e., [Sigmoid(f) M, to filter out remaining noise.
LM-D: the Decoder, input features typically contain minimal or no noise components. As il- lustrated in Figure [)), non-target information often contains more speech components that difficult to use for target speech reconstruction. Therefore, IM-D ensures mutual guidance between target and non-target features, allowing speech information from non-target features to contribute effectively to the reconstruction of the target speech, thus reducing speech distortion. As shown in Figure 4 IM-D concatenates f, and f, and processes them through convolutional module to generate an intermediate feature. This intermediate feature is then processed by two separate con- volutional modules to produce masks for and f,.. The masked f, and f, are then combined to produce the output target feature with enhanced speech information.
3.2 GLOBAL MODULE
The GM designed to overcome the limitations of the LM. which can only extract target and non- target features within a limited receptive field. By integrating the long-range contextual modeling capabilities of Transformers (Vaswani et al., (017). the Global Module facilitates the extraction of these features from global perspective of the speech signal. As depicted in Figure 2 the GM adopts a dual-path attention-based structure (Abdulatif et al.) [222 Wang et al., 202 and employs two FRI-Conformer blocks in sequence. The first stage captures time dependencies with an input shape of B'' xTx C, where B represents the batch size, while the second stage captures frequency dependencies with input shape of BT x F' x C. As depicted in Figure 5 a). each FRI-Conformer employs two half-step feedforward networks (FFNs) with Filter- citter-Recycle-ee Interguide Multi-head Self-attention (FRI-MHSA) module in between.
Recycle-Interguide

---

Published as a conference paper at ICLR 2025
The design of FRI-MHSA is based on the energy distribution in the attention feature map, where high-energy regions guide the extraction of speech features in low-energy regions, and low-energy regions guide the extraction of noise features in high-energy regions. Given a query q, key k, and value v, the output of dot-product attention is generally formulated as:
Att(q. k. v) softmax(qk )y.
(4)
In work, FRI-MHSA segments the feature map obtained from qk based on energy levels, tracting the top 25% highest energy regions, the upper-middle 25%, the lower-middle 25%, and the bottom 25% lowest energy regions. We draw inspiration from the top-k operation (Chen al., 20236 Xiao et al., 2024) to implement this extraction and assign an infinitesimal value to the un- extracted portions of each feature map. Subsequently, we apply softmax to each of the four feature maps, generating four masks that extract four types of features from V. The mask generated from the top 25% energy feature map extracts target features fJ from v. primarily containing speech informa- tion, which guides the extraction of speech information from the lower-middle 25% energy feature map f3. Conversely, the mask corresponding to the bottom 25% energy feature map extracts non- target features f4 from v. primarily containing noise information, which in turn guides the extraction of noise information from the upper-middle 25% energy feature map fz- This guided extraction method is based calculating the cross- sosssaiiml between two types of features through matrix multiplication, allowing the extraction of the parts from the guided features that are most similar to the guiding features. Therefore, the above process can be expressed as:
f3. = softmax [f1rg] )ss.
f3.n = softmax - * (f1fg) ))s
(5)
fz softmax (faf2 f2,
f2.1 softmax * (faf² f)
(6)
where fz,; and f2 represent the target and non-target features extracted from f2, and fa. and f3.n represent the target and non-target features extracted from fz. Finally, the target feature is obtained by f, = fi + f2, + f3.1, while the non-target feature is obtained by f. = f, + f2,n + fs.n. These features, f and f, are fed into IM-E for further processing.
4 EXPERIMENTAL SETUP
4.1 DATASET
We trained FIRING-Net using the Interspeech 2021 DNS Challenge dataset (Reddy et al. 2021 sampled at 16 kHz. A total of 60,000 reverberant speech clips, approximately 500 hours in dura- tion, were generated, with 55,000 clips designated for training and 5,000 for validation. The noise clips were mainly sourced from Audioset (Gemmeke et al. 2017). DEMAND (Thiemann et al., 20) and Freesound (Fonseca al. 2017). During training, the audio was randomly segmented into 4-second clips and processed with randomly selected room impulse responses (RIRs) from OpenSLR26 and Ope2SLR28 (Ko et al., 2017) (Too in the range from 0.3s 1.3s). The noisy speech was created by mixing reverberant speech with noise, with the SNR range set between -5 dB and 5 dB. We selected two datasets as the test sets for performance evaluation under various unknown noise conditions:
WSJO-SI 84 + NOISEX-92 We selected 651 utterances from speakers in the WSJO-SI 84 dataset (Paul & Baker 1992). Noise samples were taken from the NOISEX-92 dataset (Varga & Steeneken 1993), and test mixtures were created by combining these noise samples with the speech SNR levels of -5 dB, 0 dB, and 5 dB in reverberant conditions, with Tou values randomly selected between 0.3s and 1.3s.
AVSpeech AudioSet: The AVSpeech dataset consists of public instructional YouTube videos, from which 3-10s clips were automatically extracted, ensuring that the only audible sound in each clip is from single speaker (Ephrat et al. 2018). For our experiments, clips from the test set, utilizing only the audio portions. Four representative noise types: babble, engine, baby cry and laughter, and traffic, from the Audioset dataset (Gemmeke et al. 2017)) were introduced, and noisy speech is generated through a weighted linear combination of clean utterances from AV Speech and noise segments from AudioSet,
downloaded 1,199
AV Speech + 0.3 up AudioSet
I; 5)
AudioSet represent 4-second randomly sampled speech and noise segments, AV Speech and og where 8j respectively.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with selected baseline models WSJO-ST 84+NOISEX-92 with different
SNR levels. Bold and underline indicate the best and second-best results.
-3
d
d
Method
Param.
STOT
PESQ
SI-SNRi
STOI
PESQ
ST-SNRi
STOT
PESQ
ST-SNRI
Umprocessed
0.6172
1.48
0.7813
1.76
0.86669
1.92
PHASEN
6,41M
0.7886
2.25
12.84
0.8981
2.85
11.82
0.9238
3.09
9,45
SN-Net
8.14M
0.8157
2.31
13.18
0,9031
2.93
12.10
0.9305
3.14
10.05
Inter- SubNet
2.29M
0.8038
2.29
13.41
0.8968
2.89
11.95
0,9287
3.13
10.78
CMGAN
1.83M
0.8205
2.38
14.29
0.9085
2.97
12.53
0.9324
3.17
11.65
HENSE
2.05M
0.8342
2.43
15.68
0.9112
3.03
13.69
0.9384
3.21
11.74
FIRING-Net
1.81M
0.8414
2.51
16.11
0,9195
3.09
14.20
0.9430
3,28
12.06
Table 2: Comparison with selected baseline models AVSpeech AudioSet with different types
of noise. Bold and underline indicate the best and second-best results.
Method
Babble
Engine
Baby
Laughter
Traffic
STOI
PESQ
SI-SNKi
STOI
PESQ
S1-SNKi
STOI
PESQ
SI-SKKi
STOI
PESQ
SI-SNRI
U Umprocess
0.7971
1.56
0.7992
1.62
0.7876
1.75
0.8032
1.88
PHASEN
11.69
SN-Net
0.9214
2.94
12.11
0.9362
3.02
11.98
0.9198
2.94
11.86
0.9298
3.00
12.06
SuhNer
0.9226
2.92
11.96
0.9301
2.99
11.83
0.9213
2.99
12.01
0.9185
2.96
12.10
CMGAN
0.9354
3,03
12.36
0.9472
3.08
12.57
0.9387
12.34
0.9323
12.39
SENet
0.9437
3.07
13.24
0.9521
3.13
13.32
019453
3.10
12.96
0.9453
3.14
13.50
FIRING-Net
0.9558
3.17
3.21
14.12
0,9582
14.32
0,9541
3.23
14.16
4.2 MODEL SETTINGS AND EVALUATION METRICS
We trained the proposed model for 120 epochs with a batch size of 2, utilizing the Adam optimizer with an initial learning rate of 0.001. . If the best model was not identified for 15 consecutive epochs, the learning rate was halved. Early stopping was employed, terminating the training if the best model was not found after 30 consecutive epochs. The STFT was applied using Hanning win- dow with ms window length and 16 ms frame shift convert the signal into the frequency domain. The number of channels C for all convolutional layers was set to 32. In the FRI-MHSA module, we used attention heads. To mitigate overfitting, the dropout probability for all layers was set to 0.1. Four loss functions are utilized for model training: Magnitude Loss, Phase Loss, Complex Loss, and Time Loss. use the following three commonly seen SE metrics for evalua- tion purposes. PESQ: wideband Perceptual Evaluation of Speech Quality (Rix et al. 2000)) STOI: Short-Time Objective Intelligibility (Taal et al. 2011). SI-SNRi: Scale-Invariant Signal-to-Noiss Ratio improvement (Le Roux et al. 2019). For all the metrics, the higher the score, the better the performance.
More details of experiential settings be found in Appendix A.
RESULTS
5.1
MODEL COMPARISON
We conducted extensive experiments to quantitatively compare the SE performance our proposed FIRING-Net with some existing SE models. Baseline models including PHASEN (Yin al. 2020
SN-Net (Zheng et al. 2021). Inter-SubNet (Chen et al./ 2023a),, CMGAN (Abdulatif et al. 2024
and MP-SENet (Lu al. 2023). PHASEN employs interactive framework for jointly modeling magnitude and phase information, while SN-Net utilizes stual-brand architecture to directly model the interaction between speech and noise features. Inter-SubNe explores cross-bamt dependencies through subband interaction mechanisms. In contrast, CMGAN and MP-SENet, despite being SOTA SE models, not incorporate interactive feature modeling strategies. Note that these baseline models and the proposed FIRING-Net trained the same training set and evaluated the same test set.
WSJO-SI 84NSSXXX This dataset provides controlled environment combining speech with various noise types, making it ideal for evaluating the performance of SE models under standard- ized conditions. As detailed in Table PHASEN, SN-Net. and Inter-SubNet fall short compared to CMGAN and MP-SENet due to less effective feature interaction strategies. PHASEN's interaction between magnitude and phase spectra is limited by the lack of distinct features in the phase spec-
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablative analysis of LM-E and LM-D by measuring STOI, PESQ, SI-SNRi, and number of trainable parameters.
Encoder
Decoder
STOI
PESQ
SI-SNRi
Params.
LM-E
LM-D
0.9135
3.03
13.96
1.8TM
DCCM
DCCM
0.8657
2.84
12.64
1.89M
LM-E
DCCM
0.8854
2.93
12.89
1.87M
DCCM
LM-D
0.8782
2.94
12.71
1.87M
FGCM
FGCM
0.8738
2.88
12.84
1.78MI
LM-E
FGCM
0.8927
2.95
13.27
1.80M
FGCM
LM-D
0.9003
2.93
13.14
1.79M
trum, making the interaction less meaningful. SN-Net NN-Ne attempts to model noise for interaction, but the unpredictable nature of noise undermines its effectiveness. Inter-SubNet's s sub-band interaction fo- comparison, CMGAN
cuses primarily on target speech modeling, offering limited improvement. and MP-SENet perform better with direct feature processing approaches, but FIRING-Net achieves (FRI) framework. This method facili- tates mutual guidance between target and non non-target features, effectively refining speech information while suppressing noise. FIRING-Net achieves a PESQ score of 2.51 and STOI of 0.8414 under -5 dB SNR, significantly outperforming other models through more effective feature interaction.
superior results by introducing the
This dataset presents a more complex and varied set of challenges for speech enhancement models compared to the controlled WSJO-SI 84 NOISEX-92 environment. It in- cludes wide range of real-world noise types and speaker variations. As shown in Table 2 the complex and diverse noise types highlight the limitations of PHASEN, SN-Net, and Inter-SubNet. PHASEN's ineffective interaction between magnitude and phase spectra, SN-Net's 's reliance on noise prediction, and Inter- Suter-S focus on larget speech modeling restrict their ability to generalize in dynamic noise conditions. FIRING-Net leveraging its FRI framework, excels in handling overlap- ping and non-stationary noise. By using mutual guidance to refine speech and noise features, outperforms CMGAN and MP-SENet across all metrics. For example, it achieves PESQ scores of 3.17 and 3.23 under babble and traffic noise, respectively, showcasing its robust generalization and effectiveness in diverse real-world scenarios.
"SpeecchAuuiinn
5.2 ABLATION STUDY
We ablated the design choices and measured the average increase on the WSJO-SI 84 NOISEX-92 dataset. The following be summarized from the ablation results:
LM: To assess the effectiveness of the proposed LM, we introduced two alternative modules: the densely connected convolutional module (DCCM), consisting of 6 cascaded convolutional lay- ers (Pandey & Wang 2020), and the fully gated convolutional module (FGCM), with 3 ELÚ- activated gated convolutional blocks (Tan & Wang 2019). As shown in Table 3 both modules were configured to have similar number of trainable parameters for a fair comparison. The most significant performance drop occurred when LM-E and LM-D were removed. FGCM, with its gating mechanism, effectively emphasized target features and outperformed DCCM. Overall, sub- stituting LM-E or LM-D with DCCM or FGCM in the encoder or decoder led a notable decline in performance across key metrics like STOI, PESQ, and SI-SNRi.
GM: To validate the effectiveness of the GM, we conducted two sets of comparative experiments, as presented in Table 4 The first set examines the FRI-MHSA module, where the qk' feature map is initially divided into four segments based on energy levels. We adjusted the number of segments and observed a significant performance drop when reduced to one segment (equivalent to standard MHSA). As the number of segments increases from to 4, performance improves, demonstrating the advantage of subdividing the feature map and using high- and low-energy regions to guide intermediate feature extraction. However, performance plateaus beyond four segments, suggesting that further subdivision yields no additional benefit. Additionally, increasing the number of segments affects real-time processing efficiency. The second set of experiments assesses the GM's effectiveness handling features across the time and frequency dimensions. Removing the processing of either dimension results in significant performance decline.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study GM by measuring STOI, PESQ. SI-SNRi, number of trainable parame-
ters, and real-time factor.
Method
STOI
PESQ
SI-SNRi
Params.
CPU RTF
GM (FRI-MHSA with 4 parts)
0.9135
3.03
13.96
1.8TM
0.60s
FRI-MHHA (1 Part)
0.8594
2.80
12.57
1.74M
0.39s
FRI-MHSA (2 Parts)
0.8862
2.89
12.89
1.80M
0.41s
-FRIMMHA (3 Parts)
0.8987
2.97
13.14
1.81M
0.52s
-FRIMIIIA (5 Parts)
0.9152
3.04
13.85
1.83M
0.72s
-FRI-MHSA (6 Parts)
0.9201
3.06
13.99
1.84M
0.87s
w/o Frequency Processing
0.8254
2.75
11.97
1.38M
0.37s
w/o Time Processing
0.8303
2.77
12.14
1.29M
0.39s
w/o GM
0.7631
2.64
10.52
1.03M
0.24s
Table 5: Ablation study on FRI Strategy in terms of STOI, PESQ, and SI-SNRi.
Method
STOI
PESQ
SI-SNRi
GM
0,9135
3.03
13.96
Target Feature
0,8677
2.86
12.88
Target and Original Features
0.8961
2.95
13.16
-Non-Target and Original Features
0.8306
2.69
9.52
LM-E
0.9135
3.03
13.96
-Target Feature
0,8738
2.84
12.83
Target and Original Features
0.9003
2.92
13.05
-Non- Non-Target and Original Features
0.7972
2.61
9.17
LM-D
0.9135
3.03
13.96
-Target Feature
0.8832
2.90
13.08
Target and Original Features
0,9033
2.97
13.29
-Non-Targe Target and Original Features
0.8992
2.93
13.38
FRI Framework: Table 3 demonstrates the effectiveness of this FRI strategy for SE. In the eval- uation of LM, we observe that LM-E using non-target and original features performs worse than LM-E using target and original features. Additionally, LM-E -Target Feature, which processes only target features, highlights the importance of target features. LM-E also surpasses LM-E -Target Original Features, confirming the value non-target features and the effectiveness of the FRI framework in leveraging both feature types. For LM-D, results differ slightly. The performance of LM-D -Target + Original Features and LM-D -Non-Target + Original Features similar and both exceed L.-DD -Target Feature. This suggests that non-target information in the decoder contains useful speech components not involved in reconstruction, improving performance when both fea- ture types are used. The evaluation of GM mirrors LM-E, reinforcing the effectiveness of the FRI framework. Notably, GM -Target Feature significantly outperforms GM -FRI-MISS (1 Part) as shown in Table indicating the superior efficacy of target feature extraction.
CONCLUSION
In this work, we propose an FRI framework that separates input features into target and non-target sets. These two sets guide each other to refine information, leading to the clustering of speech information in the target features and noise in the non-target features. We introduce FIRING-Net a speech enhancement network comprising two main components: LM and GM. Both modules integrate interaction mechanisms to enable mutual guidance between the features, where speech extracted from the non-target features and noise is filtered out from the target features. We conducted extensive experiments to validate the effectiveness of our method. From the results, we mainly conclude that: I) Non-target features filtered by the SE network still contain speech information. and recycling them with the FRI strategy significantly boosts performance; 2) Mutual guidance between target and non-tame non-target features is crucial for filtering noise from the target and recovering speech from the non-targe non-target features. Future research focuses on designing lightweight model and enhancing real-time performance enable deployment across various devices.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is supported by the National Natural Science Foundation of China (No.6247,343, No. 62171326, No. 62071342).
REFERENCES
Sherif Abdulatif, Ruizhe Cao, and Bin Yang. Cmgan: Conformer-based metric- gattrcc-ganna for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The Conversation: Deep Audio- Visual Speech Enhancement. In Proc. Interspeech 2018, pp. 3244-3248, 2018. doi: 10.21437/ Interspeech.200 2018- 1400.
Yang Ai and Zhen-Hus Ling. Neural speech phase prediction based on parallel estimation architec- ture and anti-wrapping losses. In ICASSP 2023-2023 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe. The third 'chime' sseene''secchhhe separation and recognition challenge: Analysis and outcomes. Computer Speech & Language, 46:605-626, 2017.
Sebastian Braun and Ivan Tashev. A consolidated view loss functions for supervised deep learning-based speech enhancement. In 2021 44th International Conference Telecommuni- cations and Signal Processing (TSP), pp. 72-76. IEEE, 2021.
L Burget, K Vesely, A Ghoshal, and D Povey. Sequence-discriminne training of deep neural net-works. In Proc. Interspeech, 2013.
Jingdong Chen, Jacob Benesty, Yiteng Huang, and Simon Doclo. New insights into the noise reduc- tion wiener filter. IEEE Transactions on audio, speech, and language processing, 12(4) 1234, 2006.
Jun Chen, Zilin Wang, Deyi Tuo, Zhiyong Wu, Shiyin Kang, and Helen Meng. Fullsubnet+: Channel attention fullsubnet with complex spectrograms for speech enhancement. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7857- 7861. IEEE, 2022.
Jun Chen, Wei Rao, Zilin Wang, Jiuxin Lin, Zhiyong Wu, Yannan Wang, Shidong Shang, and Helen Meng. Inter-subnet: Speech enhancement with subband interaction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. Learning a sparse transformer network for effective image deraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5896-5990 2023b.
Hyeong-Seok Seoon Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, and Kyogu Lee. Real- time denoising and dereverberation wtih tiny recurrent u-net. In ICASSP 2021-2021 IEEE In- ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5789-5793 IEEE, 2021.
Feng Dang, Hangting Chen, and Pengyuan Zhang. DPT-FSNet: Dual-path transformer based full- band and sub-band fusion network for speech enhancement. ICASSP 2022-2022 IEEE Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6857-6861 IEEE, 2022a.
Feng Dang. Hangting Chen, and Pengyuan Zhang. Dpt-fsnet: Dual-path transformer based full-band and sub-band fusion network for speech enhancement. In ICASSP 2022-2023 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 6857-6861. IEEE, 2022b.
11

---

Published as a conference paper at ICLR 2025
Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Real Time Speech Enhancement in the Waveform Domain. In Proc. Interspeech 2020, pp. 2291-3295, 2020. doi: 10.21437/hnterspeechh. 2020-2409.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics (Proc. SIGGRAPHI), 37, 2018.
Eduardo Fonseca, Jordi Pons Puig, Xavier Favory, Frederic Font Corbera, Dmitry Bogdanov, An- dres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. Freesound datasets: platform for the creation of open audio datasets. In Hu X, Cunningham SJ, Turnbull D. Duan Z, editors. Proceedings of the 18th ISMIR Conference; 2017 oct 23-27; Suzhou, China. (Canada]: Interna- tional Society for Music Information Retrieval; 2017. p. 486-93. 93. International Society for Music Information Retrieval (ISMIR), 2017.
Szu-Wei Fu, Yu Tsao, Xugang Lu, and Hisashi Kawai. Raw waveform-base speech enhancement by fully convolutional networks. In 2017 Asia-Pacific Signal and Information Processing Associ- ation Annual Summit and Conference (APSIPA ASC), pp. 006-012. IEEE, 2017.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6633-6637. IEEE, 2021.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1664-1673, 2018
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back-projection net- work for video super-resolution. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3897-3906, 2019.
Shulin He, Wei Rao, Jinjiang Liu, Jun Chen, Yukai Ju, Xueliang Zhang, Yannan Wang, and Shidong Shang. Speech enhancement with intelligent neural homomorphic synthesis. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5. IEEE, 2023.
Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. Decrn: Deep complex convolution recurrent network for phase-aware speech en- hancement. In Interspeech 2020, pp. 2476, 2020. doi: 10.21457/Itterspeech.. 20tp://eerrpeeccc..22255525 2537.
Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on audio, speech, and language processing, 16(1):229-238, 2007.
Yuchen Hu, Chen Chen, Heqing Zou, Xionghu Zhong, and Eng Siong Chng. Unifying speech enhancement and separation with gradient modulation for end-to-end noise-robust speech sepa- ration. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Umut Isik, Ritwik Giri, Neerad Phansalkar, Jean-Mare Valin, Karim Helwani, and Arvindh Krish- naswamy. PoCoNet: Better Speech Enhancement with Prequencc-Posiition Embeddings, Semi- Supervised Conversational Data, and Biased Loss. In Proc. Interspeech 2020, pp. 2487-2491, 2020. doi: 10.24457/mierrpecch.2000000
Scokhwan Jo and Chang D Yoo. Psychoncoustically constrained and distortion minimized speech enhancement. IEEE Transactions on audio, speech, and language processing, 28(8))2099-211 2010.
12

---

Published as a conference paper at ICLR 2025
Jaeyoung Kim, Mostafa El- El-Khamy, and Jungwon Lee. T-gsa: Transformer with gaussian weegste self-attention for speech enhancement. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6649-6653. IEEE, 2020.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE inter- national conference on acousties, speech and signal processing (ICASSP), pp. 5220-5224. IEEE, 2017.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630 IEEE, 2019.
Jinkyu Lee, Jan Skoglund, Turaj Shabestary, and Hong-Goo Kang. Phase-sensitive joint learning algorithms for deep cearnnngbasse speech enhancement. IEEE Signal Processing Letters, 25(8): 1276 1280, 2018.
Andong Li, Wenzhe Liu, Chengshi Zheng. Cunhang Fan, and Xiaodong Li. Two heads are better than one: A two-stage complex spectral mapping approach for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1829-1843, 2021.
Andong Li, Shan You, Guochen Yu, Chengshi Zheng, and Xiaodong Li. Taylor, can you hear now? taylor-unoolding framework for monaural speech enhancement. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 4193-4200. International Joint Conferences Artificial Intelligence Organiza- tion, 7 2022. doi: 224466//jcai.2222/5588 URL
2022 / 582
Main Track.
Yihao Li, Meng Sun, Xiongwei Zhang, et al. Scale-aware dual-branch complex convolutional re- current network for monaural speech enhancement. Computer Speech & Language, 86: 86:101618, 2024.
Hsin-Yi Yi Lin, Huan-Hsin Tseng, Xugang Lu, and Yu Tsao. Unsupervised noise adaptive speech enhancement by discriiminator-zonsttanedd optimal transport. Advances Neural Information Processing Systems.
2021.
Liang Liu, Haixin Guan, Jinlong Ma, Wei Dai, Guangyong Wang, and Shaowei Ding. A mask free neural network for monaural speech enhancement. INTERSPEECH 2023, pp. 2468-2472, 2023. doi: 222444///neerpeech20223333
Wenzhe Liu, Andong Li, Yuxuan Ke, Chengshi Zheng, and Xiaodong Li. Know Your Enemy, Know Yourself: Unified Two-Stage Framework for Speech Enhancement. In Interspeech, pp- 186-190, 2021.
Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, and Wan-Chi Siu. Hicrarchical back projection network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 0-0, 2019.
Ye-Xin Lu, Yang Ai, and Zhen-Hua Ling. MP-SENet: Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra. In Proc. INTERSPEECH 2023, pp. 3834-3838, 2023. doi: 10.2 4377/nnterspecchh22222
Yi I Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-freyuency magnitude masking for speech separation. IEEF/ACM transactions audio, speech, and language processing, 27(8): 1256-1266, 2019.
Shubo Lv, Yanxin Hu, Shimin Zhang, and Lei Xie. DCCRN+: Channel- Wise Subband DCCRN with SNR Estimation for Speech Enhancement. In Proc. Interspeech 2021, pp. 2816-2820, 2021. doi: 10.2143 00.21437/nnerspeecc..200 2021-114822228
Jianfen Ma, Yi Hu, and Philipos C Loizou. Objective measures for predicting speech intelligibility in noisy conditions based on new -apoitttee functions. The Journal of the Acoustical Society of America, 125(5):3387-3444 2009.
13

---

Published as a conference paper at ICLR 2025
Babafemi O Odelowo and David V Anderson. A noise prediction and time-domain subtraction approach to deep neural network based speech enhancement. In 2017 16th IEEE International Conference Machine Learning and Applications (ICMLA), pp. 372-377. IEEE, 2017.
Javier Ortega-Garcis and Joaquín Sonzzlez-Rodriguezz. Overview of speech enhancement tech- niques for automatic speaker recognition. In Proceeding of Fourth International Conference Spoken Language Processing. ICSL,P'6,, volume 2, pp. 929-932. IEEE, 1996.
Keiron O'shea and Ryan Nash. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458, 2015.
Ashutosh Pandey and DeLiang Wang. TCNN: Temporal convolutional neural network for real-time speech enhancement the time domain. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6875-6879, IEEE, 2019.
Ashutosh Pandey and DeLiang Wang. Densely connected neural network with dilated convolutions for real-time speech enhancement in the time domain. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6629-6633. IEEE, 2020.
Se Rim Park and Jin Won Lee. A fully convolutional neural network for speech enhancement. Proc. Interspeech 2017, pp. 1993-1997, 2017.
Douglas B Paul and Janet Baker. The design for the wall street journal- -baee csr corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23- 26, 1992, 1992.
Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures capture local and global context for specch recognition and understanding. In International Conference on Machine Learning, pp. 17627-17643. PMLR, 2022.
Wei Rao, Chenglin Xu, Eng Siong Chng, and Haizhou Li. Target speaker extraction for multi-talker speaker verification. Proc. Interspeech 2019, pp. 1273-1277, 2019.
Chandan KA Reddy, Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan. Icassp 2021 deep noise suppression challenge. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2021.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesa)-a new method for speech quality assessment telephone networks and codecs. In 2001 IEEE international conference acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), volume 2, pp. 749-752. IEEE, 2001.
Gregory Sell, David Snyder, Alan McCree, Daniel Garcia-Roment Jesús Villalba, Matthew Ma- ciejewski, Vimal Manohar, Najim Dehak, Daniel Povey, Shinji Watanabe, et al. Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural DIHARD chal- lenge. In Interspeech, pp. 2808-28888800 2812, 2018.
V Srinivasarao and Umesh Ghanekar. Speech enhancement-an enhanced principal component anal- ysis (epca) filter approach. Computers & Electrical Engineering, 85: 10665, 2020.
Lei Sun, Jun Du, Li-Rong Dai, and Chin-Hui Lee. Multiple-targer deep learning for LSTM-RNN based speech enhancement. In 2017 Hands-free Speech Communications and Microphone Arrays (HSCMA), pp. 136-140. IEEE, 2017.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction of time- frequeeny weighted noisy speech. IEEE Transactions audio, speech, and language processing, 19(7):2125-2138) 2011.
Ke Tan and DeLiang Wang. Learning complex spectral mapping with gated convolutional recurrent networks for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:380-390, 2019.
14

---

Published as a conference paper at ICLR 2025
Chuanxin Tang, Chong Luo, Zhiyuan Zhao, Wenxuan Xie, and Wenjun Zeng. Joint time- fene-freueecc and time domain learning for speech enhancement. Proceedings of the Twenty-Ninth Interna- tional Conference International Joint Conferences on Artificial Intelligence, pp. 3816-3822, 2021.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi cultt- acoustic noise database (demand); A database of multichannel environmental noise recordings. In Proceedings of Meetings Acoustics, volume 19. ATP Publishing, 2013a.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics ICA2013, volume 19, pp. 035081. Acoustical Society of America, 2013b.
Cassia Baattiii--Btihho,,, Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investigating mn- based speech enhancement methods for noise-robust text-to-speech. In SSW, pp. 146-152, 2016.
Andrew Varga and Herman JM Steeneken. Assessment for automatic speech recognition: li. noisex- 92: A database and experiment to study the effect of additive noise on speech recognition systems. Speech communication, 12(3)2447-251, 1993.
Saeed V Vaseghi. Advanced digital signal processing and noise reduction. John Wiley & Sons, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of large regional accent speech database. In 2013 international conference oriental COCOSDA, pp. 1-4, IEEE, 2013.
Yukoh Wakabayashi, Takahiro Fukumori, Masato Nakayama, Takanobu Nishiura, and Yoichi Ya- mashita. Single-channel speech enhancement with phase reconstruction based on phase distor- tion averaging. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(9):
1559-1569, 2018
Kai Wang, Bengbeng He, and Wei-Ping Zhu. TSTNN: Two-stage transformer based neural net- work for speech enhancement the time domain. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7098-7102. IEEE, 2021.
Kevin Wilson, Michael Chinen, Jeremy Thorpe, Brian Patton, John Hershey, Rif A Saurous, Jan Skoglund, and Richard F Lyon. Exploring tradeoffs in models for low-latency speech enhance- ment. In 2018 16th International Workshop on Acoustic Signal Enhancement (wweineeett//AAAAe (/WAENC), pp. 366-370. IEEE, 2018
Scott Wisdom, John R Hershey, Kevin Wilson, Jeremy Thorpe, Michael Chinen, Brian Patton, and Rif A Saurous. Differentiable consistency constraints for improved deep speech enhancement. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 900-904. IEEE, 2019.
Yangyang Xia, Sebastian Braun, Chandan KA Reddy, Harishchandra Dubey, Ross Cutler, and Ivan Tashev. Weighted speech distortion losses for neural-network-basee real-time speech enhance- ment. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 871 1875 IEEE, 2020.
Yi Xiao, Qiangqiang Yuan, Kui Jiang, Jiang He, Chia-Wen Lin, and Liangpei Zhang. TTST: A top-k token selective transformer for remote sensing image super-resolution, IEEE Transactions on Image Processing, 2024.
Xinmeng Xu, Weiping Tu, and Yuhong Yang. CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148:31-39, 2023.
15

---

Published as a conference paper at ICLR 2025
Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee. An experimental study speech enhancement based on deep neural networks. IEEE Signal processing letters, 21(1):65 68, 2013.
Dacheng Yin, Chong Luo, Zhiwei Xiong, and Wenjun Zeng. Phasen: A hhhas--and-hammoniss speech enhancement network. In Pruceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9458-9463 2020.
Dacheng Yin, Zhiyuan Zhao, Chuanxin Tang, Zhiwei Xiong, and Chong Luo. TridentSE: Guiding Speech Enhancement with 32 Global Tokens. In Proc. INTERSPEECH 2023, pp. 3839-3843, 2023. doi: 10.24457/mnterspecch.000 2023-565.
Guochen Yu, Andong Li, Chengshi Zheng, Yinuo Guo, Yutian Wang, and Hui Wang. Dual-branch attenttoo transformer for single-channel speech enhancement. In ICASSP 2022-2022 IEFE International Conference on Acousticx, Speech and Signal Processing (ICASSP), pp. 7847- 7851. IEFE, 2022.
Changjiang Zhao, Shulin He, and Xueliang Zhang. SICRN: Advancing Speech Enhancement through State Space Model and Inplace Convolution Techniques. In ICASSP 2024-2024 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10506-10510. IEEE, 2024.
Haoran Zhao, Nan Li, Runqiang Han, Lianwu Chen, Xiguang Zheng, Chen Zhang, Liang Guo, and Bing Yu. deep hierarchical fusion network for fullband acoustic echo cancellation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9116. IEEE, 2022.
Chengyu Zheng, Xiulian Peng, Yuan Zhang, Sriram Srinivasan, and Yan Lu. Interactive speech and noise modeling for speech enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14549-14557, 2021.
A RECYCLING PROCESS
A pivotal aspect of the FRI strategy lies in recycling filtered information to derive non-target fea- tures. Inspired by Back Back-projection and Top-K techniques, we propose two distinct methodologies for non-target feature extraction: the calculation of residuals between input and output features and the utilization of energy distributions in the attention maps derived from self-attention mechanisms to identify and extract features corresponding to regions of low energy. These approaches are embed- ded within the LM and GM modules, facilitating a comprehensive and effective distinction between target and non-target features from both local and global analytical perspectives. To address the fea- sibility the proposed methods for extracting non-target features, we emphasize their design based on direct calculations rather than relying on learnable modules. This ensures that the attributes of the extracted features align closely with our hypothesis, which predominantly associates non-target features with noise information.
The extraction of non non-target features using residuals is rooted in the concept of back-projection (Haris al., 2018 2019 Liu al.) 2019). This approach assumes that the information filtered out by the network represents the dominant noise components, making highly relevant as non-target features. To ensure the success of this method, normalization layers are applied after each convolutional module. These layers maintain the energy range of the generated features within the scale of the input features, preventing the overall energy of the output from deviating. This step is critical, as unbounded variations in feature energy could distort the separation process and result in unintended feature overlaps or loss of relevant information. By maintaining consistency in energy distribution, the network effectively isolates noise-like elements as residuals, which then serve as refined non-target features.
Residual Calculation Between Input and Output:
Utilization of Energy Distributions in Self- Celf-Attention Maps: engy distributions in self-attention maps to extract non-target features. Specifically, attention mech- anisms segment the feature space into regions of varying energy. High-energy regions correspond to speech-dos ppeech-dominatt components (target features), while low-energy regions predominantly contain noise-like characteristics. By calculating the cross-similarity between energy-segmented regions,
The second method employs the en-
16

---

Published as a conference paper at ICLR 2025
Table 6: Comparison with other methods VoiceBank DEMAND dataset. "'" denotes the result
that is not provided in the original paper. Bold and underline indicate the best and second-best
results.
Models
Param. (M)
FLOPs (G)
WB-PESQ
CSIG
CBAK
COVL
STOI
Noisy
1.97
3.35
2,44
2.6.3
0.91
DEMCUS Defossez et al.
2020
33.53
77.8
3.07
4.31
3.40
3.63
0.95
TFT-Net (Tang al. 2021
5.81
295.0
2.75
3.93
3.44
3.34
SN-Net Zhenng et al.
3.12
4.39
3.60
3.77
DB-AIAT Yu
2022
2.81
68.0
3.31
4.61
3.75
3.96
DPT-FSNee Dang
2022b
0.88
55.7
3.33
4.58
3.72
4.00
0.96
CMGAN Abxlulati et a. 2024
1.83
81.3
3.41
4.63
3.94
4.12
0,96
TridentSE Yin et al.
2026
3.03
59.8
3.47
4.70
3.81
4.10
0,96
MP-SENet Lu al. 2023
2.05
84.7
3.50
4.73
4.22
0,96
FIRING-Nel
181
64.2
3.57
4.79
3,98
4.33
0,96
this approach ensures a clean separation of target and non-target features without introducing cross- information. critical design aspect here is the uniform application of energy segmentation and feature extraction across the same spectral representation. This prevents any inadvertent overlap be- tween the extracted target and non-target features. Furthermore, the segmentation approach avoids the use of additional learnable parameters, ensuring that the extracted non-target features remain in- herently noise-dominat dominant,, aligning with the hypothesis that noise information predominantly resides in low-energy regions.
B SUPPLEMENTARY EXPERIMENTAL SETUPS
B. LOSS FUNCTION
We employ multi-level loss functions to train the proposed FIRING-Net. Following the approach in (Braun Tashev| [202]] Lu et al.) 2023, Abdulatif et al. 2024), we utilize time-domain loss (LT), magnitude loss (LM), and complex loss (Lc), which defined as:
Cr E, 8|[2],
LM Es,,. S... [1|S Sm
(8)
Lc = F.S.S. s, [||,, 1.
+ S. [|S: 1111.
where (S,. S.) and ($., S.) represent the real and imaginary parts of the clean and enhanced complex spectrogram. Previous works optimize the phase spectrogram within the complex spectrogram, since the absolute distance between two phases may not be their actual distance. Following the (Lu et al., 2023 Ai & Ling, 2023), adopt anti-wrapping phase loss to optimize the phase spectrogram. The anti-wrapping phase loss includes three sub-losses, i.e., instantaneous phase loss LIP, group delay loss LGD. and instantaneous angular frequency loss LIAF, which are defined as:
LIP Esp S Illfaw -Sp-$p)l Sp)
(10)
LGD E. EA Sp) (WA
{p))
(11)
LIAF EspriSP Sp) WWW (S Sp))
(12)
where (tw = |t 2n round( 1 R is an anti- function, which is used to avoid the error expansion issue caused by phase wrapping. ADF and ÅDT represent the differential operators along the frequency axis and time axis, respectively. The anti-wrapping phase loss Lp is defined as:
Lp Cip LCD LIAF.
(13)
Finally, the total loss Loortt the linear combination of Lr, LMs Los and Lp,
L or Lr + LMLLM + acLe arLr.
(14)
where ar, ams, ac. and ap are hyperparameter and we follow (Lu et al. 2023) to set them to 0.2, 0.9, 0.1, and 0.3.
17

---

Published as a conference paper at ICLR 2025
Table 7: Comparison with other methods DNS Challenge Test Set. Bold and underline indicate
the best and second -best results.
Model
Feat.
Param.
WE-PESS
NE-PESQ
With Reveh
STOI
SESDR
WE-PESO
NE-PESO
Without Revert
STOP
SESOR
Noisy
1.822
2.753
36.62
9003
1,582
2.454
91.32
9.070
DCCRN-E Hl
RI
3.70
3.077
3.266
Com-TuN. 20000
2019
Waveform
5.08
2,570
2.730
PoCoNes (N PUAD
RI
50,00
2.832
2,748
DOCRN- 1oaavell PURLS
RI
3.50
3.300
3,330
TRU-Nat (brruee
Mag
0.38
2,740
3,350
91.29
14.87
2.860
3,360
96. 32
17.55
CTS-NNa (oueni mirnt
Mag-R1
4.99
3,020
3,470
92.70
15.58
2.940
3.420
96.66
17.99
FullSuhNer (Parre (:124)
Mag
5.64
3.057
3.584
92. 11
16.04
2,882
3.428
96. 12
17.30
FullSubNict
Mag+RI
8.67
3.177
3.618
93.61
16.41
3,002
3,503
96.67
18.00
Tayl (Cilve Policc
RI
5,10
3.330
3,600
93.99
17.10
3,220
3.590
97. 36
19.15
SICRN ZZunherit P010
RI
2.16
2.891
3433
82.59
13.14
26624
3.233
95.83
16,00
CARNNHSTTHES theseal 2013
DFT
3.063
3.519
93.30 30
16.70
2.892
3.431
96 70
18,0
MFNer (Liu braiy (000))
RI
3,400
3,740
97 7M
20.31
FIRING
PhatMag
3,632
3363
96.65
17.93
3,505
3.781
98,09
20.10
B.2 EVALUATION METRICS
Perceptual Evaluation of Speech Quality (PESQ) is standard method for objectively sessing how speech quality is perceived by listeners (Rix et al. 2001). It provides an estimate of the subjective mean opinion score (MOS) for normal-hearing individuals, specifically evaluating audio quality in noisy or distorted telephone networks (Ma et al. 2009). The PESQ scale typically ranges from 1.0 4.5, making it widely-used metric for measuring the performance of SE algorithms and the clarity of processed speech.
PESQ:
The Short-Term Objective Intelligibility (STOI) metric, which ranges from 0 to 1, provides an objective measure of speech intelligibility. It is particularly effective in evaluating speech in environments with temporally modulated noise or time-frequency processed signals, and is designed for normal-hearing listeners (Taal et al. 201I).
STOI:
SI-SNRi: The Scale-Invariant Signal-to-Noise Ratio Improvement (SI-SNRi) metric used to assess the quality of enhanced speech. It is derived from the scale-invariant signal-to-noise ratio (SNR) (L Roux et al. 2019), with higher values indicating better performance. SI-SNRi is defined as follows:
SI-SNRi(X, 8, SI-SNR(s, SI-SNR(s,
(15)
C ADDITIONAL EXPERIMENTS
C.1 EVALUATION ON BENCHMARK DATASET
To ensure fair comparison between the proposed model and other SOTA models, we trained FIRING-Net the training set of the public VoiceBank+DEMAND (Valentini- Botinhao et al.) 2016) and DNS Challenge datasets and evaluated its performance on the test set.
The VoiceBank DEMAND test includes two unseen speakers from the VoiceBank dataset (Veaux et al.) 2013). with 20 distinct noise conditions. These conditions consist of five noise types from the DEMAND dataset (Thiemann et al. (013b), each tested at four SNR levels (17.5, 12.5, 7.5, and 2.5 dB), resulting in total of 824 test samples. Each test speaker has approximately 20 different sentences per condition. To evaluate and compare the enhanced speech quality across methods, we use mean opinion score (MOS) predictors: signal distortion (CSIG), background intrusiveness (CBAK), and overall quality (COVL), with scores ranging from I to 5 (Hu & Loizou, 2007 Additionally, wide-band PESQ (WP) and STOI are also employed. The averaged SE results on the VoiceBank DEMAND dataset are presented Table in which observe that the proposed FIRING-Net achieves superior performance than several SOTA SE models across all performance measures.
VoiceBank + DEMAND:
The proposed method is further evaluated the DNS Challenge benchmark, comparing its performance against state-of-the-as art (SOTA) approaches. Table pressents the average results across key metrics, including STOI, wide-band PESQ (WP), harrow-band PESQ (NP). and scale-ine invariant source-tue source-to-distee to-distortion ratio (SI-SDR) (dB). During training, noisy mixtures were gen- erated with random SNRs ranging from -5 to 20 dB. As shown in Table FIRING-Net consistently
DNS Challenge:
18

---

Published as a conference paper at ICLR 2025
3.25
3.15
B+E
B+T
B-T
B+E+T
PESQ
2.95
2.85
2.75
2.65
RIRINGGNNet
MP-SBNet
CMGAN
Iater-Sub Net
SN-Net
PHASBN
Models
(a) PESQ Score
0.98
B+ B+E+T
B+T
B+T
0.94
0.92
0.86
0.84
PIRING-Net
MP-SENet
(b) STOI Score
15
B+E+T
SI-SNRI (dB)
12
PIRING Net
MP-SENet
CMGAN
Inter: Sub Net
SN Net
PHASEN
Models
(c) SI-SNRi Score
Figure 6: Comparison in PESQ. STOI, and SI-SNRi average for cases with multiple noise types, in which B, E, and T denote babble, engine, and traffic noisy types, respectively.
outperforms competing methods, demonstrating its robust speech enhancement capabilities on the DNS Challenge dataset.
C.2 EVALUATION MORE COMPLEX NOISE CONDITIONS
We further show the advantages and flexibility of the proposed FIRING-Net by exploring FIRING- Net performance in environments where multiple types of noise are contained in the target do- main (Lin et al. 2021), Figure presentss the PESQ and STOI results on multiple target noise types, where B. E, and T indicate "Babble", "Engine", and "Traffic". respectively. We observe that the results using 1, 2, and 3 noise types in the background environment are comparable. Figure shows the enhanced spectrograms of selected baseline models and the proposed FIRING-Net. One can observe that the proposed model sufficiently preserves the spectral details while suppressing the residual noise over the selected baselines.
19

---

Published as a conference paper at ICLR 2025
(a) Nolsy
(b) PHASEN
(C) Se-Nei
(d) unter-SubNe
(c) Clean
(1) CMGAN
(B) SE-SEN
(h) CMGAN
Figure 7: Visualization of spectrograms of enhanced speech generated from FIRING-Ne! and lected baseline models. The noisy sample contains two noise types, i.e., babble and engine, under -dBB SNR condition.
(a) Noisy Input
(b) CMGAN Enhanced
(c) MP-SENet Enhanced
(d) FIRING-Net Enhanced
(c) Clean Target
Figure 8: Visualization of spectrograms of enhanced speech generated from CMGAN, MP-SENet, and FIRING-Net in babble noise environment under -5dB SNR condition.
To better demonstrate the performance of FIRING-Net in scenarios where noise and speech compo- nents exhibit high similarity, we present spectrogram samples processed by CMGAN, MP-SENet, and FIRING-Net for Babble noise and Factory noise. As shown in Figure 8 Babble noise consists
20

---

Published as a conference paper at ICLR 2025
Noisy Input
(b) CMGAN Enhanced
(c) MP-SENet Enhanced
(d) FIRING-Net Enhanced
(e) Clean Target
Figure 9: Visualization of spectrograms of enhanced speech generated from CMGAN, MP-SENet, and FIRING-Net factory noise environment under -5dB SNR condition.
of background noise heavily influenced by speech interference from multiple speakers, making its acoustic characteristics more similar to the target speech compared to other noise types. Similarly, depicted in Figure 2
Factory noise was selected due its composition of low-SNR white noise, which heavily masks speech signals. This masking effect suppresses the inherent acoustic charac- teristies of the speech, making it challenging distinguish from the background noise. Figure and and Figure 9 illustrate the spectrograms under Babble and Factory noise conditions, respectively. The two rightmost columns provide a detailed view of the corresponding spectrograms on the left. From these figures, it is evident that in such extreme cases, where noise and speech components are highly similar and difficult to differentiate, CMGAN and MP-SENet struggle to effectively extract and restore the speech components. In contrast, FIRING-Net leverages the ReeeeeRReycll--eeeeeeee (FRI) strategy to achieve a more complete restoration speech components these challenging scenarios.
Additionally, to evaluate the robustness of FIRING-Net under extremely challenging noise environ- ments with very low signal-to-noise ratios, conducted comparative experiments against bascline models on the WSJO-SI 84+NOISEX-99 test set under -10 dB SNR conditions, specifically in Bab- ble and Factory noise environments. The results, presented in Table demonstrate that FIRING-Net consistently outperforms the baseline models, achieving superior performance across all evaluated metrics. This validates its capability to effectively handle diverse and complex noise scenarios, particularly under extreme SNR conditions where traditional approaches often struggle.
C.3 EVALUATION ON REAL-WORLDY SCENARIOS
The performance of FIRING-Net and the selected baseline models was initially evaluated the AVSpeech + AudioSet dataset, a diverse corpus featuring a wide range of speakers and noise types. The results highlighted FIRING-Net's superior performance, suggesting its strong potential for real-
21

---

Published as a conference paper at ICLR 2025
55
Noisy
PHASEN
SN-Net
50
Inter-SubNet
CMGAN
MP-SENet
45
FIRING-Net
40
35
WER
30
25
20
15
10
BUS
CAF
PED
STR
Ave.
Noise Type
Figure 10: ASR performance on FIRING-Net and baseline-enhanced speech samples on CHiME-3 real data using DNN based ASR system.
Table 8: Comparison with selected baseline models WSJO-SI 84+NOISEX-92 under -10 dB
SNR level in "Babble" and "Factory" noise environments. Bold and underline indicate the best and second-best results.
Factory
Babble
Model
STOI
PESQ
SI-SNRi
STOI
PESQ
SI-SNRi
Noisy
0.4452
1.15
0.4338
1.07
PHASEN
0.5731
1.58
11.32
0.4882
1.39
8.58
SN-Net
0.6352
1.63
11.65
0.5011
1.45
9.21
Inter-SubNet
0.6058
1.60
11.53
0.4945
1.46
9.14
CMGAN
0.6627
1.66
11.87
0.5259
1.51
9.86
MP-SENet
0.6584
1.68
12.24
0.5827
1.53
10.21
FIRING-Net
0.7286
1.83
13.36
0.7156
1.77
12.38
world applications. To further validate its effectiveness under practical conditions, conducted periments using the CHiME-3 real-world test set (Barker et al. 2017]. which includes noisy speech recorded in real-life environments such as public transport (BUS), cafeteria (CAF), street junction (STR), and pedestrian area (PED). For evaluation, we employed the official DNN-based ASR model provided by CHiME-3 (Barker al. 2017). comprising seven layers with 2048 units per hidden layer. The input layer incorporated 5 frames of left and right context (i.e., 11 x 400=440 units). The model was trained using a standard pipeline, including pre-training with restricted Boltzmann ma- chines, cross-entropy training, and sequence-diseriminatii training using the state-level minimum Bayes risk (sMBR) criterion (Burget et al.) 2013). Using this ASR system, we measured the Word Error Rate (WER) after speech enhancement, with results presented in Figure IO FIRING-Net sistently achieved the lowest WER across all tested environments, surpassing the baseline models. This demonstrates its capability manage the complexities of real-world noise while preserving speech intelligibility. The outstanding performance can be attributed to the FRI framework, which facilitates effective interaction between target and non-target features, enabling precise noise sup- pression and robust speech enhancement. These findings underscore FIRING-Net's robustness and efficacy handling diverse and dynamic noise conditions in practical applications.
C.4 EVALUATION IM-E AND IM-D
As discussed earlier, the design concepts of IM-E and IM-D are distinct. The design of IM-E focuses on the mutual guidance between target and non-target features, aiming to extract speech information with high similarity to the target features from the non-target features and supplement it into the target features. At the same time, noise information that is highly similar to the non-target features filtered out from the target features, the features in the encoder often contain substantial
22

---

Published as a conference paper at ICLR 2025
Table 9: Ablation study IM-E and IM-D by replacing them with other modules. Bold and under- line indicate the best and second-best results.
LM-E
GM
LM-D
STOI
PESQ
ST-SNRi
Param.
IM-E
IM-E
IM-D
0.9135
3.03
13,96
1.81
IM-E
IM-E
IM-E
0.9103
3.01
13.85
1.72
IM-D
IM-D
IM-D
0.8762
2.96
12.87
1.62
Conv Block
ConvBlock
ConvBlock
0.8226
2.77
11.63
1.89
Conv Block
ConvBlock
IM-D
0.8307
2.76
11.72
2.03
IM-E
IM-E
Conv Block
0.8812
2.98
12.94
1.82
Target Feature (IM-E)
Non- Tor-Target Feature (IM-E)
Noisy
Target Feature (IM-D)
Non-Target Feature (IM-D)
Clean
Target Feature (ConvBlock) Non-Target Feature (ConvBlock)
Figure 1I:
Visualization of feature maps captured from the first LM-E block and the LM-E when IM-E is replaced by IM-D and ConvBlock.
amount of noise. contrast, the design of IM-D is based on the observation that the features in the decoder typically contain little no noise. In this stage, the non-target features usually consist of speech information that has not been involved the reconstruction process. Therefore, the design of IM-D primarily considers how better fuse the speech information from both target and non- target features through mutual guidance, ensuring their effective integration in the subsequent speech reconstruction process. Therefore, validate the effectiveness of IM-E and IM-D by replacing these modules and evaluating the performance of their internal structures when disassembled.
Replacing IM-E and IM-D: For faimess in comparison, replace IM-E and IM-D with config- uration consisting of four convolutional blocks (each block containing a convolutional layer, batch normalization, and PReLU activation function) avoid performance differences arising from vari- ations in the number of trainable parameters. The evaluation results are shown in Table 9 Firstly, we replaced all IM-D modules LM-D with IM-E, which resulted in a slight performance decrease for FIRING-NE -ett along with noticeable increase in model complexity. Next, we replaced all IMs in the network with IM-D and ConvBlock. This led to significant drop in model performance, with ConvBlock performing worse than IM-D. To investigate the cause, we compared the relevant and ir- relevant feature maps of LM-E using IM-E, IM-D, and ConvBlock. as shown Figure 8. The figure reveals that with IM-E, speech and noise information are distinctly aggregated into the target and non-target features, respectively. and noise information within target and non-target features is not clear, and ConvBlock exhibits more noise in both target and non-target features compared to IM-D. This suggests that the ability of LM-E extract speech information into target features and noise information into non-target
contrast, with IM-D and ConvBlock, the separation of speech
23

---

Published as a conference paper at ICLR 2025
Original LM-E
Li Medidcationn
Sproodal
(c) IM-E Modification
(4) IM-E Modification
Figure 12: Network architecture of three modifications of IM-E. (a) We replace the "[Sigmoid(f.) M,l" and "I (igmoid((t, M,]" with "I M,," " and "1- MMM... (b) We remove the non-target feature guide the target feature for noise information extraction. (c) We remove the "[Sigmoid(f,) M These modifications are marked in red.
Table Evaluation on the structure of IM-E, Bold and underline indicate the best and second-best results.
Module
STOI
PESQ
SI-SNRi
Param.(M)
IM-E
0.9135
3.03
13.96
1.81
IM-E Modification-I
0.8884
2.94
13.25
1.81
IM-E
Modification-2 2
0.9008
2.99
13.51
1.59
IM-E Modification-3
0.8987
2.96
13.67
1.81
features closely related the structure of the interaction modules, highlighting the effectiveness of IM-E.
Internal structure disassembling for IM-E: IM-E incorporates three unique structures. First, reverses the features emphasized in M, and Mf example, M., emphasizes speech information in f,. and after reversal, we want to emphasize noise information). Instead of subtracting these masks directly from I, we subtract them from (igmmoi((t.) )) and (Sigmoid(f,) respectively, and then take the absolute value (This modified structure shown in Figure 12(a)) a)). Second, it M,, which is the complement of the one used to extract speech information from non non-target features, to instead extract noise information. This noise information is then employed guide the extraction of noise from the target features, i.e., [Sigmoid(f.) M,, (This modified structure shown Figure 12 b)). Third, during noise filtering of the target feature, another mask M,, is reversed; initially used for filtering noise and extracting speech, it is reversed to instead extract noise, i.e., [Sigmoid (This modified structure is shown in Figure c)).... The results are presented in Table 10 First, observed that the performance of IM-E Modification-1 is significantly lower than that of IM-E. Both structures attempt to reverse the features emphasized by the sigmoid activation function. To
M,
explain this phenomenon, we present the feature spectrograms of Sigmoid((f.) M, and 1- M,, in Figure [5] From this, we can see that the 1 1,M,, operation amplifies all features that are not empha- sized by M., and this amplification primarily boosts regions in the feature spectrum with originally weaker energy, introducing new interfering features into the feature processing. Additionally, we observe performance drop in IM-E after modifications 2 and 3, with IM-E Modification-3 per- forming worse than Modification-2. This is because, both modifications, M, aggregates speech information. However, in Modification-3, M, is generated from fused target and non-target features, allowing noise from the non-target features to interfere with the precision of M. In the original IM-E, the operation (f.gmoid(f.) M, shifts the mask's function from aggregating speech to ag- gregating noise, thus enhancing the non-target features' contribution.
24

---

Param. Published as a conference paper at ICLR 2025
(a) Feature Map
(b) Attention Map
(c) Atlention Map Mn
(d) Attention Map [Sigmoid M,
Figure 13: Visualizations of feature and attention maps of f, M,. Mn and (iigmoid(f.) M.
Dlonaais
02 Some
fise
TIEEE
7.
Original IM-D
(b) IM-D Modification-1
Fieus
fon
Tn
To
(c) IM-D Modification-2
(d) IM-D Modification-3
Figure 14: Network architecture three modifications of IM-D. (a) We directly add the target and non-target features. (b) We process the non-target feature only to add with target feature. (c) We process the target feature only to add with non-target feature.
The structure of IM-D can be roughly summarized as a weighted summation of target and non-target features. To validate the effectiveness of this design, we made three modifications to IM-D, as shown in Figure 10. In the first modification, we directly sum the target and non-target features without applying any weights. In second modification,
Internal structure disassembling for IM-D:
Table 11: Evaluation the structure of IM-D. Bold and underline indicate the best and second-best results.
Module
STOI
PESQ
SI-SNRi
IM-D
0.9135
3.03
13.96
1.81
IM-D Modification-1
0,8762
2.95
13.23
1.23
IM-D Modification-2
0.8987
2.98
13.82
1.81
IM-D 3
0.8934
2.97
13.76
1.81
25

---

Published as a conference paper at ICLR 2025
Target Feature (1" TM Norm)
Non-Target Festure L.M Narm)
Noisy Input
Target Fenture (1 LM w/o Norm:)
Non-Target Feature (1 LM w/o Narm)
Target Fosture (last. LM Norm)
Non Target (Last LM
Clean Target
Target Feature (last 1M w/o Norm
Non-Target Feature (Last LM w/o Norm)
Figure 15: Visualizations of target and non-target features extracted by LM-E with and without normalization operation, along with their corresponding spectrograms of noisy mixtures and clean target speech.
Table 12: Evaluation of the effect of normalization and the choice between addition and subtraction operations for the LM module.
Method
STOI
PESQ
SI-SNRi
LM-E
0.9135
3.03
13.96
-w/o Norm Subtraction
0.8338
2.76
12.25
-W: Norm Addition
0.8820
2.87
13.11
-wo Norm Addition
0.8454
2.78
12.18
LM-D
0.9135
3.03
13.96
w/o Norm Subtraction
0.8557
2.81
12.54
Norm Addition
0.8739
2.85
12.89
w/o Norm Addition
0.8420
2.82
12.73
only the non-target features are weighted, while the third modification, only the target features weighted. To ensure fair comparison, second and third modifications so that the number trainable parameters matches that of the original IM-D. The results, shown in Table [1] indicate that the performance of IM-D Modification-I] where target and non-target features are summed without weighting, is noticeably worse compared IM-D Modification-2 and IM-D Modification-3, where only feature is weighted. Additionally, while the performance difference between IM-D Modification-2 and IM-D Modification-3 is not significant, both modifications still perform significantly worse than the original IM-D.
increased the number of convolutional blocks in the
C.5 ANALYSIS OF RESIDUAL CALCULATION IN LM
the LM. residual computation is employed to extract the filtered non-target information, with the objective of maximizing the inclusion of noise-related features. However, the convolutional opera- tions within the LM can cause a mismatch in the overall energy range between the input and output features. This discrepancy compromises the ability of the residuals effectively isolate noise- dominant features, as illustrated in Figure [15] To address this issue, normalization is applied after each convolutional layer in the L.M, ensuring that the input and output features maintain comparable
26

---

Published as a conference paper at ICLR 2025
Table 13: Evaluation Phase Decoder.
Bold and underline indicate the best and second-best
results.
Phase Decoder
STOI
PESQ
SI-SNRi
LM-D
0.9135
3.03
13.96
DCCM
0,9074
2.98
13.10
FGCM
0,9103
2.99
13.54
Noisy Phase
0,8621
2.89
12.62
Clean Phase
0.9296
3.09
14.83
energy levels. This adjustment effectively mitigates the aforementioned challenge. Additionally, we present the results in Table [2] to evaluate the impact of removing the normalization operation on the subtraction process, as well as the performance effects replacing the subtraction operation with addition, both with and without normalization. The comparative results indicate that when the normalization operation removed, i.e., the features are unconstrained, there is little performance difference between subtraction and addition. However, when normalization is applied, constrain- ing the features, the performance the subtraction operation significantly superior to that of the addition operation.
C.6 EVALUATION PHASE DECODER
Since phase information, unlike magnitude information, lacks clear structural pattern, we con- ducted an evaluation to validate the effectiveness of the proposed FIRING-Net's Phase Decoder. For comparison, we replaced LM-D with DCCM and FGCM, and compared the performance of the phase output from the Phase Decoder with both the noisy input phase and the clean target phase. As shown in Table 13 although the performance of the speech synthesized with the Phase Decoder- generated phase is weaker than that of the clean phase, significantly outperforms both the noisy phase and the phase produced by DCCM and FGCM-based structures. However, it is important note that the mapping relationship between noisy and clean phase is less explicit compared to that of noisy and clean magnitude. Thus, while our proposed FRI framework theoretically does not fully apply to phase recovery, the use of IM-D in the Phase Decoder yields performance gains. Whether these gains align with the theoretical improvements described by the FRI framework remains uncer- tain and will be explored in future work, with a focus on developing models better suited for phase recovery.
27


Reviews:
[{"review_id": "5uNRhwK0qJ", "reviewer": "Reviewer_LH7g", "summary": "Speech enhancement model with local and global information, where target and non-target portions interact with each other.", "questions": "- How do you know that in Eq. (1) convolution is going to separate target and non-target? In the Fig 3 it seems to be the case, but it is not clear to me that why that happens.\n- Why in Table 6 lots of methods have same STOI value? \n- In Eq. (14) some loss hyperparameters are explained, you set those to \"0.2, 0.9, 0.1, and 0.3.\", did you try to optimize these values? Even more importantly if you did experiment with these values, then did you look at the Table 1 and 2 while experimenting?  What I am looking for here is that how britle are these selections for practical use cases. Can someone just use them and assume that they will work out of the box?\n-  Why reverb is used in the training, when testing no such cases are seen? How about removing reverb data from the training. Also, did you use the exactly same training set for all competing models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Idea appears to make sense, experimentation is done with care and results are good.", "weaknesses": "I am not completely sure that noise types used in training are not also seen in testing. What is clear is that SNR levels are same in training andf testing cases. So, it would be good to see how well the model generalizes to unseen conditions, unseen SNR level would be a fairly extreme tests case. \n\nI also noticed that SNR scale is pretty extreme (from -5 to 5 db). How about using more moderate noise levels? Would the differences to competing models then disappear? \n\nAll ideas seem to be gathered from previous publications (like CV papers). It would be good to emphasize the novel technical contributions in the Introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQYo7vHpQi", "reviewer": "Reviewer_hLf3", "summary": "This article presents FIRING-Net, a speech enhancement method based on the Filter-Recycle-Interguide strategy. This approach refines the extraction of speech and noise features through local and global modules, improving speech enhancement performance in complex noise environments. Experiments show that FIRING-Net significantly enhances performance across multiple datasets. The main contributions include a novel feature optimization strategy and experimental validation under various noise conditions.", "questions": "1. Perform qualitative and interpretable analysis of the model method.\n2. Explain how the subsequent experiments and the selection of baseline models for comparison reflect the challenges addressed by the paper’s starting point, which is “when noise and speech components are highly similar, it is difficult for the SE model to learn effective discrimination patterns.”", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a novel \"Filter-Recycle-Interguide\" (FRI) strategy, which is capable of effectively distinguishing between speech and noise features by mutual guidance, thereby achieving better noise reduction and speech restoration.\n\n2. Through the design of local modules (LM) and global modules (GM), the paper effectively extracts target and non-target features by utilizing the energy distribution of convolutional module outputs and self-attention maps, respectively, thereby enhancing noise resistance performance.\n\n3. By conducting extensive experiments on various public datasets and at different signal-to-noise ratio (SNR) levels, the proposed method demonstrates significant improvements in metrics such as PESQ and SI-SNRi compared to some existing methods, validating its effectiveness.", "weaknesses": "1. The paper's starting point is that \"when noise and speech components are highly similar, it is difficult for the SE model to learn effective discrimination patterns.\" However, this aspect of the contribution is not prominently mentioned in the subsequent methodology analysis and experiments.\n\n2. In the field of speech enhancement, there have been some related studies on refining and interacting features that were not referenced in the paper. Therefore, the selection of baseline methods for comparison in the experiments also lacks specificity.\n\n3. There is a lack of qualitative analysis of the method and experiments, which results in a lack of interpretability and fails to distinguish the proposed method from existing research and the baseline methods used in the comparative experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FFQxRqZSil", "reviewer": "Reviewer_jpRr", "summary": "The paper presents a novel approach to speech enhancement (SE) using a method called FIRING-Net, which incorporates a Filter-Recycle-Interguide (FRI) strategy. This strategy divides input features into target (speech) and non-target (noise) components, allowing for mutual refinement between these features.", "questions": "Consistent with the weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel Filter-Recycle-Interguide (FRI) strategy for speech enhancement, which is a creative approach to refining speech and noise components. This method effectively separates and processes target and non-target features, offering a fresh perspective on enhancing speech quality.\n2. The study conducted experiments on WSJ0-SI 84 + NOISEX-92 and AVSpeech + AudioSet. The results show substantial improvements in PESQ and SI-SNRi metrics compared to existing models. The paper also includes a comprehensive ablation study that validates the effectiveness of each component and the overall framework.", "weaknesses": "1. There are similar methods to this approach that also use cascaded strategies to enhance the model's ability to extract target speech, such as TaylorSENet [1]. However, these methods were not compared in the paper.\n\n2. For the experimental section, the model trained on the DNS Challenge training set should naturally be tested on the public DNS Challenge test set. However, the authors chose two other datasets, which is quite odd. I believe the authors should test the model on the DNS Challenge test set to better demonstrate the model's performance.\n\n3. In the appendix, the experimental results of VoiceBank-DEMAND show a significant discrepancy in the results of MP-SENet compared to the original paper. The authors mentioned that these results were not in the original paper, but they are detailed in the official repository: https://github.com/yxlu-0102/MP-SENet/blob/main/Figures/table.png. For detailed results, see: https://github.com/yxlu-0102/MP-SENet.\n\n4. In the ablation study section, Table 3 is not referenced for the LM ablation study and should be added.\n\n5. In the appendix, please specify whether the PESQ is narrowband or wideband, as these will yield different results.\n\n6. The visualization results should include experiments with better-performing methods, such as MP-SENet, to see if similar issues persist.\n\n7. For the experiments with IM-E Modification 2 and IM-D Modification 2, would incorporating both modules simultaneously yield better results? According to Tables 8 and 9, both modules perform second best but with fewer parameters, which is advantageous for speech enhancement. Is it possible to include both modules to reduce the parameter count and improve model performance?\n\n8. Some dimension symbols in the paper, such as $L$, are not explained.\n\n9. Word misuse: convolution operation -> convolutional operation\n\n[1] Li A, You S, Yu G, et al. Taylor, can you hear me now? a Taylor-unfolding framework for monaural speech enhancement[J]. arXiv preprint arXiv:2205.00206, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cpajoMTCk", "reviewer": "Reviewer_S2co", "summary": "This paper presents a new deep network architecture for speech enhancement (SE) called FIRING-Net (FIlter-Recycle-INterGuide NETwork). To deal with the more challenging scenarios where the interference/noise characteristics are highly similar to the target speech signal, e.g., in the case of babble noise, the authors propose to recycle filtered-out features for extracting more discriminative patterns between speech and noise through refinement. This is achieved by re-defining the filtered-out features as non-target features, which are used to guide the learning of target features in an interactive manner. More specifically, the FIRING-Net features two key components: Local Module (LM) and Global Module (GM), to perform the recycle-and-refine idea in a local and global manner. The LMs utilizes Interaction Modules (IMs) that have different designs for the encoder and decoder sides. The GM exploits energy distributions of the attention maps to guide extration of the speech and noise features. Experiments on several datasets demonstrate the effectiveness of FIRING-Net over various signal-to-noise ration (SNR) settings and across several noise types, including the challenging case of babble noise.", "questions": "- Could the authors explicitly associate the \"Network for speech extraction\" in Figure 1 to the corresponding module(s) or layer(s) in Figure 2? This will help understanding where the recycling scheme actually takes place. Similarly, association of the \"Network for noise extraction\" in Figure 1 to the place(s) in Figure 2 will be helpful.\n\n- In Figure 2, the LM-E (or LM-D) module has addition and subtraction operators. However, the subtraction operation may not lead to a different result from using addition if the feature value range is not constrained. For example, if the values of $f_{t,t}$ span all real-numbers, then it does not matter if you get $f_{t,n}$ by subtracting $f_{t,t}$ from $f_{t}$ as eq. (3), or by adding them as $f_{t,n}=f_{t}+f_{t,t}$ -- the Dense Block that outputs $f_{t,t}$ should be able to adjust its parameters to account for the opposite sign. Therefore, it is not clear if using the subtraction instead of addition is necessary.\n\n- For the target and non-target feature plots in Figure 3, are they $f_t$ and $f_n$ extracted by the Conv 2D layer as eq. (1), or the refined features $\\hat{f}_t$ and $\\hat{f}_n$  before the IM-E (or IM-D)? Also, is it possible to compare $f_t$ to $\\hat{f}_t$ and $f_n$ to $\\hat{f}_n$? Knowing the difference will be helpful for understanding the contribution of the feature refinement step.\n\n- In Figure 4 (a), the mask $M_t$ may be directly apply to extract the target features by multiplying it with the feature $\\hat{f}_t$. So what is the role of the sigmoid here? (In other words, why should $|Sigmoid(\\hat{f}_t)-M_t|\\odot\\hat{f}_t$ be considered instead of $M_t\\odot\\hat{f}_t$)? \n\n- From Figure 9, it looks like that IM-E Modification-3 (d) is just taking off one Sigmoid operation from the Original IM-E (a). Then, why does the model size become smaller (1.64M) for (d) than the original size (1.74M) for (a) as presented in Table 8? \n\n- On page 7, the paragraph below eq. (6): \"... obtained by $f_n=f_4+f_{2,n}+f_{n,t}$\" should be \"$f_n=f_4+f_{2,n}+f_{3,n}$\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed system (FIRING-Net) based on the idea of recycling intermediate speech and noise features to further improve the separation of target and non-target components seems novel.\n\n- Good performance of the FIRING-Net across various SNR and noise types (E.g., Tables 1, 2).\n\n- Relatively small model size with low latency (Eg., Tables 3, 4), which is important for many SE applications.\n\n- Extensive ablation studies showing the effect and contribution of each key component, i.e. LM, GM and IMs, on the overall SE performance.", "weaknesses": "- The main weakness is there seems lacking a clear explanation as why the proposed architecture, especially the Local Modules (LMs), can promote target speech and non-target noise features extraction in the way as claimed in the paper. To be more specific, for the proposed LM-E (and LM-D) in Figure 2, what is the actual mechanism or reason that leads to the model assigning speech to the $f_t$ branch and noise to the $f_n$ branch? Please provide more detailed explanations on how such network design promotes speech and noise separation  to carry out the proposed feature recycling scheme.\n\n- Another weakness of the paper is that the proposed SE model is tested on synthesized noisy speech data only. To better demonstrate the superiority of the proposed model, it will be nice if testing on real-world recorded noisy speech samples can be performed (e.g., using the realistic noisy samples from the CHiME-3 dataset: *J. Barker et al., “The third ‘CHiME’ speech separation and recognition challenge: Analysis and outcomes,” Comput. Speech Lang, 2017*). In this case, since there will be no ground-truth references to compute the objective metric scores used in the paper (i.e., PESQ, STOI, and SI-SNRi), the comparison with baselines may be done by using subjective listening testing or using automatic speech recognition (ASR) as evaluators.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZBCL2tLrF", "reviewer": "Reviewer_9ACG", "summary": "This paper, based on the challenges in the field of speech enhancement when noise and target speech are similar, designs a Filter-Recycle-InterGuide Network (FRIGNet). It utilizes target features and non-target features for mutual refinement, and based on the mutual refinement of positive and negative features, it has designed LM and GM modules. Additionally, it employs an attention module based on energy segmentation to complement the network.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a novel approach. While others focus only on how to better eliminate non-target features, the authors consider both target features and non-target features to refine each characteristic. Based on this, the overall network architecture and individual modules are designed based on the logic of mutual refinement between target features and non-target features. An attention mechanism based on energy levels is also introduced. The model performs well in both comparative experiments and ablation studies.", "weaknesses": "The paper extensively discusses how the model uses operations such as generating masks, convolutions, and additions/subtractions to obtain the so-called target features and non-target features mentioned in the text. However, it lacks a theoretical explanation for why these operations result in target features or non-target features. Additionally, it is unclear whether the attention module based on energy segmentation is an original innovation or borrowed from other papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)