TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation
Authors: Mohan Xu
Kai Li
Guo Chen
Xiaolin Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Speech separation
lightweight model
time-frequency domain
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In recent years, much speech separation research has focused primarily on improving model performance. However, for low-latency speech processing systems, high efficiency is equally important. Therefore, we propose a speech separation model with significantly reduced parameters and computational costs: Time-frequency Interleaved Gain Extraction and Reconstruction network (TIGER). TIGER leverages prior knowledge to divide frequency bands and compresses frequency information. We employ a multi-scale selective attention module to extract contextual features, while introducing a full-frequency-frame attention module to capture both temporal and frequency contextual information. Additionally, to more realistically evaluate the performance of speech separation models in complex acoustic environments, we introduce a dataset called EchoSet. This dataset includes noise and more realistic reverberation (e.g., considering object occlusions and material properties), with speech from two speakers overlapping at random proportions. Experimental results showed that models trained on EchoSet had better generalization ability than those trained on other datasets to the data collected in the physical world, which validated the practical value of the EchoSet. On EchoSet and real-world data, TIGER significantly reduces the number of parameters by 94.3% and the MACs by 95.3% while achieving performance surpassing state-of-the-art (SOTA) model TF-GridNet.

Published as a conference paper at ICLR 2025
TIGER: TIME-FREQUENCY INTERLEAVED GAIN Ex- TRACTION AND RECONSTRUCTION FOR EFFICIENT SPEECH SEPARATION
Mohan Xu-.*, Kai Lit.2.;, Guo Chen Xiaolin Hu1.2.31
1. Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China
2. Tsinghua Laboratory of Brain and Intelligence (THBI,, IDG/McGovern Institute for Brain Research, Tsinghua University. Beijing 100084, China
Chinese Institute for Brain Research (CIBR), Beijing 100010, China xu xu-mhl 98tsinghua org. cn
{1i-k24, cg22 @mails. ts singhua edu.
xihu@tsinghua edu.en cn
ABSTRACT
In recent years, much speech separation research has focused primarily on im- proving model performance. However, for low-latency speech processing sys- tems, high efficiency is equally important. Therefore, we propose a speech separa- tion model with significantly reduced parameters and computational costs: Time- frequency Interleaved Gain Extraction and Reconstruction network (TIGER). TIGER leverages prior knowledge divide frequency bands and compresses fre- quency information. We employ a multi-scale selective attention module to extract contextual features, while introducing full-frequency frame attention module to capture both temporal and frequency contextual information. Additionally, to more realistically evaluate the performance speech separation models in com- plex acoustic environments, we introduce a dataset called EchoSet. This dataset includes noise and more realistic reverberation (c.g., considering object occlusions and material properties), with speech from two speakers overlapping at random proportions. Experimental results showed that models trained on EchoSet had bet- ter generalization ability than those trained on other datasets to the data collected in the physical world, which validated the practical value the EchoSet. On EchoSet and real-world data, TIGER significantly reduces the number of param- eters by 94.3% and the MACs by 95.3% while achieving performance surpassing Late-of-theeard (SOTA) model TF-GridNet.
INTRODUCTION
Humans possess the ability to focus specific speech signal in noisy environments, which is phenomenon known as the "cocktail party effect" (Cherry) 1953). In speech processing, the cor- responding challenge is accurately separating different sound sources from mixed audio signals, task referred to as speech separation. Speech separation is typically used as preprocessing step it helps enhance recognition accuracy (Haykin & Chen 2005). Conse- quently, it is crucial to ensure that speech separation not only produces clear and distinct outputs on real-world audio but also meets the demand of high computational efficiency (Divenyi 2004).
for speech recognition,
In recent years, the application of deep learning methods to the speech separation task has received widespread attention (Wang al., 2023] Li et al. 2023 2022 Li Luo 2023 Subakan et al., 2021]. Although many bigh-performing speech separation methods have been proposed, two key issues remain insufficiently addressed.
First, when designing a separation model, we should fully take into account the actual applica- tion scenarios of the speech processing system, which require low latency and low computational
Mohan Xu and Kai Li contribute equally the article. Corresponding author.

---

Published as a conference paper at ICLR 2025
complexity. However, many approaches have primarily focused on improving speech separation performance. For example, TF-GridNet (Wang et al. 2023) utilizes bidirectional LSTMs and self- attention mechanisms in alternating manner, achieving good results on benchmark datasets but have large model size. To make the separation model more applicable in computationally con- strained real-world scenarios, TDANet (Li et al.] 2023) introduces an efficient lightweight architec- ture using top-down attention, achieving competitive performance with lower computational costs than SepFormer (Subakan et al. 2021). However, as a time-domain method, TDANet struggles to leverage frequency domain information. On the other hand, time-frequency domain approaches like TF-GridNet (Wang et al., (023) model both time and frequency dimensions but require higher computational resources. BSRNN (Luo & Yu 2023). which is the SOTA model for music separa- tion, reduces the computational burden by focusing on important frequency bands. The band-split strategy is enlightening but under-explored in speech separation. How to balance computational efficiency and separation quality in speech separation is still a big challenge.
Second, the commonly used speech separation datasets exhibit significant gap from real-world sce- narios. Many methods relied on the WSJO-2mix dataset (Hershey et al.] 2016) for evaluation, which only contains tully-overiapping audio without noise reverberation. Models trained this kind of dataset are subject to weak generalization and robustness in real-world environments (Kadroglu al. 2020; Cosentino et al. 2020). Although the WHAMR! dataset (Maciejewski et al.. (2020) adds noise and reverberation to WSJO-2mmix the generated reverberation fails to fully take into account factors such as object occlusion and material properties, and the diversity of acoustic scenarios re- mains limited. To more accurately train and evaluate speech separation models for practical use, a dataset that more closely resembles real-world environments is necessary. Specifically, this dataset should include different noise types, cover a wide range of realistic acoustic environments, and have randomly distributed speech overlap ratios.
To address the two issues mentioned above, our main contributions are as follows:
1. We propose a novel lightweight separation model named TIGER. TIGER adopts a band-split strategy to reduce computational costs by leveraging prior knowledge in the frequency do- main. Furthermore, TIGER introduces the frequency-frame interleaved (FFI) block, composed of two key submodules: multi-scale selective attention (MSA) and full-frequency-fran tion (F))). These submodules enable efficient integration of temporal and frequency features.
atten-
2. We propose a speech separation dataset called EchoSet. It is high- high-fidelity dataset bridging the gap between model training and wollw applications.
Experiments show that models trained on EchoSet generalized better on real-world data than those trained on benchmark dataset LRS2-2Mix (Li et al. (023) and Libri2Mix (Cosentino et al. 2020), validating that audio in EchoSet is closer the physical world. We then comprehensively eval- uated TIGER on Libri2Mix. LRS2-2Min Mix and EchoSet. As the dataset becomes more complex, TIGER's 's superiority in performance becomes more significant. On EchoSet, which the most com- plicated among the three datasets, TIGER improved the performance by about 5% compared with TF-GridNet, while reducing the parameters and MAC's by 94.3% and 95.3% respectively. When tested real-world data, TIGER also achieved the best separation performance. The remarkable result shows that TIGER provides a new solution for the design of lightweight speech separation models for practical use in the time-frequency domain.
RELATED WORK
Speech separation. Speech separation methods can be divided into time domain and frequency domain. Time domain methods directly process the original audio signal. -ony-TasNet (Luo & Mes- garani 2019) extracts features by temporal convolutional network eLea al. 2016). To improve the performance on long sequence data, DPRNN (Luo et al.) (020) divides the temporal sequence into small blocks and alternately performs intra-block and inter-block modeling, which becomes a com- mon paradigm for many following works (Wang et al.) 2023: Subakan et al.) 22021). Time-frequency domain methods apply a Short-Time Time Fourier Transform (STFT) to transform the waveform into a joint representation of time and frequency. TF-GridNe (Wang et al. 2023) enhances the temporal context information by a full- band self-attention module. Although TF-GridNet achieved SOTA performance, it entails huge computational costs.
2

---

Published as a conference paper at ICLR 2025
Lightweight models. Some models (Wang et al. 2023 Yang al. 2022 Subakan al.) 2021 with high computational complexity are difficult to be applied to real time speech processing on edge devices. To reduce computational costs, TDANet Li et al. 2023) draws on the attention mechanism of human brains and designs a lightweight structure. In music separation, BSRNN (Luo & Yu 2023) uses prior knowledge split band, performing band merging less important bands to compress the feature while retaining key band information.
Datasets for speech separation. WSJO-2mix (Hershey al. 2016 an early and commonly used fully-overlapping clean speech separation dataset. WHAM! (Wichern et al. 2019) added environ-
mental noises WSJO-2mix, and furthermore WHAMR! (Maciejewski et al. 20200 added simple reverberation. Libri2Mix (Cosentino et al.] 2020) was proposed based on the observation (Kadoglu et al. 2020) that the test performance of Conv-Tasee TasNet trained on WSJO-2mix dropped sharply on other separation datasets. The utterances in Libri2Mix were mixed with sparse overlap, and noises were added to the mixed audio, reverberation was not considered in Libri2Mix. LRS2-2Mix (Li al. (023) was mixed by video clips acquired through BBC. The audio was recorded in real acous- lic scenarios, thus containing much noise and reverberation. However, due to the different recording environments of the clips, such as the shapes and materials of the room and objects, the reverberation obtained when the clips were directly mixed is still unrealistic.
TIGER
Ii.
Decoder
n
Frooder
Bantsppk
Separster
P:
Decudar
P
Figure The overall pipeline of TIGER. We focus on scenarios with only two speakers.
3.1 OVERALL PIPELINE
Let L be the sequence length of audio. Given monaural mixture audio S E RIx. containing utterances of C' speakers and noise n E Rlx,,
S= P.P
the speech separation task is to recover the clean speech of each speaker P. RIXL
The TIGER system (Figure I can be divided into five main components: the encoder, the band- split module, the separator, the band-restoration module, and the decoder. Specifically, we first use STFT as the encoder to convert the mixed audio signal S E I into its time-frequency representation X CFT where F and T' represent the number frequency bins and time frames, respectively. Next, we apply frequency band-split strategy, dividing the whole band into K sub- bands of varying widths based on their importance. Fach sub-band is transformed into uniform channel size N using ID convolutions, and these are then stacked along the frequency dimension to produce the feature representation Z R x K'X'' Thirdly, serves as the input to the separator, which uses FFI blocks with shared parameters to model the acoustic characteristics of each speaker. Subsequently, the rent-restoration module restores the bub-bands to the full frequency range using separator output J p. IR XxX (B denotes number of blocks in separator), and the mask for each speaker M; E CFXT XT is applied element-wise product to X, producing the separated representation for each speaker H; C xT' Finally, the inverse STFT is used to generate the clean speech signal P. E R. XL for each speaker.
3.2 BAND-SPLIT MODULE
Given a time-frequency representation X, first apply a frequency band-split strategy to divide the frequency dimension into K frequency sub-bands (B₃ E CG2xI xI lk= 1,K]
K
G
(2)
A=
3

---

Published as a conference paper at ICLR 2025
The widths of the sub-bands G₁ not necessarily the same. For each frequency sub-band B merge its real Re(.) and imaginary Im(-) parts into the frequency dimension generate B₁ R2G We denote the concatenation operation as ||. then:
B Re (x) [lm(Bk)-
Next, we transform the frequency dimension 2G of Bg to the feature dimension N using a group normalization layer followed by ID convolution, which utilizes kernel size of and does not share parameters across different B In this way, we obtain feature Z RNxx of the same shape for each sub-band. We then stack the features Zy from the K frequency sub-bands along the frequency dimension yield the input feature Z R Rxxxx for the separator.
3.3 SEPARATOR
FFI block
FFI block
FFI block
(a) Pipaline of separator
Leyemorm
Layemomm
module
F'moddle
% e stoe
Zu + R.
MSA module
MSA module
Frequency path
Frequency- frame interleaved block
Figure 2: The separator of TIGER, consists of several FFI blocks which share parameters. Residual connections are used to retain original features and reduce learning difficulty.
In the separator, the input feature Z passes sequentially through B frequency-framm interlcaved (FFI) blocks with shared parameters, as shown Figure In each FFI block, the frequency path is first used to extract contextual information between different sub-bands, producing E., E N xxx Next, we feed Z, into the frame path further model the contextual information between different time frames, generating Zi,e E R N XXT
The structures are identical in both the frequency path and the frame path, modeling along the fre- quency dimension and the time dimension respectively. Each path consists of two main modules: the multi-scale selective attention (MSA) module and the cill-frequency-frane As illustrated in Figure 3 taking the frequency path as an example, we first apply the MSA module along the frequency dimension K to selectively extract features from Z,, which results in enhanced frequency features Z, E N RX Then, the F module is used to integrate information across different sub-bands of Zi, followed layer normalization, to produce the output feature of the frequency path Zb.f-
attention (F3 A) module.
3.3.1 MULTI-SCALE SELECTIVE ATTENTION MODULE
The MSA module enhances important features through a selective attention mechanism and is vided into three stages: encoding, fusing, and decoding, shown in Figure Ba). Taking the MSA module in the frequency path as an example, the input to the module ZB.
The encoding stage. This stage aims to capture multi-scale acoustic features. Specifically, first use multiple ID convolutional layers (with a stride and channel of H) to progressively K downsample the frequency dimension zr, resulting in of multi-scale acoustic features
4

---

Published as a conference paper at ICLR 2025
module
2, perute
L
R #
SA maddde
MAnod.r
B:
K
module
Conv2d
Q e (:
3, gooke
Canvłd
K R.odks(BBa)
Balt iftention
Conv2d
Ea & x NaRNT
Conv2d
VeR AARAaia
(b) module
Figure 3:
The structure of the MSA module and the A module. The structures of frequency and frame paths are the same.
{Es E R & xT = [0, D]} where d denotes the d-th layer of downsampling. Next, we ap- ply average pooling layers, denoted as X(-). downsample all Ea to the same frequency resolution 2b. Subsequently, the features with different frequency resolutions are fused into global features # xI through addition. Finally, multi-layer convolutional (MLC)
K
G EL X(Ea), G E IR H
network is used to transform G into G' E Hx H
The fusing stage. In this stage, we fuse the local Eg and global G' information using the selective attention (SA) module. Specifically, for the d-th layer, we first use two ID convolutions to map G' into # $ xT and p I. $o xT' respectively. Then, we also use one ID convolution to map Ed into ø E #x xT To match the resolution of ő, and p are upsampled through inter- polation pu-). and selective attention weights are generated using sigmoid function (-).. Finally, the attention weights are multiplied element- wiie with Φ, and p(p) is added to the result to obtain Hx K5 xT The above process can be expressed as follows:
La E R
La f(u(T), p(p)).
(4)
The function of f is defined as follows:
(5)
where & and z represent global features, represents local features, and @ denotes element-wise multiplication. This function describes the mathematical process of SA mechanism. We first apply sigmoid function to v, generating value between and 1. Then, the value is used to extract effective features from local information by calculating element-wise product of a (x) and y. Finally, we add the product to 2, fusing global information and filtered local information. In this way, [0, D]} contains both local and global information, which helps the model better extract the acoustic features in the audio mixture.
(La E R zd xT'
The decoding stage. the d-th layer, where E [0, D 1], the input consists of the decoding result from the previous layer dl + (denoted as Da+1 E IR Hx प xT) and the output La from the fusing stage at the d-th layer. D 2411 is processed through the SA module to produce Da. Specifically, Da+1 is transformed using two ID convolutions to obtain R
H ž xT" and 3 R a xT while
La is transformed through a 1D convolution to produce E Hx
We then compute:
D f(((a), 7: (3))

---

Published as a conference paper at ICLR 2025
where f is defined in equation This formulation integrates the decoding result with the output from the fusing stage to generate the next layer of decoded features. In particular, for the layer where d' =
D. DD LD E R x K. x2' 2
For the layer where d = O, hidden dimension H Du E " xT to the feature dimension N, obtaining Z, E RN*x** KxT as the output of the MSA module. In the MSA module of the frequency path, the frequency dimension K is considered the processing dimension. In the frame path, the time dimension 7 is considered the processing dimension.
ID convolution to restore the
3.3.2 FULL- FRAME ATTENTION MODULE
In the frequency path, the FA module is used to aggregate features across different sub-bands, as shown in Figure ((.). Given the input Z, and the number of attention heads A, we first use separate 1 X 1 2D convolutional layers with distinct parameters to transform Zù into query Q R(AXE)*KXXT key K E [R(AE)xXXXT and value V E [Ax xxxXT
To obtain the information of full time length on each sub-band and apply self-attention mechanism, frame dimension I' and the channel dimension E' are merged in order time step, so we get query Qi (XX(E)T) and key K; E (FXTT) for the i-th attention head. Similarly, Vi xT)) K; is transposed and then multiplied with Q: calculate the attention map of size K x K, which indicates the similarity between each sub-band and acts as the weight information of the frequency context. Then the attention map is multiplied with V, to obtain the output matrix. K a xT) is calculated as follows:
get value
For the i-th attention head, the output O₂ E IR
Q.Kc VEXT
O Softmax
V
The output matrix of each attention head is concatenated to get E (X and the full-time length is split into T time steps and transformed by 2D convolutional layer, generating the output Zo.5 E R XxT, The process of the F$ A module in the frame path is similar.
3.4 BAND RESTORATION MODULE
After going through the separator, the sub-bands need to be converted back to their original width during mask estimation. Specifically, Ja E N XxT denotes the output of the separator. For the k-th sub-band feature JB.K R NxT (k E [J, the PReLU activation function and 1D convolutions are used to transform the number of channels to twice the original dimension 2G,, corresponding to the real and the imaginary part. The complex feature is restored to generate mask for each sub-band Ms E CG2xT using the ReLU activation function. Then they are merged on the frequency dimension get the mask for the whole band M CFXT, Similar to band-split, the ID convolutions of different sub-bands do not share parameters.
4 ECHOSET
To develop models that perform better in daily scenarios, we need dataset close to the real world. We create EchoSet, a speech separation dataset with various noise and realistic reverberation, based on SoundSpaces 2.0 (Chen et al. 2022) and Matterport3D (Chang et al. 2017). An analysis of the dataset is shown in T
SoundSpaces 2.0 is an audio rendering platform in 3D environments. Given the mesh of a 3D sce- nario, it can simulate the acoustic effects of any sound captured from microphones. We followed the steps below to generate mixed speech. (1) Choose the scenario. We selected rooms where daily conversations often occur (such as office, living room, bedroom, dining room, etc.) from Matter- port3D, a large RGB-D dataset containing 90 diverse multi-floor and multi-room indoor scenes. (2) Define or sample the position. We defined microphone at a suitable position, like next to table or sofa, and sampled two sound sources in the same room. (3) Sample the direction and distance. The angle between the microphone and the sound source must be obtuse, meaning that the speaker and listener face each other. The distance between the microphone and each speaker sampled between 1 m and m. (4) Sample the height. The microphone and sound sources were randomly generated vertical height of 1.5 m 1.9 from the floor, which is about a person's
randomly

---

Published as a conference paper at ICLR 2025
height. (5) Generate the audio. With SoundSpaces 2.0, mixed audio files generated based on bidirectional path tracking algorithm (Cao et al. 2016), which can simulate various effects in the sound propagation process, including reverberation, diffraction, and absorption. Materials of the room wall and the objects were annotated by Matterport3D and considered during the generation of the audio mixture.
Based on the SoundSpaces 2.0 platform and the Matterport 3D scene dataset, we can simulate rever- berant audio from different speakers in LibriSpeech (Panayotov al. 2015) to build a new dataset, EchoSet. In total, EchoSet includes 20,268 training utterances, 4,604 validation utterances, and 2,650 test utterances. Each utterance lasts for 6 seconds. We mixed the speech of the two speakers at random overlap ratio and added some noises from WHAM! noise (Wichern et al. (2019). The two different speakers were mixed with signal-to-distortion ratio (SDR) sampled between -5 dB and 5 dB. The noises were mixed with SDR sampled between 10 dB and 10 dB. The dataset is available at: https::/ httsss///hggggggaceeeeo wtteaaaattet/teeeeeeeeeeeeeeeeee EEchoSSSeeLeee
Dataset
Noise
Reverb
Overlapping
WSJO-2mix (Hershey et al.
2016
X
Full
WHAMM! (Wichern al. 2019)
x
Full
WHAMR! (Maciejewski el al. 2020
Only room
Full
Libri2Mix (Cosentino
2020
X
Sparse but fixed Full
2Mx (Li et LRS2-2MiX
2023)
Room and objects (different scenes)
EchoSet (ours)
Room and objects (same scene)
Random
Table 1: Features of datasets for speech separation.
5 EXPERIMENTAL SETUP
Dataset. We report the performance of TIGER on EchoSet. For fair comparison with previous speech separation methods (Li al., 2023 Wang et al.) 2023 Hu et al.) (202T). we also used two benchmark datasets LRS2-2Mix (Li et al. 2023) and Libri2Mix train-1 100 min (Cosentino et al.)
2020 All of these datasets are at a sampling rate of 16 kHz.
To validate the gap between EchoSet and real-world environments, we constructed real-world data by selecting 10 real-world environments and recording audio from 40 speakers from the LibriSpeech (Panayotov et al., 2015) test set. The two audio used for mixing were recorded in the same acoustic scene (c.g.. the shape and material of the walls and objects in the room) and followed the same mixing method L.S22-2Mxx (Li et al. 2023). The duration of each audio is 60 seconds, and the sampling rate is 16 kHz. For more details of these datasets, please refer to Appendix A
Training and evaluation. During training, we utilized 3-second audio segments for EchoSet and Libri2Mix, and 2-second segments for LRS2-2Mix. The negative SI-SDR was adopted as the train- ing loss (Le Roux et al. 2019). Adam optimizer (Kingma & Ba 2014) was employed with initial learning rate of 0.001, adjusted based on validation performance. Evaluation metrics included SDRi and SI-SDRi (Vincent et al. 2006), with higher values indicating better performance. We report parameters and MAC operations for complexity, which are calculated for one second of audio at 16 kHz. Inference speed was measured on NVIDIA RTX 4090 and Intel Xeon Gold 6326. De- tailed training and evaluation configurations can be found in Appendix and Appendix C Code is available at: https: / /ii thub wtt.huub...oo/bappeeeeeeeeee...e
RESULTS AND DISCUSSION
6.1 ECHOSET IS MORE CLOSE TO THE REAL-WORLD DATA
We trained different models on Libri2Mix. LRS2-2Mrn and EchoSet, and then tested them on the data collected in the real world. The results presented in Figure 4 Compared to models trained on Libri2Mix and LRS2-2Mix. the models trained on EchoSet produced higher-quality separated speech, confirming that the gap between EchoSet and real-world audio is relatively small.
7

---

Published as a conference paper at ICLR 2025
Liri2Mix LR52 2M6
4.00
EchoGet
3.2
51
3.46
1,5
411
3.58
CusiPendtee Sumbdoe-rL..e
AFRONU-3E
Larget
TICER TICER iargat
Figure 4: SI-SDRi results of different models on the real-world data. Models were trained on Libri2Mix, LRS2-2Mix and EchoSet respectively.
Libri2Mix SDRi
LRS2-2MP 2Mix
EchoSet
Methods
SI-SDRi
SDRi
SI-SDRi
SDRi
SI-SDRi
Conv-TasNet (Luo & Mesgarani 2019
12.50
12.10
11.0
10.6
7.69
6.89
DualPathRNN Luo el al. (2200)
11.64
11.26
13.0
12.7
5.87
5.06
SudoRM-RFL, 1.00 Tzıms al.
2020
13.58
13.16
11.4
11.0
7.70
6.84
A-FRCNN-I
2021
16.73
16.32
13.3
13.0
9.64
8.76
TDANet Large Li ct al
2023
16.11
15.64
14.5
14.2
10.14
9.21
BSRNN (uo & Yu 2023)
17.38
16.96
14.4
14.1
12.75
12.23
TF-GridNet Wang
2023
19.56
19.24
15.7
15.4
13.73
12.85
TIGER (small)
17.09
16.67
14.2
13.9
13.15
12.58
TIGER (large)
18.34
17.97
15.3
15.1
14.22
13.73
Table 2:
Performance comparison of TIGER and other separation models on Libri2Mix, LRS2- 2Mix, and EchoSet. Models are trained and tested corresponding datasets. Bold denotes the best performance, and underline indicates the second-best best. SDRi and SI-SDRi are recorded in dB.
6.2 COMPARISONS WITH STATE-OF-THE: METHODS
compared TIGER with previous SOTA models including Conv-TasNe (Luo Mesgarani, 2019), DualPathRNN (Luo et al. 2020). ModdRMM-FF..0 (Tzinis et al., 2020), A-FRCNN-16 (Hu
ct al. 2021 TDANet Large (Li et al. 2023), BSRNN (Luo & Yu 2023) and TF-GridNet (Wang al. 2023 in terms of performance and efficiency. TIGER (small) and TIGER (large) denote the models with the number of FFI blocks B and B = 8, respectively.
Separation performance. TIGER obtained competitive separation performance the three datasets compared with previous SOTA models (see Table 2}. On Libri2Mix, which is relatively simple for lack of noise and reverberation, TIGER (large) was second only to the current SOTA model TF-GridNet, with a 6% drop in performance. On IRS2-2Mix, a more complicated dataset with reverberation recorded in different scenes, the drop in performance of TIGER (large) only 2% compared with TF-GridNet. On EchoSet, the only dataset with the most realistic reverberation among the three, TIGER (large) achieved an SDRi of 14.22 dB, surpassing other existing methods. On this dataset, TIGER (small) also achieved the performance that was only slightly lower than TF-GridNet. From the above experimental results, we can see that the more complex the acoustic scenarios are, the better performance TIGER will produce. Similarly, based the results in Fig- ure 4 we observed that TIGER also outperforms existing models in real-world test scenarios. This demonstrates that TIGER is applicable to complex real-world acoustic scenarios including diverse noise and reverberation. To visualize the separation result, we present the spectrogram differences between the audio separated by TIGER and TF-GridNet (Appendix []) demonstrating that TIGER is capable effectively reconstructing both low-frequency and high-frequency features.
TIGER also demonstrates advanced performance cinematic sound separation, which aims to extract different audio elements from a film's soundtrack. See Appendix for details. As for demos of speech and cinematic sound separation, please refer the project page
httpa://eslt ka1 .on/T13RR
8

---

Published as a conference paper at ICLR 2025
Paras
MACs (G/s)
Training
Inference GPU Time
Methods
(M)
GPU Time
GPU Memory
CPU Time
GPU Memory
Cot
2019
5.62
7.19
92.96
1436.94
64.21
13.17
28.78
DualPathRNN 2020
2020
2.72
45.01
67.23
1813.55
723.13
30.38
298.09
SudoRM-RFI 2020
2.72
4.65
118.46
1353.43
104.32
20.66
24,42
A-PRCNN-16
2021
6.13
81.28
230,53
2925
478.58
82.65
163.82
TDANet Large 2023
2.33
9.19
263.43
4260.36
434.44
74.27
136.96
BSRNN 2023
25.97
98.70
258.55
1093.11
897.27
78.27
130.24
TF-GriNey 2023
14.43
323.75
284.17
5432.94
2019.60
94.30
491.73
TIGER (small)
0.82
7.65
160.17
2049.46
351.15
42.38
122.23
TIGER (large)
0.82
15.27
229.23
3989.59
765,47
74.51
122.23
Table 3: Efficiency comparisons of TIGER and other models. recorded in ms, and GPU Memory is recorded in MB.
GPU Time and CPU Time
Separation effciency. The parameters TIGER were only 0.82 M, and the MACs were only 7.65 G/s and 15.27 G/s for the small and large versions respectively. Compared with TF-GridNet, the parameters of TIGER (large) dropped by 94.3%, and the MACs were reduced by 95.3%. For in- ference of one-second audio, TIGER (large) took about 1/3 of the CPU Time and 3/4 of the GPU Time compared with TF-GridNet, demonstrating a significant calculation compression effect. Be- sides, TIGER took up less memory during training and inference, making TIGER more suitable for deployment on devices with limited computational resources.
6.3 AB ABLATION STUDY
We adopted the small version of TIGER (B = 4) in the ablation studies. All the models were trained and tested on EchoSet. The training configuration of TIGER and other models the same.
SDRi
SI-SDRi
Paras (M)
MACs (G/s)
GPU Time
Schemes
Sub-band Number
(dB)
(dB)
(ms)
NonSplit
321
11.53
11.12
0.56
40.89
72.91
NormalSplit
47
12.94
12.37
0.82
5,29
40.78
EvenSplit
67
12.80
12.24
0.82
7.65
43.06
LowFFregNaroowSliic (ours)
13.15
12.58
0.82
7.65
42.38
Table 4: Comparison of performance and efficiency of models with different band-split schemes.
MSA module
F module
SDRi (dB)
SI-SDRi (dB)
Params (M)
MACs (G/s)
7.58
7.22
0.33
2.74
12.34
11.87
0.75
4.95
13.15
12.58
0.82
7.65
Table 5: Importance MSA and F modules on the EchoSet test set.
Band-split schemes. To verify the effectiveness of the band-split method on the speech separation task, we designed several experiments of different band-split schemes (see Table [10 Appendix). For these experiments, we kept the feature channel N the same. For the model NonSplit that did not adopt band-split, each frequency point was treated a sub-band and the real and imaginary channels were transformed to the feature dimension N. For the other models, the same method TIGER was used. A detailed description of the different band-split schemes can be found in Appendix
Table 4 shows the performance and efficiency of models using different band-split schemes. While adopting band-split increased the number parameters due non-shared 1D convolutions, it sig- nificantly reduced overall computational costs by decreasing the total number of sub-bands. This approach allowed the model to focus on important frequency bands, low and medium bands for speech since human speech typically ranges from 85 Hz to 100 Hz (Loizou, 1998). The LowFreq- NarrowSplit scheme offered finer splits in low- low-frequency bands compared to NormalSplit, resulting in enhanced performance. In contrast, EvenSplit maintained the same number of sub-bands with an even distribution. leading to a drop in SDRi and SI-SDRi compared to LowFreqNarrowSlii which highlights the effectiveness of band-split in capturing critical frequency information.

---

Published as a conference paper at ICLR 2025
Replacement structures
SDRi (dB)
SI-SDRi (dB)
Params (M)
MACs (G/s)
GPU Time (ms)
LSTM
13.92
13.41
2.05
49.38
83.16
Mamba SRU
13.02
12.59
1.33
21.03
59,91
12.87
12.43
1.00
20.30
53.26
MSA (ours)
13.15
12.58
0.82
7.65
42.38
Table 6: Comparison of performance with different structures to replace MSA module.
Replacement structures
SDRi (dB)
SI-SDRi (dB)
Params (M)
MACs (G/s)
LSTM
12.64
12.05
2.46
51.59
Mamba
12.20
11.78
1.74
25.43
SRU
12.48
11.97
1.41
22.71
F A (ours)
13.15
12.58
0.82
7.65
Table 7: Comparison of performance with different structures replace F A module.
Importance of MSA and F A modules. We investigated the role of the MSA and F modules in model performance. To this end, we constructed two controlled models, removing each of these modules. As shown in Table 5 removing the MSA module resulted in the worst results, which validates the effectiveness of the MSA module in speech separation, it fully integrates multi-scale features. The performance also decreased after the F module was removed, indicating that the global integration of time and frequency helps TIGER extract relevant auditory features. Overall, the results show the MSA and F modules play an important role in improving performance.
Furthermore, MSA module and F module can be replaced by other structures for sequence data modeling, such as LSTM (Graves & Graves 2012). SRU (Lei et al., 2018), and Mamba (Gu & Dao 2023). We then evaluated the impact of replacing the MSA and F*A modules with different sequence modeling structures. We first replaced the MSA module in TIGER with LSTM, SRU, and Mamba, with detailed replacement methods provided in Appendix F The experimental results are shown in Table 6 We observed that the MSA module significantly reduced the computational load while maintaining strong performance. Although LSTT demonstrated better performance in sequence data modeling, the iterative nature of RNN computations resulted in the GPU inference time being twice as long as that of the MSA-based separator. While linear RNN structures like SRU and Mamba sped up inference to some extent, there remained a gap in separation performance and efficiency compared to the MSA module. This highlights the importance leveraging multi-scale information for both temporal and frequency modeling.
Next, we replaced the self-attention mechanism in the F module with LSTM, SRU, and Mamba to evaluate the effect of different structural replacements. The experimental results are presented in Table 7 We found that the F A module produced the best results among the four experiments, mainly because long-range dependencies captured by the self-attention module help enhance the global context of frequency and temporal features.
We also verified the impact of alternating the frequency path and frame path on model performance, and explored the lightweight potential of TIGER under a smaller parameter scale. See Appendix G and for more details.
CONCLUSION
In this paper, we present TIGER, an efficient frequenyy domain speech separation model with significantly reduced parameters and computational costs. TIGER effectively extracts key acous- tic features through frequency band-split, multi-scale and www.ffuuuncy-ffaantet modeling. We also introduce the EchoSet dataset that simulates realistic acoustic scenarios. Experiments showed that TIGER outperformed existing SOTA models in complex acoustic environments, with 94.3% fewer parameters and 95.3% less computational costs, and demonstrated good generalization ability in the task of movie audio separation. TIGER provides new ideas for designing lightweight speech separation models suitable for devices with limited resources.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China (No. 202IZD0200301) and the National Natural Science Foundation of China (No. U2341228).
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine In- lelligence, 44(12):877-88777, 2018.
Chunxiao Cao, Zhong Ren, Carl Schissler, Dinesh Manocha, and Kun Zhou. Interactive sound propagation with bidirectional path tracing, ACM Transactions on Graphics, 35(6):1-11, 2016.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision, pp. 667-676. IEEE, 2017.
Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. In Advances in Neural Information Processing Systems, volume 35, pp. 8896-8911, 2022.
E Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the Acoustical Society of America, 25(5):975-979, 1953.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vin- cent. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.11262. 2020.
Pierre Divenyi. Speech separation by humans and machines. Springer Science & Business Media, 2004.
Alex Graves and Alex Graves. Long short-term memory. Supervised Sequence Labelling with Recurrent Neural Networks, pp. 37-45, 2012.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00752. 2023.
Simon Haykin and Zhe Chen. The cocktail party problem. Neural Computation, 1759)1175-1192,, 2005.
John Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrim- inative embeddings for segmentation and separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 31-35. IEEE, 2016.
Xiaolin Hu, Kai Li, Weiyi Zhang, Luo, Jean-Marie Lemercier, and Timo Gerkmann. Speech separation using asynchronous fully recurrent convolutional neural network. In Advances in Neural Information Processing Systems, volume 34, pp. 22509-22522 2021.
Berkan Kadıoglu, Michael Horgan, Xiaoyu Liu, Jordi Pons, Dan Darcy, and Vivek Kumar. An empirical study of conv-tasnet. In International Conference Acoustics, Speech and Signal Processing, pp. 7264-7268. IEEE, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.66.41..6980. 2014.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. bar-heaf-baked well done? In International Conference on Acoustics, Speech and Signal Processing, pp. 626-630. IEEE, 2019.
Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: A unified approach to action segmentation. In Computer Eision-ECCV 2016 Workshops, pp. 47-54. Springer, 2016.
11

---

Published as a conference paper at ICLR 2025
Tao Lei, Yu Zhang, Sida I. Wang. Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Empirical Methods in Natural Language Processing, 2018.
Kai Li and Yi Luo. On the design and training strategies for mn- masedaeed online neural speech separa- tion systems. In International Conference on Acoustics, Speech and Signal Processing, pp. 1-5. IEEE, 2023.
Kai Li, Xiaolin Hu, and Yi Luo. On the use of deep mask estimation module for neural source separation systems. In Conference the International Speech Communication Association, 2022.
Kai Li, Runxuan Yang, and Xiaolin Hu. An efficient encoder-decoder architecture with top-down attention for speech separation. In The Eleventh International Conference Learning Represen- tations, 2023.
Kai Li, Guo Chen, Runxuan Yang, and Xiaolin Hu. Spmamba: State-space model is all you need in speech separation. arXiv preprint 2xXiv:240.002063 2024.
Philipos C Loizou. Mimicking the human ear. IEEE Signal Processing Magazine, 15()):101-130 1998.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time- frequency magnitude masking for speech separation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 27(8): 1256-1266, 2019.
Yi Luo and Jianwei Yu. Music source separation with band-split rnn. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2023.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path mn: efficient long sequence modeling for time-domain single-channel speech separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 46-50. IEEE, 2020.
Yi Luo, Cong Han, and Nima Mesgarani. Group communication with context codec for lightweight source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 1752-1761, 2021.
Matthew Maciejewski, Gordon Wichern, Emmett McQuinn, and Jonathan Le Roux. Whamr!: Noisy and reverberant single-channel speech separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 696-700. IEEE, 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In International Conference on Acoustics, Speech and Signal Processing, pp. 5206-5210. IEEE, 2015.
Darius Petermann, Gordon Wichern, Zhong- Zhong-Qiu Wang, and Jonathan Le Roux. The cocktail fork problem: Three-stem audio separation for real-world soundtracks. In International Conference Acoustics, Speech and Signal Processing, pp. 526-530. IEEE, 2022.
BS Series. Algorithms to measure audio programme loudness and true-peak audio level. In Inter- national Telecommunication Union Radiocommunication Assembly, 2011.
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In International Conference Acoustics, Speech and Signal Processing, pp. 21-25. IEEE, 2021.
Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo mm-rf: Efficient networks for universal audio source separation. In IEEE 30th International Workshop on Machine Learning for Signal Processing, pp. 1-6. IEEE, 2020.
Stefan Uhlich, Giorgio Fabbro, Masato Hirano, Shusuke Takahashi, Gordon Wichern, Jonathan Le Roux, Dipam Chakraborty, Sharada Mohanty, Kai Li, Yi Luo, al. The sound demixing challenge 2023-cinematic demixing track. Transactions of the International Society for Music Information Retrieval, 2024.
12

---

Published as a conference paper at ICLR 2025
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462- 2006.
Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Making Lime-rrequenyy domain models great again for monaural speaker separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 1-5.
IEEE, 2023.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow. and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. arXiv preprint 1rXiiv1907.01100 2019.
Lei Yang, Wei Liu, and Weiqin Wang. Tfpsnet: Time-frequency domain path scanning network for speech separation. In International Conference on Acoustics, Speech and Signal Processing, pp- 6842-6846. IEEE, 2022.
A DATASET DETAILS
EchoSet. This dataset includes 20268 training utterances, 4604 validation and 2650 test utterances. The length of each audio is seconds. The target speech was selected from LibriSpeech (Panayotov et al. 2015), mixed with SDR ranging from -5 dB to 5 dB. The speech and noise which was sampled from WHAM! were mixed with SDR sampled between -10 dB and dB. This dataset contains realistic reverberation. The sampling rate is 16 kHz.
LRS2-2Mix eLi al. 2023). Each audio in this dataset lasts for 2 seconds, at the sampling rate of 16 kHz. The training set, validation set and test set about 1 11.1, 2.8 and 1.7 hours, respectively. The utterances were selected from the LRS2 (Afouras et al. 2018) corpus, which consists of video clips acquired through BBC, and were mixed with SDR sampled between -5 dB and 5 dB. Since the audio files were recorded in real acoustic scenarios, LRS2-2Mix contains much noise and reverberation.
Libri2Mix (Cosentino al. 2020). Fach audio in this dataset lasts for 3 seconds. The target speech for each audio mixture was randomly chosen from LibriSpeech (Panayotov et al. 2015) (Irain- 100) and combined with a uniformly sampled Loudness Units relative to Full Scale (Series 2011) between -25 and -33 dB. We adoped the 16 kHz min version with no noise or reverberation in our experiments.
Real-world data. We collected small-scale dataset from the physical world to test the performance of models trained on different datasets in real- world scenarios, with each audio clip 60 seconds long. Its data collection process is described as follows. First, we selected 10 rooms of varying sizes and shapes as distinct acoustic environments. Then, we randomly sampled approximately 1.5 hours of 16 kHz speech audio from the LibriSpeech test set (Panayotov et al. (0115), and sampled noise data from the WHAM! noise dataset (Wichern et al. 2019). During the recording process, audio content was played using the speakers of 2023 MacBook Pro and recorded via Logitech Blue Yeti Nano omnidirectional microphone placed in a fixed position. The distance between the speaker and the microphone was randomly selected from 0.3 to 2 The recording parameters were set 16 kHz sampling rate and 32-bit depth. This setup ensured that both speech and noise were recorded in the same room, preserving the authenticity of the reverberation effects. Finally, we processed the collected audio by mixing the recordings. Specifically, audio from different speakers was mixed using SDRs randomly sampled between -5 dB and dB. Noise data was added using SDRs randomly sampled between- 10 dB and 10 dB. Since the propagation paths of sounds in the air are independent of one another, mixing these components considered a reasonable approach. This design ensures the realism and diversity of the evaluation dataset, effectively capturing the complexity of speech separation real-world conditions.
B TRAINING CONFIGURATION
In the encoder and decoder, the window and hop size of STFT and iSTFT were set to 640 (40 ms) and 160 (10 ms). We use the Hanning window mitigate spectrum leakage. According to the
13

---

Published as a conference paper at ICLR 2025
Nyquist sampling theorem, the frequency range represented was 0-8 kHz for audio with sampling rate of 16 kHz. In this way, each frame was represented by -im-diimllal complex spectra, and the frequency resolution was 25 Hz. We adopt the band-split scheme LowFregNarrowSplit in Table
10 The number of total sub-bands K was 67. For each sub-band, the bandwidth was uniformly transformed into N = 128. In the separator, the FFI blocks which share parameters were repeated B = times for the small version and B = 8 times for the large version. Each MSA module's features were downsampled for D = times, and the hidden layer dimension I was
256. For F$ A module, the number of attention heads was set to 4. When calculating the query and key in each head of the FS module, the hidden channel was set to 4.
During training, We used 3-second audio segment for EchoSet and Libri2Mix, and 2-second for LRS2-2Mix. We used the maximization of SI-SDR as the training loss (Le Roux et al. 2019). The maximum training round was 500. We used Adam as the optimizer (Kingma & Baj 2014). with the initial learning rate set to 0.001. the loss on the validation set did not decrease further within 10 consecutive rounds, the learning rate was halved. When the performance the validation set did not improve further within 20 consecutive rounds, the training was stopped.
C EVALUATION CONFIGURATION
In all experiments, we reported the quality of separated audio on SDRi (Vincent et al. 2006 and SI-SDRi (Le Roux al. 2019:
SDRi (D. SDR(S,
(8)
SI-SDRI = SI-SDR(P,,,, P.) SI-SDR(S,
(9)
When evaluating model performance on real-world data, we used the training lengths (Libri2Mix: 3s, LRS2-2Mix: 2s, EchoSet: 3s) of the respective datasets inference the 60-second audio with a 50% overlap sliding scale. This approach to some extent mitigates the problem of model per- formance degradation that may be caused by the difference in training and inference lengths, thus ensuring fairness in the model's performance comparison on the real-world data.
To measure the complexity of the model, we used parameters and multiply-accumulee operations (MACs) for theoretical analysis. In the speech separation task, since the audio length is not fixed, we used MACs for separating one-second audio as an indicator for complexity evaluation. We used ptflops 0.7.3 to calculate parameters and MACs. For actual evaluation, we performed the backward process (training) and forward process (inference) 1000 times, respectively, on one second of audio at 16 kHz sampling rate, then took the average to indicate the training and inference speed. We reported the GPU time and GPU memory usage during the training process, as well as the CPU time, GPU time, and GPU memory usage during the inference process. To simulate the limited computational conditions of mobile devices on which the speech separation model is deployed in real-world situations, fixed the number of threads to when calculating CPU (Intel(R) Xeon(R) Gold 6326) time and only used a single card when calculating GPU (GeForce RTX 4090) time.
D CINEMATIC SOUND SEPARATION TASK
The cinematic sound separation task (Uhlich et al. 2024 is to separate different signals from mixed audio, including speech, music and sound effects. We migrated TIGER to cinematic sound separa- tion to test the generalization ability of the model similar tasks.
We tested TIGER's performance on the DnR dataset, which consists of three tracks: speech, music, and sound effects. The length of each audio is 60 seconds. Each track does not completely overlap, and the sampling rate is 44.1 kHz. The dataset is composed of 3295 training audio, 440 validation audio, and 652 test audio.
According to the composition of the mixed audio, the band-split scheme was adjusted as shown in Table 8 Since the frequency of human hearing ranges from 20 Hz 20000 Hz, there was no need to split the high-frequency band above 20000 Hz. The window size W of STFT was set to 2048,
htspa://pypt.tt wwtg://oojst/pppeeoooooooo lopp/// opp. /0.33/
14

---

Published as a conference paper at ICLR 2025
Range (Hz)
Width (Hz)
Total sub-band number
0-1000
50
20
1000-2000
100
10
2000-4000
250
8
4000-8000
500
8
8000-16000
1000
8
16000-20000
2000
2
20000-44100
22100
1
Table Band-split scheme on DnR
and the stride was set 512. The feature dimension was set to 132. In the separator, the FFI blocks were repeated for B = 8 times. Other settings remained unchanged.
As for the training configuration, in order to improve the speed of the training phase, each 60-second training audio in DrR was segmented using Voice Activity Detection (VAD). Then 3 seconds of audio was randomly sampled from each component to synthesize the mixed audio. The sum of mean absolute error (MAE) in the frequency domain and the time domain was used as the training loss, which was the same as (Uhlich et al. 2024):
1 f= c E
- P 6 Σ STFT(P.) STFT(P:))-
(10)
C
The maximum training epochs were 500. AdamW was used as the optimizer, and the initial learning rate set to Ir 0.001. If the loss the validation set did not decrease further within consecutive rounds, the learning rate was reduced by half. When the performance on the validation did not improve further within 10 consecutive rounds, the training process stopped.
During inference, we employed a sliding window approach, dividing the 60-second audio into 6- second overlapping segments with 50% overlap, and then reassembling the segments back their original length after processing.
Structures
Music (dB)
Speech (dB)
Sound Effect (dB)
Paras (M)
MACs (G/s)
Cony-TasNer* *RX
0.3
8.5
2.0
5.3
19.82
4.2
12.3
30.51
10.59
BSRNN TIGER
5.5
13.8
1.8
52.8
18.2
7.4
15.5
1.40
Table 9: Comparison of performance and efficiency of cinematic sound separation models on DnR. ** means the result comes from the original paper of DnR (Petermann al., 2022).
The experimental results are shown in Table 9 TIGER demonstrated outstanding reconstruction performance across the three audio tracks. Specifically. for the tasks of separating music, speech, and sound effects, TIGER achieved SI-SDR scores of 7.4 dB, 15.5 dB, and 6.5 dB, respectively, significantly outperforming BSRNN, This indicates that TIGER has a stronger capacity for capturing audio features.
Moreover, TIGER's parameters were only 1.40 million, and its computational costs were 4.07 G MACs per second, which kept resource usage at low level and was very efficient. These results further validated TIGER's effectiveness in the domain of cinematic sound separation, providing strong foundation for practical applications.
DETAILS OF DIFFERENT BAND-SPLIT SCHEMES
In Table 10 we list several band-split schemes. For datasets 16 kHz, the full band ranges from 0-8 kHz. Because real-to-complex STFT satisfies the conjugate symmetry, the result be expressed
15

---

Published as a conference paper at ICLR 2025
Scheme
Range (Hz)
Width (Hz)
Number
Total number
NonSplit
0-8000
25
321
321
0-1000
50
20
1000-2000
100
10
NormalSplit
2000-4000
250
8
47
4000-8000
500
8
8000
0-1000
25
40
1000-2000
100
10
LowFregNanoowSplitt
2000-4000
250
8
67
4000-8000
500
8
8000
EvenSplit
0-6600
100
66
67
6600-8000
1400
Table 10:
Different frequency band-split schemes and their corresponding frequency ranges, band- widths, and numbers of sub-bands.
using only one side. According to the implementation the torch.stf when the window size was 321.
set to 640, the encoding dimension was [640/2]
For the NonSplit scheme, we didn't apply band-split and kept the original frequency samples 321. The width of each sub-band was 25 Hz. The total sub-band number was 321. We write the mixed audio after STFT as X E C x2', The real and imaginary part of X were treated as two channels and stacked on the channel dimension to obtain feature X E R2xFxT Then a 2D convolutional layer was applied to X to expand the channel dimension to N. In this way, we got the input Z XxXT for the separator (K F 321 in this case).
For the NormalSplit scheme, we split finer in the low-frequency part. Specifically, we split 0-1000 Hz by a 50 Hz bandwidth. Since resolution was 25Hz, 2 frequency samples were treated as one band. The total sub-band number in 0-1000 Hz 20. Accordingly, GR = 2 when & e [1, 20). Similarly, we split 1000-2000 Hz by a 100 Hz bandwidth. 4 frequency samples were treated as one sub-band and the total sub-band number in 1000-2000 Hz was 10, i.c. G₁ 4 when & e [21, 21,30]. For 2000-4000 Hz and 4000-8000 Hz, 10 frequency samples and 20 frequency samples treated one band, respectively. Therefore G₃ 10 when k (31,38) and G₁ 20 when E [39, 46). Since there were 321 frequency points in total, there was one endpoint left, corresponding to 8000 Hz. Thus Ga when A: 47. There were 47 sub-bands in total. When adopting band-split strategy, the real part Re(-) and imaginary part Im(-) of the frequency sub-band By are no longer treated as two channels, but are merged into the frequency dimension. Then we obtain B E 222xxT xT Group normalization layers and ID convolutions are used map the frequency dimension 2G to the feature dimension N, and then K sub-bands are stacked to obtain input feature E RN xX for the separator.
scheme, we split the low-frequency area less roughly. In the range of 0-1000 Hz, we split the band by 25 Hz for each sub-band. This way, I frequency sample was treated as sub-band, and the total sub-band number in 0-1000 Hz was 40. Other bands remained the same as NormalSplit. Therefore, we had G3 when & E [1, 40]: G = 4 when E [41, 50]; G₃ = 10 when & [51, 58]; G 20 when & E [59, 66); G when A = 67. The implementation kept the same as the SprmalSplit scheme.
For the I LowFregNarrow
For EvenSplit, 0-6600 Hz split evenly by 100 Hz sub-bands Each sub-band consisted of 4 frequency samples. The remaining part was treated as one sub-band. Accordingly, we had Gg- = when & E 66); G5 57 when k 67. The band-split detail was also the same as the NormalSplit scheme.
16

---

Published as a conference paper at ICLR 2025
F DIFFERENT STRUCTURES IN MSA AND FA MODULES
In the experiments where we replaced the MSA and F'A modules, we used LSTM (Graves & Graves 2012). SRU (Lei 2018), and Mamba (Gu & Dao 2023) as the alternatiee model struc- tures. When we substituted LSTM for the MSA module, the input 2, E R xxNXXx x is first normalized by group normalization. Then we apply a bi-directional LSTM with the hidden size the same as hidden layer dimension H in the MSA module, generating the hidden feature Z E IR2H xT, Next we restore hidden layer dimension to input dimension using linear projection. The out- put of LSTM is 2 xxxx The configuration for SRU was consistent with that of LSTM. For Mamba, since a causal model and cannot access future information, we utilized the BMamba layer, as proposed in SPMamba (Li et al. 2024), to model sequence information bidirectionally, followed by linear layer to compress the feature channels.
G A ABLATION STUDY: TIME-FREQQENCY INTERLEAVING
SDRi (dB)
SI-SDRi
Paras
MACs
GPU Time
Structure
(dB)
(M)
(G/s)
(ms)
T-T
12.91
12.32
0.82
7.75
42.72
F-F
10.57
9.69
0.82
7.53
41.39
F-T (ours)
13.15
12.58
0.82
7.65
42.38
Table 11: Comparison of performance and efficiency of models with different modeling paths in the FFI block. T-T means the FFI block consists of two frame paths, while F-F means the FFI block consists of two frequency paths.
In the separator of TIGER, we model time and frequency features the mixed audio alternately. To demonstrate the effect of timeffreuenncy interleaved structure, tested the performance of F-F and T-T structures. For F-F, we replace the frame path with the frequency path in the FFI blocks. In other words, each FFI block only includes two frequency paths which process the input Zo RN KxT and Zz.s E IN X the same way but don't 't share parameters. All FFI blocks still share parameters. The implementation is similar for T-T.
According to the result shown in Table 11 compared with only modeling time or only modeling fre- quency, the fime-freyuency interleaved structure can better capture time and frequency information of audio, which facilitates improving performance while keeping the model lightweight.
H THE LIGHTWEIGHT POTENTIAL OF TIGER
To further illustrate the lightweight potential of TIGER, present the experimental results of smaller version of TIGER as well as the compressed SudoRM-RF model (Tzinis et al. 2020) based on GC3 method (Luo al. 202T) EchoSet. The hyperparameters of TIGER (tiny) were set as follows: the feature dimension N was reduced from 128 to 24; the hidden layer dimension H was reduced from 256 to 64; the number of FFI blocks B 4. The hyperparameter settings for SudoRM-RF were strictly in accordance with the publicly available configurations of GC The efficiency was evaluated on one-second audio input. As shown in Table [2] TIGER (tiny) significantly outperforms SudoRM-RF + GC3 at comparable efficiencies, which proves that TIGER a very effective lightweight model.
VISUALIZATION
In order to intuitively demonstrate the separation performance of TIGER, we provide some examples for visualization, shown in Figure [ The following spectrograms show the inference results of TIGER (large) and TF-GridNeet on the same audio, and the ground truth. Sample I and show
17

---

Published as a conference paper at ICLR 2025
SDRi (dB)
SI-SDRi
Paras
MACs
GPU Time
Method
(dB)
(K)
(G/s)
(ms)
SudoRM-RF GC3
5.0
4.6
303.57
0.81
32.68
TIGER (tiny)
10.7
10.4
102.12
0.72
30.21
Table 12: Performance and efficiency comparison of SudoRM-RF GC3 and TIGER (tiny).
that TIGER produces finer reconstruction results at high frequencies compared with TF-GridNet. TIGER also has better effects in noise reduction and spectrum leakage prevention, illustrated in Sample III and IV.
204
3000
1MC
304
3000
LMC
(a) Sample
Ground fruth
TF-GridNet
304
3/06
476
(b) Sample II
Ground truth
TIGER
TF-GridNet
(c) Sample
truth
506
500
404
AGB
(d) Sample IV
Figure 5: Comparison of the spectrograms of the ground truth, audio separated by TIGER and by TF-GridNet.
18


Reviews:
[{"review_id": "J2qQxn7cWJ", "reviewer": "Reviewer_oAuH", "summary": "The paper developed a deep learning model for speech separation; and it not only focuses on improving model performance but model efficiency. The model is lightweight, and is constructed of a new band-split strategy and a new frequency-frame interleaved (FFI) block. Additionally, to reduce the gap between synthetic data and real-world, it introduced a dataset, named EchoSet, with different noise and realistic reverberation. Experimental results demonstrates the effectiveness and efficiency of the proposed model, and the generalization of the dataset to real-world audio.", "questions": "The model source code is open, the description of EchoSet is reasonable, but it would be better if the configuration/code of data generation is also open.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper is well-written and easy to follow. The proposed model architecture looks reasonably nice with the introduction of a new band-split strategy and a new FFI block. Experimental results show that the model not only archieves competitive performance on all datasets, but also it is very lightweight in terms of model sizes, MACs. Its efficiency in training (GPU time & memory) and inference (CPU time, GPU time & memory) looks good, too.\nThe motivation of creating EchoSet dataset is clear and reasonable with a complete analysis of existing dataset. Experimental results on the real-world data with different models, trained on different datasets respectively, demonstrate its generalization ability.", "weaknesses": "The FFI block follows a common design of dual-path architecture, it consists of 2 different parts: frequency path and frame path. Each path has two main modules: multi-scale selective attention (MSA) and full-frequency-frame attention (F^3A). While F^3A looks familiar with self-attention mechanism, MSA extracts features through a selective attention mechanism at multiple scales. We may need an ablation study of the MSA architecture with different scales (e.g. 1,2,3,4) to see how it affects model performance.\nThe evalution on model efficiency is conducted with a fixed audio input length of 1 second. It would be more complete if the evaluation was with different audio input lengths (each audio in EchoSet is 6 seconds, LRS2-2Mix: 2 seconds, Libri2Mix: 3 seconds)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iw2jDvfVnk", "reviewer": "Reviewer_27Ea", "summary": "The paper introduces a novel model architecture that has small number of parameters and performs as well as other larger models on various separation tasks.\n\nThe model architecture is similar to bandsplit RNN in its encoder/decoder part and in its core part, it has two novel attention-based submodules called MSA and F3A which are applied in the frequency direction and time (frames) direction separately in identical fashion. These modules are repeated a fixed number of times. When repeating, the parameters are tied, so the number of parameters in the model do not increase when the repetition count is increased.\n\nThe paper also introduces a new dataset called EchoSet which includes reverberant speech mixtures which improved performance on a \"real\" dataset. The \"real\" dataset is not real in the sense of real speech conversations, but it is obtained through playback in a real acoustic environment of individual sources which are mixed later for evaluation.\n\nThe performance on real data shown in Figure 4 is much lower compared to the results with artificial mixtures even when training on EchoSet, which shows there is still a lot of room for improvement when applying these models on real data.", "questions": "1. What was the loss function?\n2. How does selective attention (SA) work?\n3. What is the reasoning behind repeating the same FFI block (with tied parameters)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well written and the model is described in a detailed manner even though there may be room for improvement in the model description for more clarity.\n\nThe method applies to many different sampling rate data due to bandsplit RNN based encoder/decoder. The model is also used to perform cinematic audio separation at 44.1 kHz.\n\nThe ablations and various comparisons with state-of-the-art on multiple relevant datasets are all appropriate and impressive.", "weaknesses": "Loss function was not mentioned in the main text (or I missed it). Is the loss in (10) in the appendix used for all tasks, or only for cinematic sound separation?\n\nThe math in the MSA module description gets a bit hard to follow, so maybe a more detailed Figure that tracks along with mathematical equations would help. How does selective attention (SA) work? It was not described in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K5YGdXfqg0", "reviewer": "Reviewer_kFvq", "summary": "This paper presents an approach named TIGER for speech separation, emphasizing both efficiency and performance. The proposed approach utilizes frequency band-splitting, selective attention modules, and F3A to manage both temporal and frequency information. In addition, the authors introduced a new data set named EchoSet, that considers the realistic environmental factors such as noise, reverberation, and varied speaker overlap ratios. Experimental evaluations reveal that the proposed approach achieves competitive results across multiple benchmarks.", "questions": "1) The authors mentioned that the band-split module is inspired by prior knowledge of music separation. it would be helpful to clarify if the targeted speech separation task has similar prior knowledge and to highlight any key differences or similarities between the band-split module used here and that in the original paper.\n2) In Figure 2, the authors mention that \"Residual connections are used to retain original features and reduce learning difficulty.\" Could you please indicate where these residual connections are depicted in the figure?\n3) In Figure 4, the authors use a line chart to present the results. This may be misleading because the results on the three datasets are independent of each other. A different visualization style, such as a bar chart, might provide a clearer comparison.\n4) What are the differences between TIGER (small) and TIGER (large), why do they have the same model size?\n5) In the ablation study, it looks like the proposed modules appear to involve trade-offs among various performance criteria. Could the authors discuss how to balance this tradeoff and what could be the best to consider?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The authors' provision of open-source code enables researchers in the field to reproduce and build upon this work.\n2) The authors have a detailed ablation study, as it clarifies the impact and effectiveness of each proposed module.", "weaknesses": "1) The proposed approach does not demonstrate clear performance advantages over the current SOTA method, with a noticeable performance gap.\n2) Although the authors claimed their approach is more lightweight, the comparison is not entirely fair. A comparison with other systems that specifically employ lightweight methods would provide a more accurate assessment of the model's efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5VkqVVBp8", "reviewer": "Reviewer_MkhV", "summary": "This paper presents TIGER, an efficient speech separation model with significantly reduced parameters and computational costs, optimized for low-latency applications. Leveraging frequency band segmentation and attention mechanisms, TIGER captures rich contextual information. The authors introduce EchoSet, a realistic dataset for evaluating model robustness in complex environments. TIGER achieves state-of-the-art performance with 94.3% fewer parameters and 95.3% fewer MACs than existing models.\n\nThe result is very impressive and the details is rich. They also provide a high quality dataset which would be beneficial to the field.", "questions": "1. How would the model deal with the audio with background noise? \n2. Does the method still work when two people speaking in different setup? e.g. one people is much further to the speaker than another. \n3. How should the model accommodate variable number of speakers?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Provide a new way of speech separation with very efficient paramater usage. It is very important to the industry and would be very influential \n2. provide a new dataset which provides different content compared to all other similar kinds before.", "weaknesses": "1. The model focused on two speaker separation which makes the task much simpler. May need to provide a way to generalize the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YO69A6wHEW", "reviewer": "Reviewer_Sqdv", "summary": "In this paper, the authors propose a speech separation setup which relies on multi-scale selective attention and interleaved time-frequency attention modules. The design is relatively lightweight, significantly reducing the number of model parameters and consequently, computing resources in inference. They also introduced   a speech separation corpus, EchoSet, that allegedly has more realistic noise and reverberation. They conducted comprehensive empirical evaluations of several comparable models and public data sets.", "questions": "In Table 4, how does LowFreqNarrowSplit compare with Mel-split?\nHow scalable is Tiger for multi-talker separation?\nHow scalable is Tiger for audio separation beyond speech, as alluded to in Table 9, where music and environmental sound is still a lot harder than speech. What's the path forward?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written overall, following a logical organization and making sound arguments. The development of the proposed system is easy to follow. The details of the modules inside the systems are adequately presented. The evaluation results, given the nature of the paper as an empirical study, can seem a bit under-developed, but the major evaluations are made and presented. The ablation studies are important to help readers understand the importance of the MFA and F3A modules. That being said, similar ablation studies can also be performed on the EchoSet, to determine which aspect of the simulation set this new corpus apart from the former ones. \nThe paper highlighted the strong performance of the proposed speech separation system and its small footprint in terms of model size.", "weaknesses": "The comparison between the various corpus is not strictly fair, given the data sets have different sizes. From the data provided by the authors, EchoSet contains 33.78 hrs, whereas LRS2-2Mix has 11.1 hours and Libri2Mix has 58 hrs. Did the authors augment/select the data to ensure that the same amount of data is used in training to evaluate the systems in Table 2?\nThe argument in Sect. 6.1 is not entirely clear to me. What does data collected in the real world refer to? Is there any description of this data?\nTo measure the model complexity, model parameters are acceptable, but MACs is inadequate. To go beyond theoretical analysis, the authors can measure the memory usage, power consumption, throughput and latency on server or edge devices. \nRelated to the comments above, Sect. 6 can be beefed up with more analysis. For example, does TIGER or models trained with EchoSet perform better in lower SNR/stronger reverb compared to other models? Can we break it down? How does Tiger/EchoSet fare on male-male pair vs. male-female pair, etc.?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V0PrhSFHDS", "reviewer": "Reviewer_awAb", "summary": "The paper has two main contributions:\n1. The TIGER architecture which is a lightweight time-frequency model\nfocusing on reducing model size while keeping separation accuracy high.\n2. The EchoSet dataset for two speaker speech separation which attempts\nto resemble real world scenarios more closely than previous datasets by using\n3D environments.", "questions": "1. The assembly of the real world data is somewhat confusing. As I understand\nit, the utterances as well as the noise was rerecorded in the real world and then\nmixed together at different SDRs. Why did you not play multiple utterances\nand noise from different locations in the room at the same time which would\nresemble a real world mixture more closely rather than mixing them together\nafterwards?\n2. Computational cost was measured on a one second input at 16 kHz.\nThis choice is never motivated and in the context of speech separation illogical.\nWhile there are some previous papers that have done the same [2, 3], neither\nof them had any justification for this choice either. The standard for speech\nseparation would be an input of at least 4 seconds [4, 5]. The reasoning is\nthat the WSJ0-2Mix, contains utterances with an average length of about 4-5\nseconds. The EchoSet, meaning the dataset the paper proposes, uses 6 second\nutterances. Why choose a completely different length for the calculation of\nspeed and memory usage?\n3. The sampling rate for the speech separation datasets is only specified\nfor the LRS2-2Mix as 16 kHz. Is it the same for the other two (Libri2Mix and\nEchoset)? If so, why? Typically, 8 kHz is used and while there is nothing to be\nsaid against using 16 kHz, it still makes future comparisons difficult to not also\ninclude 8 kHz data.\n\nReferences\n\n[2] Kai Li, Runxuan Yang, and Xiaolin Hu. “An efficient encoder-decoder\narchitecture with top-down attention for speech separation”. In: ICLR.\n2023.\n\n[3] Chen Chen et al. “A Neural State-Space Modeling Approach to Efficient\nSpeech Separation”. In: Proc. INTERSPEECH 2023. 2023, pp. 3784–3788.\ndoi: 10.21437/Interspeech.2023-696.\n\n[4] Zhong-Qiu Wang et al. TF-GridNet: Integrating Full- and Sub-Band Modeling\nfor Speech Separation. 2023. arXiv: 2211.12433 [cs.SD].\n\n[5] Cem Subakan et al. Exploring Self-Attention Mechanisms for Speech Separation.\n2023. arXiv: 2202.02884 [eess.AS].", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "The paper proposes a novel dataset, the EchoSet, which not only includes noise\nand reverberation, but also object occlusion and different materials through\n3D environments. The proposed speech separation architecture achieves high\naccuracy on this new dataset while having few trainable parameters.", "weaknesses": "There are many issues with the experimental part of the paper, especially the\nevaluation of the proposed model architecture.\n\nThe first issue is the datasets that were chosen. The main dataset for\nspeech separation, the WSJ0-2Mix, was not used. Instead, three others were - the proposed EchoSet, the LRS2-2Mix and the Libri2Mix. While choosing\nthe LRS2-2Mix over WHAMR! is motivated, the main problem concerns the\nLibri2Mix. The results of the Libri2Mix reported here do no coincide with\nprevious reported results. The Conv-TasNet normally reaches 14.7 dB SI-SDRi\nwhile this paper only report 12.1 dB SI-SDRi. Since all of the results reported\nin this paper disagree with previous results, I assume that this is caused by\nthe choice of limiting the sequence length of the Libri2Mix to 3 seconds, as is\nmentioned in the appendix. It also appears that this paper only uses utterances\nfrom train-100 and leaves out utterances from train-360 as is used in the original\nLibri2Mix. Therefore, this paper significantly changed the dataset of the Libri2Mix\nwhile still calling it Libri2Mix. Since the training set was clearly significantly\naltered, the results obtained here are very difficult to interpret because the\nnormal Libri2Mix contains more and significantly longer training data.\n\nIf one wanted to include two speaker speech separation, why not just use the\nstandard WSJ0-2Mix which basically everyone uses? And how are future papers\nmeant to compare themselves to the results reported in this paper? Is everyone\nnow meant to use this altered Libri2Mix? Since the source code is included,\nothers could run experiments to figure out the accuracy of the TIGER model\non the normal Libri2Mix or the WSJ0-2Mix. But it simply does not make sense\nto have others go through said effort instead of having the original paper contain\nthis basic information.\n\nThere are, however, even more aspects of the results that are unusual. The\nspeed and memory measurements were taken using an input sequence containing\n16k elements. This does not represent any of the datasets used. The EchoSet\nis reported to use 6 second inputs, meaning it should contain 96k for 16 kHz,\nthe LRS2-2Mix uses 2 second inputs at 16 kHz, meaning a sequence length of\n32k, and the Libri2Mix uses 3 second inputs, meaning a sequence length of 48k\nat 16 kHz. Choosing any (or preferably all) of the sequence lengths used in the\ndatasets would be logical, but showing the results of a much shorter sequence\nlength is not.\n\nHowever, even if all results reported in the paper were correct, they would\nstill not make a convincing case for the TIGER architecture. The biggest\nstrength of it is its low model size. While the experiments concerning accuracy\nas well as computational cost are flawed as described above, the results of the\nTIGER architecture are still not very convincing except for its accuracy on\nthe EchoSet. It is neither particularly fast nor memory efficient and was only\ncompared to a single other recent SOTA architecture, the TF-GridNet, which\nit cannot match in accuracy in 2 of the 3 datasets tested.\nWhile the paper includes an ablation study, it doesn’t answer the most\npressing questions. What kind of performance (in terms of accuracy/speed/memory\nusage) would the TF-GridNet have if it shared model weights across its blocks\nand reduced some parameters to match the TIGER’s model size? If such a\ncomparison was made and the TIGER architecture would be more accurate/faster/used\nless memory, then there would be a better argument for it. As of now, it just\nlooks like it sacrifices accuracy for model size.\n\nThe paper also fails to mention perhaps the most significant previous work\nfor speech separation models with few trainable parameters, which is Group\ncommunication [1]. It is incorrectly claimed that the TIGER architecture is\nthe first speech separation model with less than 1 million parameters, when the\naforementioned paper includes models with less than 100k parameters and is\nroughly 4 years old.\n\nThe paper also claims that the models trained on the EchoSet perform better\non real world signals than the Libri2Mix and LRS2-2Mix. This might be true,\nbut is difficult to say with certainty due to the lack of information in the paper.\nThe real world data that was used is described in the appendix - however,\nimportant details like its sampling rate and duration were left out. The sampling\nrate for the speech separation datasets is only specified for the LRS2-2Mix to be\n16 kHz but I assume it to be the same for the Libri2Mix and the EchoSet since\nthe training configuration appendix talks of frequency ranges from 0-8 kHz.\nThe duration for these datasets, however, is very different - 2 seconds for the\nLRS2-2Mix, 3 seconds for the Libri2Mix and 6 seconds for the EchoSet. If the\naverage duration of the real world data is also closer to 6 seconds, then it would\ngive the EchoSet an unfair advantage over the other datasets. Again, this might\nnot be the case, but without clarification of the duration and sampling rate of all\nthe data used the results given cannot be verified. In fact, the reported results\nin Figure 4 are very unexpected. How is the LRS2-2Mix often even worse than\nthe Libri2Mix (despite using background noise and reverb, same as the real\nworld data) while the EchoSet is massively better than both? The paper never\nattempts to explain this oddity.\n\nReferences\n\n[1] Yi Luo, Cong Han, and Nima Mesgarani. “Group Communication With\nContext Codec for Lightweight Source Separation”. In: IEEE/ACM Transactions\non Audio, Speech, and Language Processing 29 (2020), pp. 1752–1761.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)