TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation

## Abstract

Abstract In recent years, much speech separation research has focused primarily on improving model performance. However, for low-latency speech processing systems, high efficiency is equally important. Therefore, we propose a speech separation model with significantly reduced parameters and computational costs: Time-frequency Interleaved Gain Extraction and Reconstruction network (TIGER) . TIGER leverages prior knowledge to divide frequency bands and compresses frequency information. We employ a multi-scale selective attention module to extract contextual features, while introducing a full-frequency-frame attention module to capture both temporal and frequency contextual information. Additionally, to more realistically evaluate the performance of speech separation models in complex acoustic environments, we introduce a dataset called EchoSet . This dataset includes noise and more realistic reverberation (e.g., considering object occlusions and material properties), with speech from two speakers overlapping at random proportions. Experimental results showed that models trained on EchoSet had better generalization ability than those trained on other datasets to the data collected in the physical world, which validated the practical value of the EchoSet. On EchoSet and real-world data, TIGER significantly reduces the number of parameters by 94.3 % and the MACs by 95.3 % while achieving performance surpassing state-of-the-art (SOTA) model TF-GridNet.
This is the first speech separation model with fewer than 1 million parameters that achieves performance comparable to the SOTA model.

## 1 Introduction

Humans possess the ability to focus on a specific speech signal in noisy environments, a phenomenon known as the “cocktail party effect” (Cherry, ). In speech processing, the corresponding challenge is accurately separating different sound sources from mixed audio signals, a task referred to as speech separation. Speech separation is typically used as a preprocessing step for speech recognition, as it helps enhance recognition accuracy (Haykin & Chen, ). Consequently, it is crucial to ensure that speech separation not only produces clear and distinct outputs on real-world audios but also meets the demands of low latency and computational efficiency (Divenyi, ). Therefore, in practical applications, optimizing the parameters and computational costs of separation models to reduce resource consumption becomes particularly important.

In recent years, the application of deep learning methods to the speech separation task has received widespread attention (Wang et al., ; Li et al., ; ; Li & Luo, ; Subakan et al., ). Although many high-performing speech separation methods have been proposed, two key issues remain insufficiently addressed.

First, when designing a separation model, we should fully take into account the actual application scenarios of the speech processing system, which require low latency and low computational complexity. However, many approaches have primarily focused on improving speech separation performance. For example, TF-GridNet (Wang et al., ) utilizes bidirectional LSTMs and self-attention mechanisms in an alternating manner, while SepFormer (Subakan et al., ) incorporates Transformer-based architectures to enhance sequential modeling capabilities. These models have achieved good results on benchmark datasets but have large model sizes. To make the model more applicable in computationally constrained real-world scenarios, it is critical to make the model more lightweight while maintaining high performance.
TDANet (Li et al., ) introduces an efficient lightweight architecture using top-down attention, achieving competitive performance with lower computational costs than SepFormer (Subakan et al., ). However, as a time-domain method, TDANet struggles to leverage frequency-domain information. On the other hand, time-frequency domain approaches like TF-GridNet (Wang et al., ) model both time and frequency dimensions but require higher computational resources. BSRNN (Luo & Yu, ), which is the SOTA model for music separation, reduces the computational burden by focusing on frequency bands, but the band-split strategy is under-explored in speech separation. How to balance computational efficiency and separation quality by effectively utilizing frequency characteristics in speech separation is still a big challenge.

Second, the commonly used speech separation datasets still exhibit a significant gap from real-world scenarios. Many methods relied on the WSJ0-2mix dataset (Hershey et al., ) for evaluation, which only contains clean audio without noise or reverberation. In addition, the speech signals from two speakers are fully overlapping. Models trained on this kind of dataset are subject to weak generalization and robustness in real-world environments (Kadıoğlu et al., ). Although the WHAMR! dataset (Maciejewski et al., ) introduces noise and reverberation to WSJ0-2mix, the generated reverberation fails to fully take into account factors such as object occlusion and material properties, and the diversity of acoustic scenarios remains limited. Therefore, to more accurately train and evaluate speech separation models for practical use, a dataset that more closely resembles real-world environments is necessary. Specifically, this dataset should include different noise types, cover a wide range of realistic acoustic environments, and have speech overlap ratios that are randomly distributed.

To address the aforementioned two issues, we propose a novel time-frequency domain speech separation model called Time-frequency Interleaved Gain Extraction and Reconstruction network (TIGER), along with a speech separation dataset containing reverberation and noise (EchoSet). TIGER employs frequency band-split and alternating time-frequency modeling, resulting in a model with significantly smaller parameters and higher computational efficiency. EchoSet features diverse background noise and realistic reverberation, with the shape and material of objects in the room specially considered in reverberation generation. Besides, we select diverse scenarios and add speaker and microphone constraints to simulate real-life acoustic scenarios.

Our main contributions are as follows:

- 1.
We propose a novel lightweight separation model named TIGER. TIGER adopts a band-split strategy to reduce computational costs by leveraging prior knowledge in the frequency domain. Furthermore, TIGER introduces the frequency-frame interleaved (FFI) block, composed of two key submodules: multi-scale selective attention (MSA) and full-frequency-frame attention (F^3A). These submodules enable efficient integration of temporal and frequency features.
- 2.
We propose a speech separation dataset called EchoSet. It is a high-fidelity dataset bridging the gap between model training and real-world applications.

Experiments show that models trained on EchoSet generalized better on real-world data than those trained on benchmark dataset LRS2-2Mix (Li et al., ) and Libri2Mix (Cosentino et al., ), validating that audio in EchoSet are closer to the physical world. We then comprehensively evaluated TIGER on Libri2Mix, LRS2-2Mix and EchoSet. As the dataset becomes more complex, TIGER’s superiority in performance becomes more significant. On EchoSet, which is the most complicated among the three datasets, TIGER improved the performance by about 5% compared with TF-GridNet, while reducing the parameters and MACs by 94.3% and 95.3% respectively. When tested on real-world data, TIGER also achieved the best separation performance. The remarkable result shows that TIGER provides a new solution for the design of lightweight speech separation models for practical use in the time-frequency domain.

## 2 Related Work

Speech separation. Speech separation methods can be divided into time domain and time-frequency domain. Time domain methods directly process the original audio signal. Conv-TasNet (Luo & Mesgarani, ) is the cornerstone of time domain separation models. It extracts features by temporal convolutional network (Lea et al., ). To improve the performance on long sequence data, DPRNN (Luo et al., ) divides the temporal sequence into small blocks and alternately performs intra-block and inter-block modeling, which becomes a common paradigm for many following works (Wang et al., ; Subakan et al., ). The time-frequency domain methods apply a Short-Time Fourier Transform (STFT) to transform the waveform into a joint representation of time and frequency. TF-GridNet (Wang et al., ) enhances the temporal context information by a full-band self-attention module. Although TF-GridNet achieved SOTA performance, it involves huge computational costs.

Lightweight models. Some models (Wang et al., ; Yang et al., ; Subakan et al., ) with high computational complexity are difficult to be applied to real-time speech processing on edge devices. To reduce computational costs, TDANet (Li et al., ) draws on the attention mechanism of human brains and designs a lightweight structure. TDANet achieved the same performance with only 5% MACs of SepFormer (Subakan et al., ), inspiring the design of lightweight separation models. In music separation, BSRNN (Luo & Yu, ) uses prior knowledge to split band, performing band merging on less important bands to compress the feature while retaining key frequency band information. BSRNN surpasses the previous music separation model in performance. The band-split strategy is also inspiring for exploring lightweight speech separation models.

Datasets for speech separation. WSJ0-2mix (Hershey et al., ) is an early and commonly used speech separation dataset.
However, there are two main problems with WSJ0-2mix. First, utterances of different speakers fully overlap during the conversation, which is not the case in real world. Second, audio in WSJ0-2mix contain no noise or reverberation. WHAM! (Wichern et al., ) added environmental noises to WSJ0-2mix. To take a step further, WHAMR! (Maciejewski et al., ) added reverberation to the audio. However, it only considered rooms with rectangular shapes and ignored the influence of materials and other non-rectangular shapes in the room on reverberation. Libri2Mix (Cosentino et al., ) was proposed based on the observation (Kadıoğlu et al., ) that the test performance of Conv-TasNet trained on WSJ0-2mix dropped sharply on other separation datasets.
The utterances in Libri2Mix were mixed with sparse overlap, and noises were added to the mixed audio, but reverberation was not considered in Libri2Mix. LRS2-2Mix (Li et al., ) was mixed by video clips acquired through BBC. The audio was recorded in real acoustic scenarios, thus containing much noise and reverberation. However, due to the different recording environments of the clips, such as the shapes and materials of the room and objects, the reverberation obtained when the clips were directly mixed is still unrealistic.

## 3 TIGER

Figure: Figure 1: The overall pipeline for TIGER. We focus on scenarios with only two speakers.
Refer to caption: x1.png

### 3.1 Overall Pipeline

Let $L$ be the sequence length of an audio. Given a monaural mixture audio $\boldsymbol{S}\in\mathbb{R}^{1\times L}$ containing utterances of $C$ speakers and noise $\boldsymbol{n}\in\mathbb{R}^{1\times L}$:

$$ $\boldsymbol{S}=\sum_{i}^{C}\boldsymbol{P}_{i}+\boldsymbol{n},$ (1) $$

the speech separation task is to recover the clean speech of each speaker $\boldsymbol{P}_{i}\in\mathbb{R}^{1\times L}$.

The TIGER system (Figure 1) can be divided into five main components: the encoder, the band-split module, the separator, the band-restoration module, and the decoder. Specifically, we first use STFT as the encoder to convert the mixed audio signal $\boldsymbol{S}\in\mathbb{R}^{1\times L}$ into its time-frequency representation $\boldsymbol{X}\in\mathbb{C}^{F\times T}$, where $F$ and $T$ represent the number of frequency bins and time frames, respectively. Next, we apply a frequency band-split strategy, dividing the frequency bands into $K$ sub-bands of varying widths based on their importance. Each sub-band is transformed into a uniform channel size $N$ using 1D convolutions, and these are then stacked along the frequency dimension to produce the feature representation $\boldsymbol{Z}\in\mathbb{R}^{N\times K\times T}$. Thirdly, $\boldsymbol{Z}$ serves as the input to the separator, which uses FFI blocks with shared parameters to model the acoustic characteristics of each speaker. Subsequently, the band-restoration module restores the sub-bands to the full frequency range using separator output $\boldsymbol{J}\in\mathbb{R}^{N\times K\times T}$, and the mask for each speaker $\boldsymbol{M}_{i}\in\mathbb{C}^{F\times T}$ is applied element-wise product to $\boldsymbol{X}$, producing the separated representation for each speaker $\boldsymbol{H}_{i}\in\mathbb{C}^{F\times T}$. Finally, the inverse STFT is used to generate the clean speech signal $\bar{\boldsymbol{P}}_{i}\in\mathbb{R}^{1\times L}$ for each speaker.

### 3.2 Band-split module

Given a time-frequency representation $\boldsymbol{X}$, we first apply a frequency band-split strategy to divide the frequency dimension into $K$ frequency sub-bands {$\boldsymbol{B}_{k}\in\mathbb{C}^{G_{k}\times T}|k=[1,K]$}:

$$ $F=\sum_{k=1}^{K}G_{k}.$ (2) $$

The widths of the sub-bands $G_{k}$ are not necessarily the same. For each frequency sub-band $\boldsymbol{B}_{k}$, we merge its real $\text{Re}(\cdot)$ and imaginary $\text{Im}(\cdot)$ parts into the frequency dimension to generate $\dot{\boldsymbol{B}}_{k}\in\mathbb{R}^{2G_{k}\times T}$. We denote the concatenation operation as $||$, then:

$$ $\dot{\boldsymbol{B}}_{k}=\text{Re}(\boldsymbol{B}_{k})||\text{Im}(\boldsymbol{ B}_{k}).$ (3) $$

Next, we transform the frequency dimension $2G_{k}$ of $\dot{\boldsymbol{B}}_{k}$ to the feature dimension $N$ using a group normalization layer followed by a 1D convolution, which utilizes a kernel size of 1 and does not share parameters across different $\dot{\boldsymbol{B}}_{k}$. In this way, we obtain feature $\boldsymbol{Z}_{k}\in\mathbb{R}^{N\times T}$ of the same shape for each sub-band. We then stack the features $\boldsymbol{Z}_{k}$ from the $K$ frequency sub-bands along the frequency dimension to yield the input feature $\boldsymbol{Z}\in\mathbb{R}^{N\times K\times T}$ for the separator.

### 3.3 Separator

Figure: Figure 2: The separator of TIGER, consists of several FFI blocks which share parameters. Residual connections are used to retain original features and reduce learning difficulty.
Refer to caption: x2.png

Figure: Figure 3: The structure of the MSA module and the F^3A module. The structures of frequency and frame paths are the same.
Refer to caption: x3.png

In the separator, the input feature $\boldsymbol{Z}$ passes sequentially through $B$ frequency-frame interleaved (FFI) blocks with shared parameters, as shown in Figure 2. In each FFI block, the frequency path is first used to extract contextual information between different sub-bands, producing $\boldsymbol{Z}_{b,f}\in\mathbb{R}^{N\times K\times T}$. Next, we feed $\boldsymbol{Z}_{b,f}$ into the frame path to further model the contextual information between different time frames, generating $\boldsymbol{Z}_{b+1}\in\mathbb{R}^{N\times K\times T}$.

The structures are identical in both the frequency path and the frame path, modeling along the frequency dimension and the time dimension respectively.
Each path consists of two main modules: the multi-scale selective attention (MSA) module and the full-frequency-frame attention (F^3A) module. As illustrated in Figure 3, taking the frequency path as an example, we first apply the MSA module along the frequency dimension $K$ to selectively extract features from $\boldsymbol{Z}_{b}$, which results in enhanced frequency features $\bar{\boldsymbol{Z}}_{b}\in\mathbb{R}^{N\times K\times T}$. Then, the F^3A module is used to integrate information across different sub-bands of $\bar{\boldsymbol{Z}}_{b}$, followed by layer normalization, to produce the output feature of the frequency path $\boldsymbol{Z}_{b,f}$.

#### 3.3.1 MSA module

The MSA module enhances important features through a selective attention mechanism and is divided into three stages: encoding, fusion, and decoding, as shown in Figure 3(a). Taking the MSA module in the frequency path as an example, the input to the module is $\boldsymbol{Z}_{b}$.

The encoding stage. This stage aims to capture multi-scale acoustic features. Specifically, we first use multiple 1D convolutional layers (with a stride of 2 and channel of $H$) to progressively downsample the frequency dimension to $\frac{K}{2^{D}}$, resulting in a set of multi-scale acoustic features $\{\boldsymbol{E}_{d}\in\mathbb{R}^{H\times\frac{K}{2^{d}}\times T}|d=[0,D]\}$, where $d$ denotes the $d$-th layer of downsampling. Next, we apply average pooling layers, denoted as $\lambda(\cdot)$, to downsample all $\boldsymbol{E}_{d}$ to the same frequency resolution $\frac{K}{2^{D}}$. Subsequently, the features with different frequency resolutions are fused into global features $\boldsymbol{G}=\sum_{d=0}^{D}\lambda(\boldsymbol{E}_{d}),\boldsymbol{G}\in
\mathbb{R}^{H\times\frac{K}{2^{D}}\times T}$ by summing. Finally, a multi-layer convolutional (MLC) network is used to transform $\boldsymbol{G}$ into $\boldsymbol{G}^{\prime}\in\mathbb{R}^{H\times\frac{K}{2^{D}}\times T}$.

The fusing stage. In this stage, we fuse the local $\boldsymbol{E}_{d}$ and global $\boldsymbol{G}^{\prime}$ information using the selective attention (SA) module. Specifically, for the $d$-th layer, we first use two 1D convolutions to map $\boldsymbol{G}^{\prime}$ into $\tau\in\mathbb{R}^{H\times\frac{K}{2^{D}}\times T}$ and $\rho\in\mathbb{R}^{H\times\frac{K}{2^{D}}\times T}$ , respectively. Then, we also use one 1D convolution to map $\boldsymbol{E}_{d}$ into $\phi\in\mathbb{R}^{H\times\frac{K}{2^{d}}\times T}$. To match the resolution of $\phi$, $\tau$ and $\rho$ are upsampled through interpolation $\mu(\cdot)$, and selective attention weights are generated using a sigmoid function $\sigma(\cdot)$. Finally, the attention weights are multiplied element-wise with $\phi$, and $\mu(\rho)$ is added to the result to obtain $\boldsymbol{L}_{d}\in\mathbb{R}^{H\times\frac{K}{2^{d}}\times T}$. The above process can be expressed as follows:

$$ $\boldsymbol{L}_{d}=f(\mu(\tau),\phi,\mu(\rho)).$ (4) $$

The function of $f$ is defined as follows:

$$ $f(x,y,z)=\sigma(x)\odot y+z,$ (5) $$

where $\odot$ denotes element-wise multiplication.
In this way, $\{\boldsymbol{L}_{d}\in\mathbb{R}^{H\times\frac{K}{2^{d}}\times T}~{}|~{}d=[0,
D]\}$ contains both local and global information, which helps the model better extract the acoustic feature in the audio mixture.

The decoding stage. In the $d$-th layer, where $d\in[0,D-1]$, the input consists of the decoding result from the previous layer $d+1$ (denoted as $\boldsymbol{D}_{d+1}\in\mathbb{R}^{H\times\frac{K}{2^{d+1}}\times T}$) and the output $\boldsymbol{L}_{d}$ from the fusion stage at the $d$-th layer. $\boldsymbol{D}_{d+1}$ is processed through the SA module to produce $\boldsymbol{D}_{d}$. Specifically, $\boldsymbol{D}_{d+1}$ is transformed using two 1D convolutions to obtain $\alpha\in\mathbb{R}^{H\times\frac{K}{2^{d+1}}\times T}$ and $\beta\in\mathbb{R}^{H\times\frac{K}{2^{d+1}}\times T}$, while $\boldsymbol{L}_{d}$ is transformed through a 1D convolution to produce $\gamma\in\mathbb{R}^{H\times\frac{K}{2^{d}}\times T}$. We then compute:

$$ $\boldsymbol{D}_{d}=f(\mu(\alpha),\gamma,\mu(\beta)),$ (6) $$

where $f$ is defined in equation 5. This formulation integrates the decoding result with the output from the fusion stage to generate the next layer of decoded features.
In particular, for the layer where $d=D$, $\boldsymbol{D}_{D}=\boldsymbol{L}_{D}\in\mathbb{R}^{H\times\frac{K}{2^{D}}
\times T}$. For the layer where $d=0$, we use one 1D convolution to restore the hidden dimension $H$ in $\boldsymbol{D}_{0}\in\mathbb{R}^{H\times K\times T}$ to the feature dimension $N$, obtaining $\bar{\boldsymbol{Z}}_{b}\in\mathbb{R}^{N\times K\times T}$ as the output of the MSA module. In the MSA module of the frequency path, the frequency dimension $K$ is considered the processing dimension. In the frame path, the time dimension $T$ is considered the processing dimension.

#### 3.3.2 Full-frequency-frame attention module

In the frequency path, the F^3A module is used to aggregate features across different sub-bands, as shown in Figure 3(b). Given the input $\bar{\boldsymbol{Z}}_{b}$ and the number of attention heads $A$, we first use separate $1\times 1$ 2D convolutional layers with distinct parameters to transform $\bar{\boldsymbol{Z}}_{b}$ into query $\boldsymbol{Q}\in\mathbb{R}^{(A\times E)\times K\times T}$, key $\boldsymbol{K}\in\mathbb{R}^{(A\times E)\times K\times T}$, and value $\boldsymbol{V}\in\mathbb{R}^{(A\times\frac{N}{A})\times K\times T}$.

To obtain the information of full time length on each sub-band and apply self-attention mechanism, frame dimension $T$ and the channel dimension $E$ are merged in order of time step, so we get query $\boldsymbol{Q}_{i}\in\mathbb{R}^{K\times(E\times T)}$ and key $\boldsymbol{K}_{i}\in\mathbb{R}^{K\times(E\times T)}$ for the $i$-th attention head. Similarly, we get value $\boldsymbol{V}_{i}\in\mathbb{R}^{K\times(\frac{N}{A}\times T)}$. $\boldsymbol{K}_{i}$ is transposed and then multiplied with $\boldsymbol{Q}_{i}$ to calculate the attention map of size $K\times K$, which indicates the similarity between each sub-band and acts as the weight information of the frequency context. Then the attention map is multiplied with $\boldsymbol{V}_{i}$ to obtain the output matrix. For the $i$-th attention head, the output $\boldsymbol{O}_{i}\in\mathbb{R}^{K\times(\frac{N}{A}\times T)}$ is calculated as follows:

$$ $\boldsymbol{O}_{i}=\text{Softmax}\left(\frac{\boldsymbol{Q}_{i}\boldsymbol{K}_ {i}^{\text{T}}}{\sqrt{E\times T}}\right)\boldsymbol{V}_{i},$ (7) $$

The output matrix of each attention head is concatenated to get $\boldsymbol{O}\in\mathbb{R}^{K\times(N\times T)}$, and the full-time length is split into $T$ time steps and transformed by 2D convolutional layer, generating the output $\boldsymbol{Z}_{b,f}\in\mathbb{R}^{N\times K\times T}$. The process of the F^3A module in the frame path is similar.

### 3.4 Band restoration module

After going through the separator, the sub-bands need to be converted back to their original width during mask estimation. Specifically, $\boldsymbol{J}\in\mathbb{R}^{N\times K\times T}$ denotes the output of the separator. For the $k$-th sub-band feature $\boldsymbol{J}_{k}\in\mathbb{R}^{N\times T}$ ($k\in[1,K]$), the PReLU activation function and 1D convolutions are used to transform the number of channels to twice the original dimension $2G_{k}$, corresponding to the real and the imaginary part. The complex feature is restored to generate a mask for each sub-band $\boldsymbol{M}_{k}\in\mathbb{C}^{G_{k}\times T}$ using the ReLU activation function. Then they are merged on the frequency dimension to get the mask for the whole band $\boldsymbol{M}\in\mathbb{C}^{F\times T}$. Similar to band-split, the 1D convolutions of different sub-bands do not share parameters.

## 4 EchoSet

To develop models that perform better in daily scenarios, we need a dataset close to the real world. We create EchoSet, a speech separation dataset with various noise and realistic reverberation, based on SoundSpaces 2.0 (Chen et al., ) and Matterport3D (Chang et al., ). An analysis of the dataset is shown in Table 1.

SoundSpaces 2.0 is an audio rendering platform in 3D environments. Given the mesh of a 3D scenario, it can simulate the acoustic effects of any sound captured from microphones. We followed the steps below to generate mixed speech. (1) Choose the scenario. We selected rooms where daily conversations often occur (such as office, living room, bedroom, dining room, etc.) from Matterport3D, a large RGB-D dataset containing 90 diverse multi-floor and multi-room indoor scenes. (2) Define or sample the position. We defined a microphone at a suitable position, like next to a table or sofa, and sampled two sound sources in the same room. (3) Sample the direction. The angle between the microphone and the sound source must be obtuse, meaning that the speaker and listener face each other. (4) Sample the height. The microphone and sound sources were randomly generated at a vertical height of 1.5 m to 1.9 m from the floor, which is about a person’s height. (5) Generate the audio. With SoundSpaces 2.0, mixed audio files were generated based on bidirectional path tracking algorithm (Cao et al., ), which can simulate various effects in the sound propagation process, including reverberation, diffraction, and absorption. Materials of the room wall and the objects were annotated by Matterport3D and considered during the generation of the audio mixture.

Based on the SoundSpaces 2.0 platform and the Matterport 3D scene dataset, we can simulate reverberant audio from different speakers in LibriSpeech (Panayotov et al., ) to build a new dataset, EchoSet. In total, EchoSet includes 20,268 training utterances, 4,604 validation utterances, and 2,650 test utterances. Each utterance lasts for 6 seconds. We mixed the speech of the two speakers at a random overlap ratio and added some noises from WHAM! noise (Wichern et al., ). The two different speakers were mixed with signal-to-distortion ratio (SDR) sampled between -5 dB and 5 dB. The noises were mixed with SDR sampled between -10 dB and 10 dB.

**Table 1: Features of datasets for speech separation.**
| Dataset | Noise | Reverb | Overlapping |
| --- | --- | --- | --- |
| WSJ0-2mix (Hershey et al., ) | $\times$ | $\times$ | Full |
| WHAM! (Wichern et al., ) | $\surd$ | $\times$ | Full |
| WHAMR! (Maciejewski et al., ) | $\surd$ | Only room | Full |
| Libri2Mix (Cosentino et al., ) | $\surd$ | $\times$ | Sparse but fixed |
| LRS2-2Mix (Li et al., ) | $\surd$ | Room and objects (different scenes) | Full |
| EchoSet (ours) | $\surd$ | Room and objects (same scene) | Random |

## 5 Experimental Setup

Dataset. We report the performance of TIGER on EchoSet. For fair comparison with previous speech separation methods (Li et al., ; Wang et al., ; Hu et al., ), we also used two benchmark datasets LRS2-2Mix (Li et al., ) and Libri2Mix (Cosentino et al., ).

To validate the gap between EchoSet and real-world environments, we constructed real-world data by selecting 10 real-world environments and recording audio from 40 speakers from the LibriSpeech (Panayotov et al., ) test set.
The two audio used for mixing were recorded in the same acoustic scene (e.g., the shape and material of the walls and objects in the room) and followed the same mixing method as LRS2-2Mix (Li et al., ).
For more details of these datasets, please refer to Appendix A.

Training and evaluation.
For training, adam optimizer (Kingma & Ba, ) was employed with an initial learning rate of 0.001, adjusted based on validation performance. Evaluation metrics included SDRi and SI-SDRi (Vincent et al., ), with higher values indicating better performance. We report parameters and MAC operations for complexity, which are calculated for one second of audio at 16 kHz. Inference speed was measured on NVIDIA RTX 4090 and Intel Xeon Gold 6326. Detailed training and evaluation configurations can be found in Appendix B and Appendix C. The code will be made publicly available after the paper is accepted.

## 6 Results and Discussion

### 6.1 EchoSet is more close to the real-world data

We trained different models on Libri2Mix, LRS2-2Mix and EchoSet, and then tested them on the data collected in the real world.
The results are presented in Figure 4. Compared to models trained on Libri2Mix and LRS2-2Mix, the models trained on EchoSet produced higher-quality separated speech, confirming that the gap between EchoSet and real-world audio is relatively small.

Figure: Figure 4: Test results of different models on the real-world data. Models were trained on LibriMix, LRS2-2Mix and EchoSet respectively.
Refer to caption: x4.png

### 6.2 Comparisons with state-of-the-art methods

**Table 2: Performance comparison of TIGER and other separation models on Libri2Mix, LRS2-2Mix, and EchoSet. Models are trained and tested on corresponding datasets. Bold denotes the best performance, and underline indicates the second-best.**
| Methods | Libri2Mix | LRS2-2Mix | EchoSet |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| SDRi | SI-SDRi | SDRi | SI-SDRi | SDRi | SI-SDRi |  |
| Conv-TasNet (Luo & Mesgarani, ) | 12.50 | 12.10 | 11.0 | 10.6 | 7.69 | 6.89 |
| DualPathRNN (Luo et al., ) | 11.64 | 11.26 | 13.0 | 12.7 | 5.87 | 5.06 |
| SudoRM-RF1.0x (Tzinis et al., ) | 13.58 | 13.16 | 11.4 | 11.0 | 7.70 | 6.84 |
| A-FRCNN-16 (Hu et al., ) | 16.73 | 16.32 | 13.3 | 13.0 | 9.64 | 8.76 |
| TDANet Large (Li et al., ) | 16.11 | 15.64 | 14.5 | 14.2 | 10.14 | 9.21 |
| BSRNN (Luo & Yu, ) | 17.38 | 16.96 | 14.4 | 14.1 | 12.75 | 12.23 |
| TF-GridNet (Wang et al., ) | 19.56 | 19.24 | 15.7 | 15.4 | 13.73 | 12.85 |
| TIGER (small) | 17.09 | 16.67 | 14.2 | 13.9 | 13.15 | 12.58 |
| TIGER (large) | 18.34 | 17.97 | 15.3 | 15.1 | 14.22 | 13.73 |

**Table 3: Efficiency comparisons of TIGER and other models. GPU Time and CPU Time are recorded in ms, and GPU Memory is recorded in MB.**
| Methods | Paras | MACs | Training | Inference |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| (M) | (G/s) | GPU Time | GPU Memory | CPU Time | GPU Time | GPU Memory |  |
| Conv-TasNet | 5.62 | 7.19 | 92.96 | 1436.94 | 64.21 | 13.17 | 28.78 |
| DualPathRNN | 2.72 | 45.01 | 67.23 | 1813.55 | 723.13 | 30.38 | 298.09 |
| SudoRM-RF1.0x | 2.72 | 4.65 | 118.46 | 1353.43 | 104.32 | 20.66 | 24.42 |
| A-FRCNN-16 | 6.13 | 81.28 | 230.53 | 2925.83 | 478.58 | 82.65 | 163.82 |
| TDANet Large | 2.33 | 9.19 | 263.43 | 4260.36 | 434.44 | 74.27 | 136.96 |
| BSRNN | 25.97 | 98.70 | 258.55 | 1093.11 | 897.27 | 78.27 | 130.24 |
| TF-GridNet | 14.43 | 323.75 | 284.17 | 5432.94 | 2019.60 | 94.30 | 491.73 |
| TIGER (small) | 0.82 | 7.65 | 160.17 | 2049.46 | 351.15 | 42.38 | 122.23 |
| TIGER (large) | 0.82 | 15.27 | 229.23 | 3989.59 | 765.47 | 74.51 | 122.23 |

We compared TIGER with previous SOTA models including Conv-TasNet  (Luo & Mesgarani, ), DualPathRNN (Luo et al., ), SudoRM-RF1.0x (Tzinis et al., ), A-FRCNN-16 (Hu et al., ), TDANet Large (Li et al., ), BSRNN (Luo & Yu, ) and TF-GridNet (Wang et al., ) in terms of performance and efficiency.
TIGER (small) and TIGER (large) denote the models with the number of FFI blocks $B=4$ and $B=8$, respectively.

Separation performance. TIGER obtained competitive separation performance on the three datasets compared with previous SOTA models (see Table 2). On Libri2Mix, which is relatively simple for lack of noise and reverberation, TIGER (large) was second only to the current SOTA model TF-GridNet, with a 6% drop in performance. On LRS2-2Mix, a more complicated dataset with reverberation recorded in different scenes, the drop in performance of TIGER (large) was only 2% compared with TF-GridNet. On EchoSet, the only dataset with the most realistic reverberation among the three, TIGER (large) achieved an SDRi of 14.22 dB, surpassing other existing methods. On this dataset, TIGER (small) also achieved the performance that was only slightly lower than TF-GridNet. From the above experimental results, we can see that the more complex the acoustic scenarios are, the better performance TIGER will produce. Similarly, based on the results in Figure 4, we observed that TIGER also outperforms existing models in real-world test scenarios. This demonstrates that TIGER is applicable to complex real-world acoustic scenarios including diverse noise and reverberation. To visualize the separation result, we present the spectrogram differences between the audio separated by TIGER and TF-GridNet (Appendix H), demonstrating that TIGER is capable of effectively reconstructing both low-frequency and high-frequency features.

Separation effciency. The parameters of TIGER were only 0.82 M, and the MACs were only 7.65 G/s and 15.27 G/s for the small and large versions respectively. Compared with TF-GridNet, the parameters of TIGER (large) dropped by 94.3%, and the MACs were reduced by 95.3%. For inference of one-second audio, TIGER (large) took about 1/3 of the CPU Time and 3/4 of the GPU Time compared with TF-GridNet, demonstrating a significant calculation compression effect.
Besides, TIGER took up less memory during training and inference, making TIGER more suitable for deployment on devices with limited computational resources.
To our knowledge, TIGER is the first neural network-based speech separation model with a parameter less than 1M.

Performance on cinematic sound separation. To demonstrate the generalization ability of TIGER on similar tasks, we migrate TIGER to cinematic sound separation, which aims to extract different audio elements from a film’s soundtrack.
Compared with the previous SOTA model BSRNN, the parameters drop by 97.3% and MACs drop by 77.6%, with a 39.4% increase of SDR in performance. Details of the dataset, training configuration and result analysis can be found in Appendix D. We have provided audio and video demonstrations of the speech and cinematic sound separation. Please refer to the project page(^1^11[https://cslikai.cn/TIGER/](https://cslikai.cn/TIGER/)).

### 6.3 Ablation study

We adopted the small version of TIGER ($B=4$) in the ablation studies. All the models were trained and tested on EchoSet. The training configuration of TIGER and other models was the same.

**Table 4: Comparison of performance and efficiency of models with different band-split schemes.**
| Schemes | Sub-band Number | SDRi | SI-SDRi | Paras | MACs | GPU Time |
| --- | --- | --- | --- | --- | --- | --- |
| (dB) | (dB) | (M) | (G/s) | (ms) |  |  |
| NonSplit | 321 | 11.53 | 11.12 | 0.56 | 40.89 | 72.91 |
| NormalSplit | 47 | 12.94 | 12.37 | 0.82 | 5.29 | 40.78 |
| EvenSplit | 67 | 12.80 | 12.24 | 0.82 | 7.65 | 43.06 |
| LowFreqNarrowSplit (ours) | 67 | 13.15 | 12.58 | 0.82 | 7.65 | 42.38 |

**Table 5: Importance of MSA and F^3A modules on the EchoSet test set.**
| MSA module | F^3A module | SDRi (dB) | SI-SDRi (dB) | Params (M) | MACs (G/s) |
| --- | --- | --- | --- | --- | --- |
| $\times$ | $\surd$ | 7.58 | 7.22 | 0.33 | 2.74 |
| $\surd$ | $\times$ | 12.34 | 11.87 | 0.75 | 4.95 |
| $\surd$ | $\surd$ | 13.15 | 12.58 | 0.82 | 7.65 |

**Table 6: Comparison of performance with different structures to replace MSA module.**
| Replacement structures | SDRi (dB) | SI-SDRi (dB) | Params (M) | MACs (G/s) | GPU Time (ms) |
| --- | --- | --- | --- | --- | --- |
| LSTM | 13.92 | 13.41 | 2.05 | 49.38 | 83.16 |
| Mamba | 13.02 | 12.59 | 1.33 | 21.03 | 59.91 |
| SRU | 12.87 | 12.43 | 1.00 | 20.30 | 53.26 |
| MSA (ours) | 13.15 | 12.58 | 0.82 | 7.65 | 42.38 |

**Table 7: Comparison of performance with different structures to replace F^3A module.**
| Replacement structures | SDRi (dB) | SI-SDRi (dB) | Params (M) | MACs (G/s) |
| --- | --- | --- | --- | --- |
| LSTM | 12.64 | 12.05 | 2.46 | 51.59 |
| Mamba | 12.20 | 11.78 | 1.74 | 25.43 |
| SRU | 12.48 | 11.97 | 1.41 | 22.71 |
| F^3A (ours) | 13.15 | 12.58 | 0.82 | 7.65 |

Band-split schemes. To verify the effectiveness of the band-split method on the speech separation task, we designed several experiments of different band-split schemes (see Table 10 in Appendix). For these experiments, we kept the feature channel $N$ the same. For the model NonSplit that did not adopt band-split, each frequency point was treated as a sub-band and the real and imaginary channels were transformed to the feature dimension $N$. For the other models, the same method as TIGER was used. A detailed description of the different band-split schemes can be found in Appendix E.

Table 4 shows the performance and efficiency of models using different band-split schemes. While adopting band-split increased the number of parameters due to non-shared 1D convolutions, it significantly reduced overall computational costs by decreasing the total number of sub-bands. This approach allowed the model to focus on important frequency bands, low and medium bands for speech since human speech typically ranges from 85 Hz to 1100 Hz (Loizou, ). The LowFreqNarrowSplit scheme offered finer splits in low-frequency bands compared to NormalSplit, resulting in enhanced performance. In contrast, EvenSplit maintained the same number of sub-bands with an even distribution, leading to a drop in SDRi and SI-SDRi compared to LowFreqNarrowSplit, which highlights the effectiveness of band-split in capturing critical frequency information.

Importance of MSA and F^3A modules. We investigated the role of the MSA and F^3A modules in model performance. To this end, we constructed two controlled models, removing each of these modules. As shown in Table 5, removing the MSA module resulted in the worst results, which validates the effectiveness of the MSA module in speech separation, as it fully integrates multi-scale features. The performance also decreased after the F^3A module was removed, indicating that the global integration of time and frequency helps TIGER extract relevant auditory features. Overall, the results show the MSA and F^3A modules play an important role in improving performance.

Furthermore, MSA module and F^3A module can be replaced by other structures for sequence data modeling, such as LSTM (Graves & Graves, ), SRU (Lei et al., ), and Mamba (Gu & Dao, ). We then evaluated the impact of replacing the MSA and F^3A modules with different sequence modeling structures. We first replaced the MSA module in TIGER with LSTM, SRU, and Mamba, with detailed replacement methods provided in Appendix F. The experimental results are shown in Table 6. We observed that the MSA module significantly reduced the computational load while maintaining strong performance. Although LSTM demonstrated better performance in sequence data modeling, the iterative nature of RNN computations resulted in the GPU inference time being twice as long as that of the MSA-based separator. While linear RNN structures like SRU and Mamba sped up inference to some extent, there remained a gap in separation performance and efficiency compared to the MSA module. This highlights the importance of leveraging multi-scale information for both temporal and frequency modeling.

Next, we replaced the self-attention mechanism in the F^3A module with LSTM, SRU, and Mamba to evaluate the effect of different structural replacements. The experimental results are presented in Table 7. We found that the F^3A module produced the best results among the four experiments, mainly because long-range dependencies captured by the self-attention module help enhance the global context of frequency and temporal features.

We also verified the impact of alternating the frequency path and frame path on model performance (Appendix G). The results show that the time-frequency interleaved structure captures acoustic information of audio better than modeling only time or only frequency.

## 7 Conclusion

In this paper, we present TIGER, an efficient time-frequency domain speech separation model with significantly reduced parameters and computational costs. TIGER effectively extracts key acoustic features through frequency band-split, multi-scale and full-frequency-frame modeling. We also introduce the EchoSet dataset that simulates realistic acoustic scenarios. Experiments showed that TIGER outperformed existing SOTA models in complex acoustic environments, with 94.3% fewer parameters and 95.3% less computational costs, and demonstrated good generalization ability in the task of movie audio separation. TIGER provides new ideas for designing lightweight speech separation models suitable for devices with limited resources.

## Appendix A Dataset details

EchoSet. This dataset includes 20268 training utterances, 4604 validation
and 2650 test utterances. The length of each audio is 6 seconds. The target speech was selected from LibriSpeech  (Panayotov et al., ), mixed with SDR ranging from -5 dB to 5 dB. The speech and noise which was sampled from WHAM! were mixed with SDR sampled between -10 dB and 10 dB. This dataset contains realistic reverberation.

LRS2-2Mix (Li et al., ). Each audio in this dataset lasts for 2 seconds, and the training set, validation set and test set are about 11.1, 2.8 and 1.7 hours, respectively. The utterances were selected from the LRS2 (Afouras et al., ) corpus, which consists of video clips acquired through BBC, and were mixed with SDR sampled between -5 dB and 5 dB. Since the audio files were recorded in real acoustic scenarios, LRS2-2Mix contains much noise and reverberation.

Libri2Mix (Cosentino et al., ). Each audio in this dataset lasts for 3 seconds. The target speech for each audio mixture was randomly chosen from LibriSpeech (Panayotov et al., ) (train-100) and combined with a uniformly sampled Loudness Units relative to Full Scale (Series, ) between -25 and -33 dB. This dataset contains no noise or reverberation.

Real-world data. We collected a smaller-scale real-world dataset to test the performance of models trained on different datasets. First, we randomly selected audio from various speakers in the LibriSpeech (Panayotov et al., ) test set, totaling 1.5 hours. All of this audio was clean, without reverberation or noise. The noise data were selected from the WHAM! noise dataset (Wichern et al., ). Next, we recorded the selected audio in 10 real-world environments, primarily consisting of rectangular conference rooms of various sizes and rooms with different shapes (such as fan-shaped and circular conference rooms). Finally, we mixed the audio from different speakers, which moved in different trajectories using SDRs sampled between -5 dB and 5 dB, while the noise was mixed using SDRs sampled between -10 dB and 10 dB.

## Appendix B Training Configuration

In the encoder and decoder, the window and hop size of STFT and iSTFT were set to 640 (40 ms) and 160 (10 ms). We use the Hanning window to mitigate spectrum leakage. According to the Nyquist sampling theorem, the frequency range represented was 0-8 kHz for audio with a sampling rate of 16 kHz. In this way, each frame was represented by 321-dimensional complex spectra, and the frequency resolution was 25 Hz. We adopt the band-split scheme LowFreqNarrowSplit in Table 10. The number of total sub-bands $K$ was 67. For each sub-band, the bandwidth was uniformly transformed into $N=128$. In the separator, the FFI blocks which share parameters were repeated $B=4$ times for the small version and $B=8$ times for the large version. Each MSA module’s features were downsampled for $D=4$ times, and the hidden layer dimension $H$ was set to 256. For F^3A module, the number of attention heads was set to $4$. When calculating the query and key in each head of the F^3A module, the hidden channel $E$ was set to 4.

During training, We used a 3-second audio segment for EchoSet and Libri2Mix, and a 2-second for LRS2-2Mix. We used the maximization of SI-SDR as the training loss (Li et al., ). The maximum training round was 500. We used Adam as the optimizer (Kingma & Ba, ), with the initial learning rate set to 0.001. If the loss on the validation set did not decrease further within 10 consecutive rounds, the learning rate was halved. When the performance on the validation set did not improve further within 20 consecutive rounds, the training was stopped.

## Appendix C Evaluation Configuration

In all experiments, we reported the quality of separated audio on SDRi (Vincent et al., ) and SI-SDRi (Le Roux et al., ):

$$ $\text{SDRi}=\text{SDR}(\bar{\boldsymbol{P}}_{i},\boldsymbol{P}_{i})-\text{SDR} (\boldsymbol{S},\boldsymbol{P}_{i}),$ (8) $$

$$ $\text{SI-SDRi}=\text{SI-SDR}(\bar{\boldsymbol{P}}_{i},\boldsymbol{P}_{i})- \text{SI-SDR}(\boldsymbol{S},\boldsymbol{P}_{i}),$ (9) $$

To measure the complexity of the model, we used parameters and multiply-accumulate operations (MACs) for theoretical analysis.
In the speech separation task, since the audio length is not fixed, we used MACs for separating one-second audio as an indicator for complexity evaluation. We used ptflops 0.7.3(^2^22[https://pypi.org/project/ptflops/0.7.3/](https://pypi.org/project/ptflops/0.7.3/)) to calculate parameters and MACs. For actual evaluation, we performed the backward process (training) and forward process (inference) 1000 times, respectively, on one second of audio at a 16 kHz sampling rate, then took the average to indicate the training and inference speed. We reported the GPU time and GPU memory usage during the training process, as well as the CPU time, GPU time, and GPU memory usage during the inference process. To simulate the limited computational conditions of mobile devices on which the speech separation model is deployed in real-world situations, we fixed the number of threads to 1 when calculating CPU (Intel(R) Xeon(R) Gold 6326) time and only used a single card when calculating GPU (GeForce RTX 4090) time.

## Appendix D Cinematic sound separation task

The cinematic sound separation task (Uhlich et al., ) is to separate different signals from mixed audio, including speech, music and sound effects. We migrated TIGER to cinematic sound separation to test the generalization ability of the model on similar tasks.

We tested TIGER’s performance on the DnR dataset, which consists of three tracks: speech, music, and sound effects. The length of each audio is 60 seconds. Each track does not completely overlap, and the sampling rate is 44.1 kHz. The dataset is composed of 3295 training audio, 440 validation audio, and 652 test audio.

**Table 8: Band-split scheme on DnR**
| Range (Hz) | Width (Hz) | Total sub-band number |
| --- | --- | --- |
| 0-1000 | 50 | 20 |
| 1000-2000 | 100 | 10 |
| 2000-4000 | 250 | 8 |
| 4000-8000 | 500 | 8 |
| 8000-16000 | 1000 | 8 |
| 16000-20000 | 2000 | 2 |
| 20000-44100 | 22100 | 1 |

According to the composition of the mixed audio, the band-split scheme was adjusted as shown in Table 8. Since the frequency of human hearing ranges from 20 Hz to 20000 Hz, there was no need to split the high-frequency band above 20000 Hz. The window size $W$ of STFT was set to 2048, and the stride $J$ was set to 512. The feature dimension was set to $N=132$. In the separator, the FFI blocks were repeated for $B=8$ times. Other settings remained unchanged.

As for the training configuration, in order to improve the speed of the training phase, each 60-second training audio in DnR was segmented using Voice Activity Detection (VAD). Then 3 seconds of audio was randomly sampled from each component to synthesize the mixed audio. The sum of the mean absolute error (MAE) in the frequency domain and the time domain was used as the training loss, which was the same as (Uhlich et al., ):

$$ $\mathcal{L}=\frac{1}{C}\sum_{i=1}^{C=3}|\bar{\boldsymbol{P}}_{i}-\boldsymbol{P }_{i}|+\frac{1}{C}\sum_{i=1}^{C=3}|\text{STFT}(\bar{\boldsymbol{P}}_{i})-\text {STFT}(\boldsymbol{P}_{i})|.$ (10) $$

The maximum training epochs were 500. AdamW was used as the optimizer, and the initial learning rate was set to $lr=0.001$. If the loss on the validation set did not decrease further within 5 consecutive rounds, the learning rate was reduced by half. When the performance on the validation set did not improve further within 10 consecutive rounds, the training process was stopped.

During inference, we employed a sliding window approach, dividing the 60-second audio into 6-second overlapping segments with a 50% overlap, and then reassembling the segments back to their original length after processing.

**Table 9: Comparison of performance and efficiency of cinematic sound separation models on DnR. ‘*’ means the result comes from the original paper of DnR (Petermann et al., ).**
| Structures | Music (dB) | Speech (dB) | Sound Effect (dB) | Paras (M) | MACs (G/s) |
| --- | --- | --- | --- | --- | --- |
| Conv-TasNet* | 0.3 | 8.5 | 2.0 | 5.3 | 19.82 |
| MRX* | 4.2 | 12.3 | 5.7 | 30.51 | 10.59 |
| BSRNN | 5.5 | 13.8 | 1.8 | 52.8 | 18.2 |
| TIGER | 7.4 | 15.5 | 6.5 | 1.40 | 4.07 |

The experimental results are shown in Table 9. TIGER demonstrated outstanding reconstruction performance across the three audio tracks. Specifically, for the tasks of separating music, speech, and sound effects, TIGER achieved SI-SDR scores of 7.4 dB, 15.5 dB, and 6.5 dB, respectively, significantly outperforming BSRNN. This indicates that TIGER has a stronger capacity for capturing audio features.

Moreover, TIGER’s parameters were only 1.40 million, and its computational costs were 4.07 G MACs per second, which kept resource usage at a low level and was very efficient. These results further validated TIGER’s effectiveness in the domain of cinematic sound separation, providing a strong foundation for practical applications.

## Appendix E Details of different band-split schemes

**Table 10: Different frequency band-split schemes and their corresponding frequency ranges, bandwidths, and numbers of sub-bands.**
| Scheme | Range (Hz) | Width (Hz) | Number | Total number |
| --- | --- | --- | --- | --- |
| NonSplit | 0-8000 | 25 | 321 | 321 |
| NormalSplit | 0-1000 | 50 | 20 | 47 |
| 1000-2000 | 100 | 10 |  |  |
| 2000-4000 | 250 | 8 |  |  |
| 4000-8000 | 500 | 8 |  |  |
| 8000 | - | 1 |  |  |
| LowFreqNarrowSplit | 0-1000 | 25 | 40 | 67 |
| 1000-2000 | 100 | 10 |  |  |
| 2000-4000 | 250 | 8 |  |  |
| 4000-8000 | 500 | 8 |  |  |
| 8000 | - | 1 |  |  |
| EvenSplit | 0-6600 | 100 | 66 | 67 |
| 6600-8000 | 1400 | 1 |  |  |

In Table 10, we list several band-split schemes. For datasets of 16 kHz, the full band ranges from 0-8 kHz. Because real-to-complex STFT satisfies the conjugate symmetry, the result can be expressed using only one side. According to the implementation of the torch.stft(^3^33[https://pytorch.org/docs/stable/generated/torch.stft.html](https://pytorch.org/docs/stable/generated/torch.stft.html)), when the window size was set to 640, the encoding dimension was $\lfloor 640/2\rfloor+1=321$.

For the NonSplit scheme, we didn’t apply band-split and kept the original frequency samples 321. The width of each sub-band was 25 Hz. The total sub-band number was 321. We write the mixed audio after STFT as $\boldsymbol{X}\in\mathbb{C}^{F\times T}$. The real and imaginary part of $\boldsymbol{X}$ were treated as two channels and stacked on the channel dimension to obtain feature $\dot{\boldsymbol{X}}\in\mathbb{R}^{2\times F\times T}$. Then a 2D convolutional layer was applied to $\dot{\boldsymbol{X}}$ to expand the channel dimension to $N$. In this way, we got the input $\boldsymbol{Z}\in\mathbb{R}^{N\times K\times T}$ for the separator ($K=F=321$ in this case).

For the NormalSplit scheme, we split finer in the low-frequency part. Specifically, we split 0-1000 Hz by a 50 Hz bandwidth. Since the resolution was 25Hz, 2 frequency samples were treated as one band. The total sub-band number in 0-1000 Hz was 20. Accordingly, $G_{k}=2$ when $k\in[1,20]$. Similarly, we split 1000-2000 Hz by a 100 Hz bandwidth. 4 frequency samples were treated as one sub-band and the total sub-band number in 1000-2000 Hz was 10, i.e. $G_{k}=4$ when $k\in[21,30]$. For 2000-4000 Hz and 4000-8000 Hz, 10 frequency samples and 20 frequency samples were treated as one band, respectively. Therefore $G_{k}=10$ when $k\in[31,38]$ and $G_{k}=20$ when $k\in[39,46]$. Since there were 321 frequency points in total, there was one endpoint left, corresponding to 8000 Hz. Thus $G_{k}=1$ when $k=47$. There were 47 sub-bands in total. When adopting band-split strategy, the real part $\text{Re}(\cdot)$ and imaginary part $\text{Im}(\cdot)$ of the frequency sub-band $\boldsymbol{B}_{k}$ are no longer treated as two channels, but are merged into the frequency dimension. Then we obtain $\dot{\boldsymbol{B}}_{k}\in\mathbb{R}^{2G_{k}\times T}$. Group normalization layers and 1D convolutions are used to map the frequency dimension $2G_{k}$ to the feature dimension $N$, and then $K$ sub-bands are stacked to obtain the input feature $\boldsymbol{Z}\in\mathbb{R}^{N\times K\times T}$ for the separator.

For the LowFreqNarrowSplit scheme, we split the low-frequency area less roughly. In the range of 0-1000 Hz, we split the band by 25 Hz for each sub-band. This way, 1 frequency sample was treated as a sub-band, and the total sub-band number in 0-1000 Hz was 40. Other bands remained the same as NormalSplit. Therefore, we had $G_{k}=1$ when $k\in[1,40]$; $G_{k}=4$ when $k\in[41,50]$; $G_{k}=10$ when $k\in[51,58]$; $G_{k}=20$ when $k\in[59,66]$; $G_{k}=1$ when $k=67$. The implementation kept the same as the NormalSplit scheme.

For EvenSplit, 0-6600 Hz was split evenly by 100 Hz sub-bands. Each sub-band consisted of 4 frequency samples. The remaining part was treated as one sub-band. Accordingly, we had $G_{k}=4$ when $k\in[1,66]$; $G_{k}=57$ when $k=67$. The band-split detail was also the same as the NormalSplit scheme.

## Appendix F Different structures in MSA and F 3 A modules

In the experiments where we replaced the MSA and F^3A modules, we used LSTM (Graves & Graves, ), SRU (Lei et al., ), and Mamba (Gu & Dao, ) as the alternative model structures. When we substituted LSTM for the MSA module, the input $\boldsymbol{Z}_{b}\in\mathbb{R}^{N\times K\times T}$ is first normalized by group normalization. Then we apply a bi-directional LSTM with the hidden size the same as the hidden layer dimension $H$ in the MSA module, generating the hidden feature $\boldsymbol{Z}_{b}^{\prime}\in\mathbb{R}^{2H\times K\times T}$. Next we restore the hidden layer dimension to the input dimension using linear projection. The output of LSTM is $\bar{\boldsymbol{Z}}_{b}\in\mathbb{R}^{N\times K\times T}$. The configuration for SRU was consistent with that of LSTM. For Mamba, since it is a causal model and cannot access future information, we utilized the BMamba layer, as proposed in SPMamba (Li et al., ), to model sequence information bidirectionally, followed by a linear layer to compress the feature channels.

## Appendix G Ablation study: Time-frequency Interleaving

**Table 11: Comparison of performance and efficiency of models with different modeling paths in the FFI block. T-T means the FFI block consists of two frame paths, while F-F means the FFI block consists of two frequency paths.**
| Structure | SDRi | SI-SDRi | Paras | MACs | GPU Time |
| --- | --- | --- | --- | --- | --- |
| (dB) | (dB) | (M) | (G/s) | (ms) |  |
| T-T | 12.91 | 12.32 | 0.82 | 7.75 | 42.72 |
| F-F | 10.57 | 9.69 | 0.82 | 7.53 | 41.39 |
| F-T (ours) | 13.15 | 12.58 | 0.82 | 7.65 | 42.38 |

In the separator of TIGER, we model time and frequency features of the mixed audio alternately. To demonstrate the effect of time-frequency interleaved structure, we tested the performance of F-F and T-T structures. For F-F, we replace the frame path with the frequency path in the FFI blocks. In other words, each FFI block only includes two frequency paths which process the input $\boldsymbol{Z}_{b}\in\mathbb{R}^{N\times K\times T}$ and $\boldsymbol{Z}_{b,f}\in\mathbb{R}^{N\times K\times T}$ in the same way but don’t share parameters. All FFI blocks still share parameters. The implementation is similar for T-T.

According to the result shown in Table 11, compared with only modeling time or only modeling frequency, the time-frequency interleaved structure can better capture time and frequency information of audio, which facilitates improving performance while keeping the model lightweight.

## Appendix H Visualization

In order to intuitively demonstrate the separation performance of TIGER, we provide some examples for visualization, as shown in Figure 5. The following spectrograms show the inference results of TIGER (large) and TF-GridNet on the same audio, and the ground truth. Sample I and II show that TIGER produces finer reconstruction results at high frequencies compared with TF-GridNet. TIGER also has better effects in noise reduction and spectrum leakage prevention, as illustrated in Sample III and IV.

Figure: Figure 5: Comparison of the spectrograms of the ground truth, audio separated by TIGER and by TF-GridNet.
Refer to caption: x5.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The model source code is open, the description of EchoSet is reasonable, but it would be better if the configuration/code of data generation is also open.
soundness: 3
strengths: Overall the paper is well-written and easy to follow. The proposed model architecture looks reasonably nice with the introduction of a new band-split strategy and a new FFI block. Experimental results show that the model not only archieves competitive performance on all datasets, but also it is very lightweight in terms of model sizes, MACs. Its efficiency in training (GPU time & memory) and inference (CPU time, GPU time & memory) looks good, too.
The motivation of creating EchoSet dataset is clear and reasonable with a complete analysis of existing dataset. Experimental results on the real-world data with different models, trained on different datasets respectively, demonstrate its generalization ability.
summary: The paper developed a deep learning model for speech separation; and it not only focuses on improving model performance but model efficiency. The model is lightweight, and is constructed of a new band-split strategy and a new frequency-frame interleaved (FFI) block. Additionally, to reduce the gap between synthetic data and real-world, it introduced a dataset, named EchoSet, with different noise and realistic reverberation. Experimental results demonstrates the effectiveness and efficiency of the proposed model, and the generalization of the dataset to real-world audio.
weaknesses: The FFI block follows a common design of dual-path architecture, it consists of 2 different parts: frequency path and frame path. Each path has two main modules: multi-scale selective attention (MSA) and full-frequency-frame attention (F^3A). While F^3A looks familiar with self-attention mechanism, MSA extracts features through a selective attention mechanism at multiple scales. We may need an ablation study of the MSA architecture with different scales (e.g. 1,2,3,4) to see how it affects model performance.
The evalution on model efficiency is conducted with a fixed audio input length of 1 second. It would be more complete if the evaluation was with different audio input lengths (each audio in EchoSet is 6 seconds, LRS2-2Mix: 2 seconds, Libri2Mix: 3 seconds)

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. What was the loss function?
2. How does selective attention (SA) work?
3. What is the reasoning behind repeating the same FFI block (with tied parameters)?
soundness: 4
strengths: The paper is well written and the model is described in a detailed manner even though there may be room for improvement in the model description for more clarity.

The method applies to many different sampling rate data due to bandsplit RNN based encoder/decoder. The model is also used to perform cinematic audio separation at 44.1 kHz.

The ablations and various comparisons with state-of-the-art on multiple relevant datasets are all appropriate and impressive.
summary: The paper introduces a novel model architecture that has small number of parameters and performs as well as other larger models on various separation tasks.

The model architecture is similar to bandsplit RNN in its encoder/decoder part and in its core part, it has two novel attention-based submodules called MSA and F3A which are applied in the frequency direction and time (frames) direction separately in identical fashion. These modules are repeated a fixed number of times. When repeating, the parameters are tied, so the number of parameters in the model do not increase when the repetition count is increased.

The paper also introduces a new dataset called EchoSet which includes reverberant speech mixtures which improved performance on a "real" dataset. The "real" dataset is not real in the sense of real speech conversations, but it is obtained through playback in a real acoustic environment of individual sources which are mixed later for evaluation.

The performance on real data shown in Figure 4 is much lower compared to the results with artificial mixtures even when training on EchoSet, which shows there is still a lot of room for improvement when applying these models on real data.
weaknesses: Loss function was not mentioned in the main text (or I missed it). Is the loss in (10) in the appendix used for all tasks, or only for cinematic sound separation?

The math in the MSA module description gets a bit hard to follow, so maybe a more detailed Figure that tracks along with mathematical equations would help. How does selective attention (SA) work? It was not described in the paper.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) The authors mentioned that the band-split module is inspired by prior knowledge of music separation. it would be helpful to clarify if the targeted speech separation task has similar prior knowledge and to highlight any key differences or similarities between the band-split module used here and that in the original paper.
2) In Figure 2, the authors mention that "Residual connections are used to retain original features and reduce learning difficulty." Could you please indicate where these residual connections are depicted in the figure?
3) In Figure 4, the authors use a line chart to present the results. This may be misleading because the results on the three datasets are independent of each other. A different visualization style, such as a bar chart, might provide a clearer comparison.
4) What are the differences between TIGER (small) and TIGER (large), why do they have the same model size?
5) In the ablation study, it looks like the proposed modules appear to involve trade-offs among various performance criteria. Could the authors discuss how to balance this tradeoff and what could be the best to consider?
soundness: 3
strengths: 1) The authors' provision of open-source code enables researchers in the field to reproduce and build upon this work.
2) The authors have a detailed ablation study, as it clarifies the impact and effectiveness of each proposed module.
summary: This paper presents an approach named TIGER for speech separation, emphasizing both efficiency and performance. The proposed approach utilizes frequency band-splitting, selective attention modules, and F3A to manage both temporal and frequency information. In addition, the authors introduced a new data set named EchoSet, that considers the realistic environmental factors such as noise, reverberation, and varied speaker overlap ratios. Experimental evaluations reveal that the proposed approach achieves competitive results across multiple benchmarks.
weaknesses: 1) The proposed approach does not demonstrate clear performance advantages over the current SOTA method, with a noticeable performance gap.
2) Although the authors claimed their approach is more lightweight, the comparison is not entirely fair. A comparison with other systems that specifically employ lightweight methods would provide a more accurate assessment of the model's efficiency.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. How would the model deal with the audio with background noise? 
2. Does the method still work when two people speaking in different setup? e.g. one people is much further to the speaker than another. 
3. How should the model accommodate variable number of speakers?
soundness: 4
strengths: 1. Provide a new way of speech separation with very efficient paramater usage. It is very important to the industry and would be very influential 
2. provide a new dataset which provides different content compared to all other similar kinds before.
summary: This paper presents TIGER, an efficient speech separation model with significantly reduced parameters and computational costs, optimized for low-latency applications. Leveraging frequency band segmentation and attention mechanisms, TIGER captures rich contextual information. The authors introduce EchoSet, a realistic dataset for evaluating model robustness in complex environments. TIGER achieves state-of-the-art performance with 94.3% fewer parameters and 95.3% fewer MACs than existing models.

The result is very impressive and the details is rich. They also provide a high quality dataset which would be beneficial to the field.
weaknesses: 1. The model focused on two speaker separation which makes the task much simpler. May need to provide a way to generalize the framework.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In Table 4, how does LowFreqNarrowSplit compare with Mel-split?
How scalable is Tiger for multi-talker separation?
How scalable is Tiger for audio separation beyond speech, as alluded to in Table 9, where music and environmental sound is still a lot harder than speech. What's the path forward?
soundness: 3
strengths: The paper is well-written overall, following a logical organization and making sound arguments. The development of the proposed system is easy to follow. The details of the modules inside the systems are adequately presented. The evaluation results, given the nature of the paper as an empirical study, can seem a bit under-developed, but the major evaluations are made and presented. The ablation studies are important to help readers understand the importance of the MFA and F3A modules. That being said, similar ablation studies can also be performed on the EchoSet, to determine which aspect of the simulation set this new corpus apart from the former ones. 
The paper highlighted the strong performance of the proposed speech separation system and its small footprint in terms of model size.
summary: In this paper, the authors propose a speech separation setup which relies on multi-scale selective attention and interleaved time-frequency attention modules. The design is relatively lightweight, significantly reducing the number of model parameters and consequently, computing resources in inference. They also introduced   a speech separation corpus, EchoSet, that allegedly has more realistic noise and reverberation. They conducted comprehensive empirical evaluations of several comparable models and public data sets.
weaknesses: The comparison between the various corpus is not strictly fair, given the data sets have different sizes. From the data provided by the authors, EchoSet contains 33.78 hrs, whereas LRS2-2Mix has 11.1 hours and Libri2Mix has 58 hrs. Did the authors augment/select the data to ensure that the same amount of data is used in training to evaluate the systems in Table 2?
The argument in Sect. 6.1 is not entirely clear to me. What does data collected in the real world refer to? Is there any description of this data?
To measure the model complexity, model parameters are acceptable, but MACs is inadequate. To go beyond theoretical analysis, the authors can measure the memory usage, power consumption, throughput and latency on server or edge devices. 
Related to the comments above, Sect. 6 can be beefed up with more analysis. For example, does TIGER or models trained with EchoSet perform better in lower SNR/stronger reverb compared to other models? Can we break it down? How does Tiger/EchoSet fare on male-male pair vs. male-female pair, etc.?

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: 1. The assembly of the real world data is somewhat confusing. As I understand
it, the utterances as well as the noise was rerecorded in the real world and then
mixed together at different SDRs. Why did you not play multiple utterances
and noise from different locations in the room at the same time which would
resemble a real world mixture more closely rather than mixing them together
afterwards?
2. Computational cost was measured on a one second input at 16 kHz.
This choice is never motivated and in the context of speech separation illogical.
While there are some previous papers that have done the same [2, 3], neither
of them had any justification for this choice either. The standard for speech
separation would be an input of at least 4 seconds [4, 5]. The reasoning is
that the WSJ0-2Mix, contains utterances with an average length of about 4-5
seconds. The EchoSet, meaning the dataset the paper proposes, uses 6 second
utterances. Why choose a completely different length for the calculation of
speed and memory usage?
3. The sampling rate for the speech separation datasets is only specified
for the LRS2-2Mix as 16 kHz. Is it the same for the other two (Libri2Mix and
Echoset)? If so, why? Typically, 8 kHz is used and while there is nothing to be
said against using 16 kHz, it still makes future comparisons difficult to not also
include 8 kHz data.

References

[2] Kai Li, Runxuan Yang, and Xiaolin Hu. “An efficient encoder-decoder
architecture with top-down attention for speech separation”. In: ICLR.
2023.

[3] Chen Chen et al. “A Neural State-Space Modeling Approach to Efficient
Speech Separation”. In: Proc. INTERSPEECH 2023. 2023, pp. 3784–3788.
doi: 10.21437/Interspeech.2023-696.

[4] Zhong-Qiu Wang et al. TF-GridNet: Integrating Full- and Sub-Band Modeling
for Speech Separation. 2023. arXiv: 2211.12433 [cs.SD].

[5] Cem Subakan et al. Exploring Self-Attention Mechanisms for Speech Separation.
2023. arXiv: 2202.02884 [eess.AS].
soundness: 1
strengths: The paper proposes a novel dataset, the EchoSet, which not only includes noise
and reverberation, but also object occlusion and different materials through
3D environments. The proposed speech separation architecture achieves high
accuracy on this new dataset while having few trainable parameters.
summary: The paper has two main contributions:
1. The TIGER architecture which is a lightweight time-frequency model
focusing on reducing model size while keeping separation accuracy high.
2. The EchoSet dataset for two speaker speech separation which attempts
to resemble real world scenarios more closely than previous datasets by using
3D environments.
weaknesses: There are many issues with the experimental part of the paper, especially the
evaluation of the proposed model architecture.

The first issue is the datasets that were chosen. The main dataset for
speech separation, the WSJ0-2Mix, was not used. Instead, three others were - the proposed EchoSet, the LRS2-2Mix and the Libri2Mix. While choosing
the LRS2-2Mix over WHAMR! is motivated, the main problem concerns the
Libri2Mix. The results of the Libri2Mix reported here do no coincide with
previous reported results. The Conv-TasNet normally reaches 14.7 dB SI-SDRi
while this paper only report 12.1 dB SI-SDRi. Since all of the results reported
in this paper disagree with previous results, I assume that this is caused by
the choice of limiting the sequence length of the Libri2Mix to 3 seconds, as is
mentioned in the appendix. It also appears that this paper only uses utterances
from train-100 and leaves out utterances from train-360 as is used in the original
Libri2Mix. Therefore, this paper significantly changed the dataset of the Libri2Mix
while still calling it Libri2Mix. Since the training set was clearly significantly
altered, the results obtained here are very difficult to interpret because the
normal Libri2Mix contains more and significantly longer training data.

If one wanted to include two speaker speech separation, why not just use the
standard WSJ0-2Mix which basically everyone uses? And how are future papers
meant to compare themselves to the results reported in this paper? Is everyone
now meant to use this altered Libri2Mix? Since the source code is included,
others could run experiments to figure out the accuracy of the TIGER model
on the normal Libri2Mix or the WSJ0-2Mix. But it simply does not make sense
to have others go through said effort instead of having the original paper contain
this basic information.

There are, however, even more aspects of the results that are unusual. The
speed and memory measurements were taken using an input sequence containing
16k elements. This does not represent any of the datasets used. The EchoSet
is reported to use 6 second inputs, meaning it should contain 96k for 16 kHz,
the LRS2-2Mix uses 2 second inputs at 16 kHz, meaning a sequence length of
32k, and the Libri2Mix uses 3 second inputs, meaning a sequence length of 48k
at 16 kHz. Choosing any (or preferably all) of the sequence lengths used in the
datasets would be logical, but showing the results of a much shorter sequence
length is not.

However, even if all results reported in the paper were correct, they would
still not make a convincing case for the TIGER architecture. The biggest
strength of it is its low model size. While the experiments concerning accuracy
as well as computational cost are flawed as described above, the results of the
TIGER architecture are still not very convincing except for its accuracy on
the EchoSet. It is neither particularly fast nor memory efficient and was only
compared to a single other recent SOTA architecture, the TF-GridNet, which
it cannot match in accuracy in 2 of the 3 datasets tested.
While the paper includes an ablation study, it doesn’t answer the most
pressing questions. What kind of performance (in terms of accuracy/speed/memory
usage) would the TF-GridNet have if it shared model weights across its blocks
and reduced some parameters to match the TIGER’s model size? If such a
comparison was made and the TIGER architecture would be more accurate/faster/used
less memory, then there would be a better argument for it. As of now, it just
looks like it sacrifices accuracy for model size.

The paper also fails to mention perhaps the most significant previous work
for speech separation models with few trainable parameters, which is Group
communication [1]. It is incorrectly claimed that the TIGER architecture is
the first speech separation model with less than 1 million parameters, when the
aforementioned paper includes models with less than 100k parameters and is
roughly 4 years old.

The paper also claims that the models trained on the EchoSet perform better
on real world signals than the Libri2Mix and LRS2-2Mix. This might be true,
but is difficult to say with certainty due to the lack of information in the paper.
The real world data that was used is described in the appendix - however,
important details like its sampling rate and duration were left out. The sampling
rate for the speech separation datasets is only specified for the LRS2-2Mix to be
16 kHz but I assume it to be the same for the Libri2Mix and the EchoSet since
the training configuration appendix talks of frequency ranges from 0-8 kHz.
The duration for these datasets, however, is very different - 2 seconds for the
LRS2-2Mix, 3 seconds for the Libri2Mix and 6 seconds for the EchoSet. If the
average duration of the real world data is also closer to 6 seconds, then it would
give the EchoSet an unfair advantage over the other datasets. Again, this might
not be the case, but without clarification of the duration and sampling rate of all
the data used the results given cannot be verified. In fact, the reported results
in Figure 4 are very unexpected. How is the LRS2-2Mix often even worse than
the Libri2Mix (despite using background noise and reverb, same as the real
world data) while the EchoSet is massively better than both? The paper never
attempts to explain this oddity.

References

[1] Yi Luo, Cong Han, and Nima Mesgarani. “Group Communication With
Context Codec for Lightweight Source Separation”. In: IEEE/ACM Transactions
on Audio, Speech, and Language Processing 29 (2020), pp. 1752–1761.

Meta-review: The authors presented a deep learning model for speech separation that emphasizes not only improving performance but also enhancing model efficiency. The proposed model is lightweight and incorporates a novel band-split strategy along with a new frequency-frame interleaved (FFI) block. To bridge the gap between synthetic data and real-world scenarios, the authors introduce a dataset called EchoSet, which includes diverse noise types and realistic reverberation. Experimental results demonstrate the model's efficiency.

The key strengths of the proposed solution is the lightweight architecture that allows deployment on resource-constrained edge devices in practical applications. Moreover, the paper is extremely well written and easy to follow. Details of the modules inside the systems are adequately presented. The novel band-split strategy along with a new frequency-frame interleaved (FFI) block.

 A key weakness of the work is that TIGER demonstrates approximately a 6% and 2% performance gap compared to the current SOTA model. Although the authors' point of view is valid, it would have been good to have results on WSJ0-2Mix.

 This work might boost  future research in lightweight speech separation architecture.

Decision: Accept (Poster)