An efficient encoder-decoder architecture with top-down attention for speech separation
Authors: Kai Li
Runxuan Yang
Xiaolin Hu
Venue: ICLR 2023 poster (2023)
Keywords: encoder-decoder
top-down attention
speech separation

Abstract: Deep neural networks have shown excellent prospects in speech separation tasks. However, obtaining good results while keeping a low model complexity remains challenging in real-world applications. In this paper, we provide a bio-inspired efficient encoder-decoder architecture by mimicking the brain’s top-down attention, called TDANet, with decreased model complexity without sacrificing performance. The top-down attention in TDANet is extracted by the global attention (GA) module and the cascaded local attention (LA) layers. The GA module takes multi-scale acoustic features as input to extract global attention signal, which then modulates features of different scales by direct top-down connections. The LA layers use features of adjacent layers as input to extract the local attention signal, which is used to modulate the lateral input in a top-down manner. On three benchmark datasets, TDANet consistently achieved competitive separation performance to previous state-of-the-art (SOTA) methods with higher efficiency. Specifically, TDANet’s multiply-accumulate operations (MACs) are only 5% of Sepformer, one of the previous SOTA models, and CPU inference time is only 10% of Sepformer. In addition, a large-size version of TDANet obtained SOTA results on three datasets, with MACs still only 10% of Sepformer and the CPU inference time only 24% of Sepformer. Our study suggests that top-down attention can be a more efficient strategy for speech separation.

Published as a conference paper at ICLR 2023
AN EFFICIENT ENCODER-DECODER ARCHITECTURE WITH TOP-DOWN ATTENTION FOR SPEECH SEPARA-
TION
Kai Li', Runxuan Yang* Xiaolin H2.3..3+
1. Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China
2. Tsinghua Laboratory of Brain and Intelligence (THBI), IDG/McGovern Institute for Brain Research, Tsinghua University. Beijing 100084, China
Chinese Institute for Brain Research (CIBR), Beijing 100010, China www.nnguaaeuuu.rr... edu. cr.
{1k21, yangrx20 (amiggxxxx0@mmiiii J@mails tes.ssiigg twsstnningaa.....
xlhuêts sinhttsinggu...m edu.en
ABSTRACT
Deep neural networks have shown excellent prospects in speech separation tasks. However, obtaining good results while keeping a low model complexity remains challenging in real-worid applications. In this paper, we provide bio-inspired efficient encoder-decoder architecture by mimicking the brain's top-down atten- tion, called TeANet, with decreased model complexity without sacrificing per- formance. The top-down attention in TDANet is extracted by the global attention (GA) module and the cascaded local attention (LA) layers. The GA module takes multi-scale acoustic features input to extract global attention signal, which then modulates features of different scales by direct top-down connections. The LA layers use features of adjacent layers as input to extract the local attention signal, which used to modulate the lateral input in top-down manner. On three bench- mark datasets, TDANet consistently achieved competitive separation performance to previous state-of-the-ar (SOTA) methods with higher efficiency. Specifically, TDANet's multiply-accumuut operations (MACs) are only 5% of Sepformer, one of the previous SOTA models, and CPU inference time is only 10% of Sep- former. In addition, large-size version of TDANet obtained SOTA results three datasets, with MACs still only 10% Sepformer and the CPU inference time only 24% of Sepformer. Our study suggests that top-down attention be a more efficient strategy for speech separation.
INTRODUCTION
In cocktail parties, people's communications are inevitably disturbed by various sounds (Bronkhorst) 2015] Cherry 1953), such as environmental noise and extraneous audio signals, potentially affect- ing the quality of communication. Humans can effortlessly perceive the speech signal of target speaker in a cocktail party to improve the accuracy of speech recognition (Haykin & Chen 2005). In speech processing field, the corresponding challenge is to separate different speakers' audios from the mixture audio, known speech separation.
Due to rapid development of deep neural networks (DNNs), DNN-based speech separation methods have significantly improved (Luo & Mesgarani] 2019: Luo et al. 2020 Tzinis et al. 2020 Chen
et al. 2020. (Subakan et al., 2021: Hu et al. 2021]Li & Luo 2022). As in natural language pro- cessing, the SOTA speech separation methods
embracing increasingly complex models to achieve better separation performance, such as DPTNet (Chen et al. 2020) and Sepformer (Subakan
al. 202T. These models typically use multiple transformer layers (Vaswani et al. 2017) cap-
ture longer contextual information, leading to large number of parameters and high computational
"Corresponding author.

---

Published as a conference paper at ICLR 2023
cost and having a hard time deploying to edge devices. We question whether such complexity is always needed in order to improve the separation performance.
Human brain has the ability to process large amounts of sensory information with extremely low energy consumption (Attwell & Laughlin 2001] Howarth et al. 2012). We therefore resort to our brain for inspiration. Numerous neuroscience studies have suggested that in solving the cocktail part problem, the brain relies on a cognitive process called top-down attention (Wood & Cowan 1995 Haykin & Chen 2005; Fernández al., 2015). It enables human focus on task-relevant stimuli and ignore irrelevant distractions. Specifically, top-down attention modulates (enhance or inhibit) cortical sensory responses to different sensory information (Gazzaley et al.) 2005 Johnson & Zatorre, 2005). With neural modulation, the brain is able to focus on speech of interest and ignore others in a multi speak-speaker scenario (Mesgarani Chang. 2012).
We note that encoder-decoder speech separation networks (e.g., SuDORM-RF (Tzinis et al. 2020) and A-FRCNN (Hu et al. (02I)) contain top-down, bottom-up, and lateral connections, similar to the brain's 's hierarchical structure for processing sensory information (Park & Friston 2013). These models mainly simulate the interaction between lower (e.g., Al) and higher (e.g.. A2) sensory areas in primates, neglecting the role of higher cortical areas such frontal cortex and occipital cortex in accomplishing challenging auditory tasks like the cocktail party problem (Bareham et al. 2018 Co- hen et al. 2005). But they provide good frameworks for applying top-down attention mechanisms.
IN.
Top-dem connections
Avenae Poolinx
Bution-p connection:
Lateral csanections
Pooling
Tranafiemer Layar
Sum operator
Avergee
prodict
F2
08. g
DWCanv
Up sample 2
18. 1
F
LA Layer
DWCComy
Upsempie4xx
F
(N. 7
Layer
DWCCony
Up-umppl x
(N.
LA Layer
Ispul
Ohrgut
(4) Facodar
(b) Global attuntiun (GA) madule
Docnder
Figure 1: Main architecture of TDANet. N and T" denote the number of channels and length of features, respectively. By down-sampling times, TDANet contains 5 + 1 features with different temporal resolutions. Here, we set S to 3. The red, blue, and orange arrows indicate bottom-up, top-down, and lateral connections, respectively. (a) The structure of the encoder, where "DWConv' denotes depthwise convolutional layer with a kernel size of 5 and stride size of followed by GLN. (b) The "Up-sample" layer denotes nearest neighbor interpolation.
(c) The structure of decoder, where the LA layer adaptively modulates features of different scales by set of learnable parameters.
In the speech separation process, the encoder and decoder information are not always useful, so we need an automatic method to modulate the features transmitted by the lateral and top-down con- nections. We propose an encoder-decoder architecture equipped with top-down attention for speech separation, which is called TDANet. As shown in Figure 1 TDANet adds a global attention (GA) module to the encoder-decoder architecture, which modulates features different scales in the en- coder top-down through the attention signal obtained from multi-scale features. The modulated fea- tures are gradually restored to high-resolution auditory features through local attention (LA) layers in the top-down decoder. The experimental results demonstrated that TDANet achieved competi- tive separation performance on three datasets (LRS2-2Mix, Libri2Mix (Cosentino et al.) (020), and WHAM! (Wichern et al., 2019)) with far less computational cost. Taking LRS2-2Mixx dataset as
2

---

Published as a conference paper at ICLR 2023
example- when comparing with Sepformer, TDANet's MACs only 5% of it and CPU inference time is only 10% it.
RELATED WORK
In recent years, DNN-based speech separation methods have received widespread attention, and speech separation performance has been substantially improved. Existing DNN-based methods
categorized into two major groups: time-freeee fene-freyeency domain (Hershey et al. [0166] [Chen et al. 2017
Yu al. 2017) and time domain (Luo & Mesgarani 2019 Luo al. 2020, Tzinis et
2020
Chen et al. 2020 Subakan et al., 2021; Hu et al. 2021 Time domain methods obtain better results compared to time-frequecy domain methods they avoid explicit phase estimation. Recently, time domain methods Sepformer (Subakan et al. 2021) and DPTNet (Chen et al. 2020) replace LSTM layers with transformer layers to avoid performance degradation due long-ten term dependencies and to process data in parallel for efficiency. These methods achieve better separation performance, but the number of model parameters and computational cost become larger due to additional transformer layers. Although large-scale neural networks achieve better performance in various tasks such as BERT (Devlin et al. 2019) and DALL-E (Ramesh al. 2022). is also an important topic to design lightweight models that can be deployed to low-resource platforms.
The mncodrr-decoder speech separation model SuDORM-RF (Tzinis et al. 2020) achieves a certain extent of trade-off between model peformance and complexity, but its performance is still far from SOTA methods. Other researchers have designed fully recurrent convolutional neural network A- FRCNN (Hu et al. (202T) with asynchronous update scheme and obtained competitive separation results with a relatively small number of parameters, but the computational complexity still has room improve.
Some earlier works (Xu et al., 2018 Shi et al. 2018 2019) make use of top-down attention in designing speech separation models, but they only consider the projection of attentional signals to the top layer of the multilayer LSTM and without applying attention to lower layers. These approaches may not fully take advantage top-down attention. In addition. these approaches have large gap in model performance and complexity with recent models including SuDORM-HR and A-FRCNN.
For survey on the use of top-down attention in other domains, see Appendix A
3 METHOD
Audio decoder
Audio encoder
Sepuration network
separsted audio
Mixture audio
Audio decoder
SPK supontted andio
Figure 2:
The overall pipeline for the speech separation task. We assume that the mixture audio contains two speakers here. The parameters are shared between the two audio decoders.
3.1 OVERALL PIPELINE
Given audio containing multiple speakers, the speech separation methods aim to extract different speakers' utterances and route them to different output channels. Let y E T 1x7 be multi-speaker time-domain audio signal with the length T:
C
=
Σ
E; + n,
(1)
3

---

Published as a conference paper at ICLR 2023
consisting of C' speaker signals æ, R IxT plus the noise signal R IxT, We expect the separated speeches F; E IR IxT from all speakers to be closer to target speeches T;,
Our proposed method uses the same three-part pipeline as Conv-TasNet (Luo & Mesgarani 2019); an audio encoder, a separation network and an audio decoder. The audio encoder transtorms into
frame-basee N-dimensional embedding sequence E RN with length T", called mixture audio embedding, which can be implemented using a 1-D convolutional layer with kernel size and stride size T./4]. Similar to existing separation methods (Luo & Mesgarani 2019 2018 Luo
eet al. 2020: Subakan et al. 2021 Chen et al. 2020
Li ct al. 2022a). instead of directly estimating the target speech embeddings, we use a ease-asee separation network f(E; 0) generate a set of
masks M = {M, E NR = i,... O} associated with the target speakers. Each target speech
embedding E, E RR-x22 is generated by applying the corresponding mask M, to the mixture audio embedding E:
E, E M,,
(2)
where denotes clement-wisc product. Finally, the target waveform is reconstructed using the target speech embedding E, through an audio decoder, which can be implemented using 1-D transposed convolutional layer with the same kernel and stride sizes as the audio encoder.
Overall there are three main components: (a) encoder (see Section 3.2) (b) global attention (GA) module (see Section 3.3) and (c) decoder (see Section 3.4). The process detail is follows. First, the encoder obtains multi-scale features using the bottom-up connections. Second, the GA module extracts global features the top layer using multi-scale features, and then the glohal features used as attention to modulate features from different scales using top-down connections. Third, the adjacent layer features are used as the input into the decoder to extract the acoustic features through LA layers. This completes one cycle of information processing inside TDANet. Similar to A-FRCNN Hu al. (202T). we cycle TDANet several times and use the output features from the last cycle as the separation network output (see Section 3.5)
3.2 ENCODER OF TDANET
The encoder of TDANet is designed for extracting features at different temporal resolutions. Lower layers have higher temporal resolutions, while higher layers have lower temporal resolutions. The encoder works as follows (see Figure [īa). For a mixture audio embedding E, the encoder processes E in a bottom-up manner step by step. The bottom-up connections in the encoder are implemented by down-sampling layers, which consist of a 1-D convolutional layer followed by a global layer normalization (GLN) (Luo & Mesgarani, 2019) and PReLU. To efficiently expand the perceptual field, these convolutional layers use dilation convolution with kernels with size 5, stride size and dilation size 2 instead of standard convolution aggregate longer contexts. In this way,
S.... 1}, where S denotes the obtain features with different resolutions {F, R Nx number down-sampling.
3.3 GLOBAL ATTENTION (GA) MODULE OF TDANET
The GA module works in two steps:
It receives multi-scale features {Flli I,..., S + 13 as input and computes global feature
m
ER.VV 2
(2) It uses the T as top-down attention to modulate (Fr = 1,.... S + 1}. controlling the redundancy of information passed to the decoder.
Details of the first step in the GA module: Taking multi-scale features (F.ji = 1,..,, S S 1} as T' input, we use the average pooling layers to compress these features in temporal dimension from z
T" for reducing computational cost. The features with different temporal resolutions are fused to 2
Td
into the global feature G E R can be written as:
by a summation operation. The calculation of global feature 2
=
Σ (F.).
(3)
4

---

Published as a conference paper at ICLR 2023
where p()) denotes the average pooling layer. G will be treated as input to the transformer layer to obtain the speaker's 's acoustic patterns from global contexts.
The transformer layer contains two components: multi-head self-attention (MHSA) and feed- forward network (FFN) layers, as shown Figure The MHSA layer is the same as the encoder structure defined in Transformer /Vaswani al. (017). This layer has been widely used in recent speech processing tasks (Chen et al. 2020: Subakan et al. [0211] Lam et (021). First, the position
+ d, G E NR 22 Then, each attention head calculates the dot product attention between all elements of G in dif- ferent feature spaces to direct model focus to different aspects of information. Finally, the output RRN222 22 25 of MHSA is fed into GLN and then connected G through the residual connection to
information d of different frames is added to global feature to get G
G R
obtain G E G
The FFN layer followed by MHSA layer consists of three convolutional layers. First, G is processed in 1 convolu- tional layer with GLN mapped to the 2N- dimensional representation space. Then, the second convolutional layer composed of a depthwise convolutional layer followed by GLN extracts relationship among 2N sequences. Finally, a 1 x 1 convolutional layer followed by GLN compresses the 2.V -imeeenniia features into N-dimensions obtain
G
T. g
(N,
Figure 3: The structure of transformer layer.
GT E 2
24 and also use a residual connection to alleviate the vanishing gradient problem.
Details of the second step in the GA module: We use Gm as top-down atten- tion (see Figure [p)) to modulate F; be- fore adding it to the decoder. Specifically, up-sample Gm along the time dimen- sion using nearest neighbor interpolation to obtain the same time dimension F, and then use the Sigmoid function to ob- tain the attention signal. In this way, the multi-scale semantic information of Gm guide the local features F, to foeus more on feature details that may be lost in the bottom-up path in order to improve quality of separated audios. The modu- lated features are calculated as:
neighbor interpolation
ConviD
Cuns ID
Figure 4: The structure of LA layer in the decoder, where "Conv ID" denotes the 5 x 5 depthwise convo- lutional layer followed by GLN.
F' = (<
(4)
where & denotes nearest neighbor interpolation, a stands for the Sigmoid function and @ denotes dlement-wise product. The modulated features F E N. x are used input the decoder to extract different speakers' features.
The GA module functions as higher cortical areas such as frontal cortex and occipital cortex in the brain which play an important role in accomplishing complicated sensory tasks. Here, it is only designed for top-down modulation performing speech separation tasks (Bareham et al. 2018 Cohen et al. 2005).
3.4 DECODER TDANET
The decoder progressively extracts acoustic features through top-down connections whose com- ponent is the LA layer, as shown in Figure [T.. The LA layer uses small amount of parameters (~ 0.01M) to generate adaptive parameters p and 7 as local top-down attention perform affine trans- formation on features from the current layer to reconstruct fine-grained features. Figure 4 shows detailed architecture of the LA layer. First, LA layer uses N 1-D kernels with length 5 to perform convolution in temporal dimension using features F/41 resulting in E R Nx - Meanwhile,
5

---

Published as a conference paper at ICLR 2023
other identical 1-D convolutional layer followed by GLN and Sigmoid uses F/41 input, resulting in attention signal p E R पी. Consequently,
ha(((FF41)) p ((
(5)
where 11 and h2 denote two different parameters of 1-D convolutional layers followed by GLN. F o and have the same dimensions. The learnable parameters are used to adaptively modulate F!
via a top-down local attention p. This process is formulated by
= F +
After the feature extraction from the highest layer to the lowest layer, the output of the decoder use C' convolutional layers with N kernels with length of and stride of followed by an activation function (ReL.U) to obtain C masks M,, respectively.
3.5 UNFOLDING METHOD OF TDANET
Clearly, the bottom-up, lateral and top-down connections (Figure [ make TDANet a recurrent model. We need to unfold it through time for training and testing. We adopt the unfolding scheme summation connection (SC) as in A-FRCNN (Hu et al. 2021). whole unfolding scheme. We repeat the block 7 times (weight sharing), such that the model's input adds up with each block's output as the next block's 's input.
a single block in the
4 EXPERIMENT CONFIGURATIONS
4.1 DATASET SIMULATIONS
We evaluated TDANet and other existing methods on three datasets: LibriMix, WHAM!, LRS2- 2Mix). The speaker identities of training// /val idation and test are non-intersecting. meaning unseen speaker data from the training phase were used during testing.
Libri2Mix (Cosentino al. 2020). In this dataset, the target speech in each mixture audio was randomly selected from a subset of LibriSpeech's 's train-100 (Panayotov et al. (0155) and mixed with uniformly sampled Loudness Units relative Full Scale (LCFS) (Series) (011)) between -25 and -33 dB. Each mixture audio contains two different speakers and have duration of 3 seconds with 8 kHz sample rate.
WHAM! (Wichern et al. 2019). This dataset acts noisy version of WSJO-2Mix (Hershey et al. 2016). In the WSJO-2Mix dataset, the overlap between two speaker audios in the mixture is 100%, which is too idealistic, and thus model trained on this dataset does not achieve generalization to speeches from a broader range of speakers (Cosentino et al. 2020).. In the WHAM! dataset, speeches were mixed with noise recorded in scenes such cafes, restaurants and bars. The signal-to-noise ratio of the noise was uniformly sampled between -6db and 3db, making it more challenging mixture audios than WSJO-2Mix. Each mixture audio contains two different speakers and have duration of seconds with 8 kHz sample rate.
LRS2-2Mix. The LRS2 dataset Afouras et al. 2018) contains thousands of video clips acquired through BBC. LRS2 contains a large amount of noise and reverberation interference, which is more challenging and closer the actual environment than the WSJO (Garofolo et al.. 1993) and Lib- riSpeech (Panayotov et al. 2015)) corpora. We created a new speech separation dataset, LRS2-2Mix, using the LRS2 corpus, where the training set, validation set and test contain 20000, 5000 and 3000 utterances, respectively. The two different speaker audios from different scenes with 16 kHz sample rate were randomly selected from the LRS2 corpus and were mixed with signal-to-noke ratios sampled between -5 dB and 5 dB. The data were simulated using script consistent with WSJ0-2MixI The length of mixture audios seconds. It publicly available?

---

Published as a conference paper at ICLR 2023
4.2 MODEL AND TRAINING CONFIGURATIONS
We set the kernel size L of the audio encoder and audio decoder in the overall pipeline to 4 ms and stride size [L/4] 1 ms. The number of down- wwwmppiiiingggg was set The number of channels N of all convolutional layers in each layer was set to 512. and the SC method proposed by A-FRCNN (Hu al.] (02T) was used unfold = 16 times at the macro level. For the MHSA layer, we set the number of attention heads to 8, the dimension to 512, and used sine and cosine functions with different frequencies for the position encoding. The mask of the MHSA layer was disabled because our task allows the model to see future information. In addition, we set the number of channels for the three convolutional layers in the FFN layer to (512, 1024, 512), the kernel sizes to (1, 5, 1), the stride sizes to (1, 1, 1), and the bias settings to (False, True, False). To avoid overfitting, set the probability of all dropouts to 0.1.
We trained all models for 500 epochs. The batch size set to I at the utterance level. Our proposed model used the Adam (Kingma & Ba 2015) optimizer with an initial learning rate of 0.001. We used maximization of scale-invariant signal-to-noise ratio (SI-SNR) (Hershey al.) 2016) the training goal (See Appendix for details). Once the best model was not found for 15 successive epochs, we adjusted the learning rate to half of the previous one. Moreover, we stopped training early when the best model was not found for 30 successive epochs. During the training process, we used gradient clipping with maximum L2 norm of avoid gradient explosion. For all experiments, we used 8x GeForce RTX 3080 for training and testing. The PyTorch implementation of our method is publicly available This project is under the MIT license.
4.3 EVALUATION METRICS
In all experiments, we report the results of scale-invariant signal-to-noise ratio improvement (SI- SNRi) (Le Roux et al., 2019) and signal-to-distortion ratio improvement (SDRi) (Vincent et al, 2006) used to measure clarity of separated audios. For measuring model efficiency, we report the processing time consumption per second (real-time factor, RTF) for all models, indicated by "RTF" in the tables throughout this paper. was calculated by processing ten audio tracks of second in length and 16 kHz in sample rate on CPU and GPU (total processing time 10), represented as "CPU RTF" and "GPU RTF", respectively. The numbers were then averaged after running 1000 times. Also, we used the parameter size and the number of MACs to measure the model size, where the MACs were calculated using the open-source tool under the MIT license.
This toolkit is used
5 RESULTS
5.1 ABLATION STUDY
To better understand the effectiveness of top-down attention, we investigated the impacts of two types of attention components in the model for separation performance, including GA module and LA layer. All experimental results were obtained by training and testing on the LRS2-2Min dataset.
Table Comparison over separation performance and efficiency among different types of top-down attention. "V" indicates the where component is used in the TDANet. "x" " indicates the
opposite.
GA module
LA layer
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
CPU RIF (s)
X
10.1
10.5
0.7
2.5
0.65
12,4
12.7
2.3
4,6
0.79
11.9
12.2
0.7
2.6
0.65
13.2
13.5
2.3
4.7
0.79
We first constructed control model that does not contain the GA module and LA layer. It takes the top-layer features of the encoder input to the decoder. Note that excluding the LA layer means
httaass/lolllaaa.ccooooooee////p http:://ggthbbb.cmm///eeee OpCoutee

---

Published as a conference paper at ICLR 2023
This architecture then becomes typical U-Net (Ronneberger jet al. 2013), similar to a basic block used as in SuDORM-RE (Tzinis et al. 2020). From Table | seen that SI-SNRi of this model is only 10.1 dB.
removing the gray box in Figure 4
When only adding the GA module into TDANet, the SI-SNRi increased by 2.3 dB compared to when it absent (Table The GA module contains transformer layer and top-down projections. We are interested in whether the performance improvement came from the transformer layer or the top- down projections. Table | from Appendix C.I demonstrates that the performance improvement is independent of the transformer layer. When only adding the LA layer, we observed that the SI- SNRi increased by 1.8 dB without increasing computational cost. We noticed that the GA module had larger impact on performance improvement. One possible explanation is that the GA module modulates on a wide range of features, while the LA layer only focuses on adjacent ones. When both GA module and LA layer were used, the performance were the best.
More ablation studies and discussions are reported in Appendix C Firstly, we observed that multi- scale features using summation fusion strategy GA module input were efficient for perfor- mance improvement without significantly increasing computational cost (Appendix C.2). Secondly, TDANet obtained the best separation performance when both MHSA and FFN layers were present in the GA module (Appendix C.3). Thirdly, we found that top-down attention is essential in im- proving separation performance (Appendix C.4). Finally, we discussed the possible reasons for the effectiveness of LA and further verified its performance on SuDoRM-RF (Appendix C.4).
5.2 COMPARISONS WITH STATE-OF-THE-ART METHODS
We conducted extensive experiments to quantitatively compare speech separation performance of our proposed Tet with some existing speech separation models on three datasets. The results shown in Table 2 We selected some typical models that have shown good separation per- formance, including -LSTM-TasNe (Luo & Mesgaranil (018)). Conv-TasNet (Luo & Mesgarani 2019), SUDORM-RF (Tzinis et al. 2020). DuhNNNN (Luo al.. 2020). DPTNet (Chen al., 2020 Sepformer (Subakan et al. 2021,, and A-FRCNN (Hu et al. 2021). SuDORM-RF con-
tains two variants with good performance, suffixed with 1.0x and 2.5x, indicating that these variants consist of 16 and 40 blocks, respectively. A-FRCNN has a suffix "16" denoting that the model is expanded 16 times at the macro level. LRS2-2Mix is the newly proposed speech separation dataset in this paper, so none of these models reported results on this dataset.
Separation performance. TDANet obtained competitive separation performance with much lower complexity than previous SOTA models on the three datasets (see Table 2). clearly demonstrating the importance of top-down attention for the encoder-decoder network. On the LRS2-2Mix dataset, TDANet lost only 0.3 dB SI-SNRi in separation performance but with only 8% of number of param- eters compared to Sepformer, one of previous SOTA models. On the other two datasets, TDANet also obtained competitive results compared to previous SOTA models.
Some audio examples from LRS2-2Mix separated by different models are provided in Supplemen- tary Material. In most examples, found that separation results with TDANet sound better than those from other models.
Separation effciency. In addition number of model parameters. MACs as well as training and inference time are also important indicators of model complexity, as models with a small amount of parameters may also have large number of MACs and could be slow to train and infer. By using RTF and MACs as metrics, we evaluated complexity of pervious models (Table We observed that TDANet outperformed the previous SOTA models model complexity. For example, compared to DPTNet and Sepformer, TDANet's MACS are 5% of them, and backward time is 13% and 47% of them, respectively, significantly reducing the time consumption during training. In addition, compared with A-FRCNN and SuDORM-RE RFF-FFF2 2.5x. TDANet's GPU inference time is 19% and 18% of them, GPU training time is 47% and 37% of them, and CPU inference time is 15% and 46% of them, respectively. These results suggest that TDANet can be more easily deployed to low-resource devices.
In addition, verified the performance of TDANet model at low latency (causal) in Appendix D and the results show that TDANet can still obtain better performance than the lightweight models (SuDORM-RF and A-FRCNN).
8

---

Published as a conference paper at ICLR 2023
Table 2: Quantitative comparison between TDANet and other existing models three datasets (test set). "-" indicates that their results have not been reported other papers. "*" denotes that they were not reported in the original paper but were trained and tested using the Asteroid Toolkit (Pariente et al. 2020 The results of DPTNet on Libri2Mix and WHAM! datasets were reported in Yao
al. 2022). but not in the original paper. "f" denotes that the results of Sepformer were trained and Tested using the SraechBrain Toolkit (Ravanelli et al. 2021)
LRS2-2Mix
Libri2Mix
WHAM!
Params
Model [Authors, Years]
SI-SNRi
SDRi
SI-SNRi
SDRi
SI-SNRi
SDRi
(M)
BLSTM-TasNet [Luo et al, 2018]
6.1
6,8*
7.9
8.7
9.8
23.6
Conv-TasNet [I .uo et al, 2019]
10.6*
11.0*
12.2
12.7
12.7
5.6
SuDoRM-RFI 0x [Tzinis al, 2020]
11.0*
11.4*
13.5
14.0
12.9
13.3
2.7
SuDoRM-RF2.55 [Tzinis et al, 2020)
11.3*
11.7*
14.0
14.4
13.7
14.1
DualPathRNN [Luo et al, 2020]
12.7
13.0
16.1
16.6
13.7
14.1
2.7
DPTNet [Chen et al, 2020]
13.3*
13.6
16.7
17.1
14.9
15.3
2.7
A-FRCNN-16 [Hu et al, 2021]
13.0
13.3*
16.7
17.2
14.5
14.8
6.1
Sepformer [Subakan et al, 2021]
13.51
13.8"
16.5
17.0
14.4
15.01
26.0
TDANet (ours)
13.2
13.5
16.9
17.4
14.8
15.0
2.3
TDANet Large (ours)
14.2
14.5
17.4
17.9
15.2
15.4
2.3
Large-size version. For the proposed method, we also constructed a large-size version of NeeNet. namely TDANet Large, by modifying the kernel length of the audio encoder and audio decoder from 4ms to 2ms and the stride size from 2ms 0.5ms. Note that this does change the model size while also increasing complexity (Tiinis et al. 2020 Luo et al. 2020) BBahmaninehhad et al., 2019). TDANet Large obtained SOTA separation performance without significant increase in computational cost on all three datasets (see Table Sableeano and Table (3. TDANet Large can still maintain a small computational complexity compared to other existing lightweight models. For example, the inference time on the GPU is about times faster, and MACs are about 14 times smaller compared to A-FRCNN. We supplement the results for other models (A-FRCNN and Sepformer) with different kernel size and stride size; see Appendix for details.
Table 3: Comparison of inference time and MACs on the LRS2-2Mix test set. The test environment for feedforward RTF (F GPU RTF) and backward RTF (B GPU RTF) Nvidia GeForce RTX 2080 Ti, while the test environment for CPU RTF is Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz, single-threaded.
Model
F GPU RTF (ms)
B GPU RTF (ms)
CPU RTF
MACs (G/s)
BLSTM-TasNet
233.85
654.14
5.9
43.0
Conv-TasNet
15.28
56.91
0.82
10.2
SuDoRM-RF 1.0x
27.86
95.37
4.6
SuDoRM-RF 2.5x
64.70
228.57
1.73
10.1
DualPathRNN
88.79
241.54
8.13
85.3
DPTNet
103.26
689.00
10.49
102.5
A-FRCNN-16
61.16
183.65
5.32
125.3
Sepformer
65.61
184.91
7.55
86.9
TDANet (ours)
11.76
86.56
0.79
4.7
TDANet Large (ours)
23.77
97.92
1.78
9.1
CONCLUSIONS
To fill the gap between the performance of SOTA and lightweight models, we designed an encoder- decoder speech separation network with top-down global attention and local attention, inspired by our brain's 's top-down attention for solving the cocktail party problem. Our extensive experimental results showed that TDANet could significantly reduce model complexity and inference speed while ensuring separation performance.
The TDANet only reflects certain working principles of the auditory system and is not faithful model of the auditory system. For example, requires a bottom-up up encoder and top-down decoder, and is unclear how such components implemented in the brain.
Limitations:

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by the National Key Research and Development Program of China (No. 2021ZD0200301 and the National Natural Science Foundation of China (Nos. 61836014, 62061136001, U19B2034).
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine telligence, 2018.
David Attwell and Simon Laughlin. An energy budget for signaling in the grey matter of the brain. Journal of Cerebral Blood Flow & Metabolism, 21(10):1133-144 2001.
Fahimeh Bahmaninezhad, Shi-Xiong Zhang, Yong Xu, Meng Yu, John HL Hansen, and Dong Yu. A unified framework for speech separation. arXiv preprint arXiv: /9t.:/:99/..7744 2019.
Corinne A Bareham, Stanimira D Georgieva, Mare R Kamke, David Lloyd, Tristan A Bekinschtein, and Jason Mattingley. Role of the right inferior parietal cortex in auditory selective attention: An rtms study. Cortex, 99:30-38 1..30-88,220 2018.
Adelbert W Bronkhorst. The cocktail-party problem revisited: early processing and selection of multi-talker speech. Attention, Perception, Psychophysics, 776)) 2015.
Jingjing Chen, Qirong Mao, and Dong Liu. Dual-path transformer network: direct context-aware
modeling for end-to-end monaural speech separation. In Annual Conference of the International Speech Communication Association (Itterppeech), pp. 2642-2646, 2020.
Blang-Chictl Chen, Yi Yang. Jiang Wang. Wei Xu, and Alan Yuille. Attention to scale: Scale- aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3640-3649, 2016.
Zhuo Chen, Yi Luo, and Nima Mesgarani. Deep attractor network for single-microphone speaker separation. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 246-250. IEEE, 2017.
E Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the Acoustical Society of America, 25(5):975-979, 1953.
Yale E Cohen, Brian E Russ, and Gordon W Gifford III. Auditory processing in the posterior parietal cortex. Behavioral and Cognitive Neuroscience Reviews, 4(3):218-231, 2005.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vin- cent. Librimix: open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.11262 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: pre-training of deep bidirectional transformers for language understanding. Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HI.T), pp. 4171 4111-4186 2019.
Luis Morís Fernández, Maya Visser, Noelia Ventura-Campos, César Ávila, and Salvador Soto- Faraco. Top-down attention regulates the neural expression of audiovisual integration. Neu- rolmage, 119:272-285. 2015.
John Garofolo, David Graff, Doug Paul, and David Pallett. Csr-i (wsjO) complete 1dc93s6a. Web Download. Philadelphia: Linguistic Data Consortium, 83, 1993.
Adam Gazzaley, Jeffrey W Cooney, Kevin McEvoy, Robert T Knight, and Mark D' D'esposito. Top- down enhancement and suppression of the magnitude and speed of neural activity. Journal of Cognitive Nearoscience, 17(3)500-5517, 2005. 2005.
10

---

Published as a conference paper at ICLR 2023
Simon Haykin and Zhe Chen. The cocktail party problem. Neural Computation, 1779)11875-1902, 2005.
John Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: discrimina- tive embeddings for segmentation and separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35. IEEE, 2016.
Clare Howarth, Padraig Gleeson, and David Auwell. Updated energy budgets for neural compu- tation in the neocortex and cerebellum. Journal of Cerebral Blood Flow & Metabolism, 32(7): 1222-1232, 2012.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excitaee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 7132-7141, 2018
Xiaolin Hu, Kai Li, Weiyi Zhang, Yi Luo, Jean-Marie Lemercier, and Timo Gerkmann. Speech separation using an asynchronous fully recurrent convolutional neural network. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 22509-22522, 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1700-4706 2017.
Jennifer A Johnson and Robert Zatorre. Attention to simultaneous unrelated auditory and visual events: behavioral and neural correlates. Cerebral Cortex, 16110))1609-162, 2005.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Sandglasset: a light multi aulti-granularity self-attentive network for time-domain speech separation. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5759-5763. IEEE, 2021.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626 630. IEEE, 2019.
Kai Li and Luo. On the design and training strategies for ran-based online neural speech separa- tion systems. arXiv preprini arXiv:2206.07344 2022.
Kai Li, Xiaolin Hu, and Yi Luo. On the Use of Deep Mask Estimation Module for Neural Source Separation Systems. In Annual Conference of the International Speech Communication Associa- tion (Interspeech), pp. 5328-5332 2022a.
Xiaoling Li, Houjin Chen, Yanfeng Li, and Yahui Peng. Mafusion: Multiscale attention network for infrared and visible image fusion. IEEE Transactions on Instrumentation and Measurement, 71: 1-16, 2022b.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 696-700. IEFF, 2018.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal fime-freyuency magnitude masking for speech separation. JEEE/ACM Transactions on Audio, Speech, and Language processing, 27(8): 1256-1266, 2019.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rn:: efficient long sequence modeling for time-domain single-channel speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 46-50. IEEE, 2020.
Nima Mesgarani and Edward Chang. Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7397):233-
2012.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 520--521001 IEFF, 2015.
11

---

Published as a conference paper at ICLR 2023
Manuel Pariente, Samuele Cornell, Joris Cosentino, Sunit Sivasankaran, Efthymios Tzinis, Jens Heitkaemper, Michel Olvera, Fabian-Rober Stöter, Mathieu Hu, Juan M. Martín-Doñas, David Ditter, Ariel Frank, Antoine Deleforge, and Emmanuel Vincent. Asteroid: the Py PyTocc-basee audio source separation toolkit for researchers. In Anmual Conference of the International Speech Communication Association (Interspeech), 2020.
Hae-Jeong Park and Karl Friston. Structural and functional brain networks: from connections to cognition. Science, (42(61588: 2013.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204006/25 2022.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. Speechbrain: A general-purpose speech toolkit. arXiv preprint (rXiv:2106.06622 2021.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234-241. Springer, 2015.
BS Series. Algorithms to measure audio programme loudness and true-peak audio level. 2011.
Jing Shi, Jiaming Xu, Guangean Liu, Bo Xu, et al. Listen, think and listen again: Capturing top- speech separation. In International Joint Con- ference on Artificial Intelligence (IICAI). pp. 4353-4360, 2018.
down auditory attention for
Jing Shi, Jiaming Xu, and Bo Xu. Which ones are speaking? speaker-infenne model for multi-talker speech separation. In Annual Conference of the International Speech Communication Association (Interspeech), pp. 4609 4613, 2019.
Ashish Sinha and Jose Dolz. Multi-scale self- self-guided attention for medical image segmentation. IEEE journal of biomedical and health informatics, 25(1):121-130, 2020.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 2015.
Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 21-25, IEEE, 2021.
Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo rm-rf: efficient networks for universal audio source separation. In IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. volume 30, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, 1462- 1469, 2006.
Jianfeng Wang and Xiaolin Hu. Convolutional neural networks with gated recurrent connections. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. WHAM!: Extending speech separation to noisy environments. In Annual Conference of the International Speech Communication Association (Interspeech), pp. 1368-138 1372, 2019.
Noelle L Wood and Nelson Cowan. The cocktail party phenomenon revisited: attention and memory in the classic selective listening procedure of cherry (1953). Journal of Experimental Psychology: General, 124(3):243, 1995.
12

---

Published as a conference paper at ICLR 2023
Jiaming Xu, Jing Shi, Guangean Liu, Xiuyi Chen, and Bo Xu. Modeling attention and memory for auditory selection in a cocktail party environment. In Proceedings of the AAA! Conference on Artificial Intelligence, volume 32, 2018.
Zengwei Yao, Wenjie Pei, Fanglin Chen, Guangming Lu, and David Zhang. Stepwise-refining speech separation network via fine-grained encoding in high-order latent domain. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:378-393, 2022.
Dong Yu, Morten Kolbæk. Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independen multi-talker speech separation. In IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245. IEEE, 2017.
A RELATED WORKS IN OTHER FIELDS
To the best of our knowledge, there is no existing method on speech separation that uses top-down attention in an tention (similar to top-down attention in the GA module) in hierarchical models in other domains, e.g., image segmentation Chen al. 2): [Sinaa Dolz (2020) and image fusion et al. (202bb). However, these methods are all forward network structures. Besides, these methods use dif- ferent scale features obtain the corresponding top-down attention respectively and use top-down attention to modulate the features in complicated way. Furthermore, we surveyed local top-down attention approaches (similar to the LA layer) in other domains, such as SENet (Hu et al. 2018). Highway network (Srivastava al. 2013) and GRCNN (Wang & Hu 2021). These methods intro- duce the attention mechanism (gate) to achieve filtering and integration of information flow order obtain better performance. In this paper, focus designing simple neural networks by using multi-scale features as top-down attention to achieve good results with high efficiency.
architecture. Thus, we investigated the use of global top-down
B TRAINING METHOD
We apply the permutation serrutaatonninvariiatt invariant training (PIT) method (Yu et al. 2017) solve the permutation problem (Hershey et al. 2016) order to maximize the scale-invariant signal-to-noise ratio (SI- SNR)
(Le Roux et al. 2019). The SI-SNR for each speaker is defined as:
Atarget |noise
SI-SNR, = 20 x logio
(7)
where
(Di x3 D;
Atarget
Enoise
D;
Atarget.
(8)
|
In the above equation, (.)) denotes the inner product and
|1.112 denotes the I.2-ormm
C ADDITIONAL EXPERIMENTS
C. EFFECT OF TRANSFORMER LAYER IN THE GA MODULE
The GA module contains a transformer layer and top-down projections. We are interested in whether the performance gain is from the transformer layer or the top-down projections. For TDANet with only the GA module, we removed top-down projections from the GA module, keeping only the transformer layer. When the transformer layer is present, we take the output of as the decoder input. When the transformer layer is absent, we the top-layer features of the encoder as input of the decoder. Table 4 shows the experimental results. We found that the presence the transformer layer does not significantly affect separation performance; both configurations yielded poor results compared with the configuration with top-down projections (SI-SNRi 12.4 dB, Table []. These results further demonstrate the importance of top-down global attention.
13

---

Published as a conference paper at ICLR 2023
Table 4: Effect of transformer layer. "TL." denotes transformer layer in the GA module the LRS2-2Mix dataset.
TL
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
CPU RTF (s)
10.3
10.7
2.3
4.6
0.77 0.65
10.1
10.5
0.7
2.5
C.2 EFFECT OF MULTI-SCALE INPUT TO GA MODULE
To examine the influence separation performance between different features input the GA module: multi-scale fused features G and top-layer features Fs+1 (ResNet-style encoder), we experimented with each of them separately. The experimental results are shown in Table We chose G instead of Fs+1 because the former achieved better separation performance (0.4 dB f) without a remarkable increase computational cost. One possible explanation that we added dense connections to the ResNet-style structure to project features at different temporal scales to the top layer, which similar to DenseNet Huang et al. (2017), boosting the back-propagation of gradients and making the network easier to train. Another possible explanation is that, in ResNet-style structure, features may lose details gradually while propagating bottom-up. Using skip connections to project to the top layer enables more efficient use of multi-scal features.
Table 5: Comparison over separation performance and model complexity between different multi- scale fused top-layer features used for input into the GA module on the LRS2-2Mix dataset.
Methods
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
CPU RTF (s)
G Fs+1
13.2
13.5
2.3
4.7
0.79
12.8
13.1
2.3
4.7
0.74
In addition, we investigated two different strategics for multi-scale feature fusion: summation and concatenation. Table shows the results. Summation outperformed concatenation in terms of both separation performance and computational efficiency. Therefore, we employed summation opera- tion the fusion strategy.
Table 6: Comparison separation performance and model complexity between different fusion strategies on the LRS2-2Mix dataset.
Fusion strategy SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
CPU RTF (s)
Summation 13.2
13.5
2.3
4.7
0.79
Concatenation
13.0
13.3
3.6
6.1
0.83
C.3 EFFECT OF TRANSFORMER LAYER COMPONENTS
The transformer layer contains two components: MHSA and FFN layers. We have investigated the effectiveness of MHSA and FFN layers on separation performance and inference efficiency in Table 7 We used the TDANet with both GA module and LA layers the control model. When we remove MHSA and FFN layers from GA, the top-layer features of the encoder are used as top-down global attention. We observed that the these layers played a critical role in improving separation performance. When adding either MHSA FFN layer, observed that the impact of FFN (t 1.3 dB) on performance was more significant than that MHSA (1 0.7 dB). One possible reason may be that the perceptual field of the top-layer features is already large enough that the transformer structure, known to be good at modeling long sequences, does not improve separation performance much.
14

---

Published as a conference paper at ICLR 2023
Table 7: Comparison the importance of MHSA and FFN layers the LRS2-2Mix dataset.
MHSA
FFN
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
CPU RTF (s)
X
X
11.2
11.5
0.2
2.6
0.68
V
11.9
12.2
1.3
3.6
0.73
X
12.5
12.8
1.3
3.7
0.73
V
13.2
13.5
2.3
4.7
0.79
C.4 ADVANTAGES OF TOP-DOWN ATTENTION IN THE GA MODULE AND DECODER
We verify the effect of global top-down attention SuDoRM-RE RF We found that adding down attention to the SuDoRM-RU RR can also improve separation performance, as shown in Table g
Table 8: Comparison on the importance of top-down attention (TDA) on SuDoRM-RE on the LRS2-
2Mix dataset.
Models
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
SuDoRM-RF 1.0x SuDoRM-RF 1.0x + TDA
11.0
11.4
2.7
4.6
11.9
12.3
2.7
4.6
We add LA layers to the decoder of SuDoRM-RF to verify the importance of the LA layer. We found that adding LA layers also improve separation performance, shown in Table
Table 9: Comparison the importance of LA layers on SuDoRM-RE the LRS2-2Mix dataset.
Models
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
SuDoRM-RF 1.0x
11.0
11.4
2.7
4.6
SuDoRM-RF 1.0x + LA
11.7
12.0
3.3
4.9
D TDANET IN LOW LATENCY CASE
For the real-time problem, we tried to modify SuDoRM-RF (Tzinis et al. 2020). A-FRCNN (Hu al. 202I) and TDANet causal versions. We replaced the standard convolutional layers both models with causal convolutional layers to mask out future information (modified from SuDoRM- R We supplemented the experiments with SuDoRM-RF (causal), A-FRCNN-I (causal) and TDANet (causal). The results are shown in Table [O] The results show that modifying the models to causal versions degrade the separation performance to varying degrees (compare Table 2 and Table 10). However, TDANet is still able obtain better performance than SuDoRM-RF and A- FRCNN.
E DIFFERENT SETTINGS OF THE HYPERPARAMETER L
For the hyperparameter L, different models have different settings in their original papers. We compare the results of TDANet and the SOTA model Sepformer and lightweight model A-FRCNN with different values of L. is seen from the Table [T that, for any model, when L is larger, the inference speed is faster, but the separation performance goes down. In addition, for any L value, our proposed TDANet achieves the best results with the highest efficiency.
model s/causal immroved_ssdormeeee p3.pp
15

---

Published as a conference paper at ICLR 2023
Table 10: Comparison on separation performance of different models at low latency on the LRS2-
2Mix dataset.
Models
SI-SNRi (dB)
SDRi (dB)
Params (M)
MACs (G/s)
SuDoRM-RF 2.5x (causal)
4.2
5.1
6.4
10.1
A-FRCNN-IE (causal)
8.9
9.5
6.1
125.3
TDANet (causal)
9.5
10.0
2.3
4.7
Table 11: Comparison on separation performance of models with different L's on the LRS2-2Mix
dataset.
Name
S1-SNRi
SDRi
Params (M)
MACs (G/s)
Inference Time (s)
L 4 ms
A-FRCNN-16
12.5
12.8
6.2
37.3
1.32
Sepformer
12.0
12.3
26.0
23.5
1.89
TDANet
13.2
13.5
2.3
4.7
0.79
ms
A-FRCNN-16
12.9
13.2
6.2
61.5
2.43
Sepformer
12.6
12.9
26.0
49.0
3.13
TDANet
14.2
14.5
2.3
9.1
1.78
L ms
A-FRCNN-16
13.0
13.3
6.1
125.3
5.32
Sepformer
13.5
13.8
26.0
86.9
7.55
TDANet
14.9
15.2
2.3
18.1
2.42
16


Reviews:
[{"review_id": "ct5doRry9r", "reviewer": "Reviewer_88kn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, a lightweight blind source separation model (TDANet) is presented. The model is inspired from the brain’s top-down attention architecture.   The separation is carried out using an embedder, a separation and a decoder (with the  same parameters for all speakers) on the time samples. \nThe model is consisted with down and up sample units, global attention, and transformer to output acoustic features on this information and LA layers in the decoder.\n", "summary_of_the_review": "Major concerns :\n\nThe bio-inspiration in this paper is emphasized many times although it is not clear at all to the reader. A short description can help understand the motivation for this approach. \n\nThe main idea of this paper is to build an efficient model for speech separation. The minimum number of parameters in the proposed method is 2.3 M. It is compared to other method with more parameters. Interestingly, the SuDoRM-RF0.5x and SuDoRM-RF0.25x with less parameters are not mentioned – please explain.\n\nB equals to 16, it is not clear if this is considered in the complexity measurement. \n\nThe authors chose L=4ms and only 3 1d conv layers. The other models are using the same parameters ? if not, it is interesting to understand the contribution of these parameters to the method efficiency and performance.\n\nThe new LRS2-2mix dataset – It seems that the random method can chose for the same scenario 2 reverberated signals from different enclosures. Please elaborate on this.\n \nMinor concerns :\n\nTo make it easier to follow, I think that it will be better to add the parameters names (G_m,F_i for instance) to the architecture figure. \n\nIn the experimental study, only the improvement is reported. To better understand the performance of the model, the noisy signal should also be mentioned as a reference. For instance, if the noisy input’ SI-SDR is -10, 10 dB improvement does not improve the intelligibility so much.\n\nThe SI-SDR measurement reported in the training method (eq. 8,9) is different from the original paper (I guess it is only a typo but if not, we cannot compare the results to the other methods)\n\nThe Sepformer model on the wham! Is reported in their paper with sisdri=16.4 while in this paper it is only 14.4 – please check\n\nIt is not clear what is the frequency sample used (8KHz/16KHz ?)\n\n\n", "strength_and_weaknesses": "Experimental results show that although this model is small it gets competitive results to SOTA models which are larger less efficient. Furthermore, they show that each part of the architecture is essential for the best results", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DGCgjBiIC0", "reviewer": "Reviewer_HQhT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors have proposed a deep learning-based framework for speech separation.\nThe main claim is the effectiveness of the top-down attention mechanism that works across multiple scales to recover the signal.", "summary_of_the_review": "Please see my detailed comments.", "strength_and_weaknesses": "Overall the paper is easy to follow, and the topic addressed is interesting.\n\n- The proposed attention mechanism has similar ideas to the UNet autoencoder and other similar generative models in the area of empirical deep learning that combine evidence from multiple scales. The authors have not discussed such ideas in this paper.\n- There is no theoretical discussion on the impact of individual modules and their interaction for overall performance gains. e.g., why adopt such a complicated pipeline?\nEncoder: is Resnet style model with additional pooling layers at each layer. I see no advantage of this over a regular ResNet style architecture which also combines information from a previous block/layer operating at a different scale.\n- The use of the transformer layer is also not clear. It seems like an unnecessary complication. Transformers are for sequence modeling, but I don't see such a requirement here. Also, the input positional encoding and training of a transformer using masking is very important for its effectiveness.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "02VO50kwEtG", "reviewer": "Reviewer_tcwu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new top-down attention framework for the task of speech separation. The knowledge is from human brain mechanism. The authors show that their method can achieve better separation results with a rather smaller model compared with recent approaches.", "summary_of_the_review": "This paper provides a good trial on applying top-down bottom-up processes with MHSA on the task of speaker separation. The motivation of reducing model complexity is aligned with top-down bottom-up mechanism in human perception. However, the paper merely focus on intuitive design of network modules, without insightful & thorough analysis of the proposed top-down attention solution, especially compared with many related trials in other application fields. It is necessary to add more analysis for a wider interest in the ICLR community instead of solving a specific downstream task of speaker separation. As an efficient solution, it would be better if causal version performance can be shown for the potential of real-time processing. Paper writing should be carefully examined to avoid mistakes.\n\nAlthough there are several flaws as mentioned, this work still proposes a novel architecture based on a clear intuition and achieves SOTA on more challenging scenarios of speaker separation task. It would be inspiring of this work in the field of speech processing. Hence I would suggest a weak accept of this paper to ICLR.", "strength_and_weaknesses": "Strengths: the authors work towards realistic app scenario for the speech separation task, which is a good point. The top-down attention is another good point which can be inspired by human brain knowledge.\n\nWeaknesses: Beyond model size, the computational complexity is also unavoidable for practical uses on low-resource devices. It’s quite curious how the model behaves on the low latency (causal) mode, whether it is potential for a real-time processing speed. The design of TDNet also needs more discussions, seems the current one uses more existing architectures which can be interpreted as a top-down process. It would be better to analyze what the model learns in such a top-down process while previous methods did not. Or more analysis should be provided to show the insightful contribution of such a top-down network design.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster