Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation
Authors: Ui-Hyeop Shin
Sangyoun Lee
Taehan Kim
Hyung-Min Park
Venue: NeurIPS 2024 poster (2024)
Keywords: Speech Separation
Time-Domain Audio Separation
Asymmetric Encoder-Decoder
Discriminative Learning
Transformer
Primary area: speech_and_audio

Abstract: In speech separation, time-domain approaches have successfully replaced the time-frequency domain with latent sequence feature from a learnable encoder. Conventionally, the feature is separated into speaker-specific ones at the final stage of the network. Instead, we propose a more intuitive strategy that separates features earlier by expanding the feature sequence to the number of speakers as an extra dimension. To achieve this, an asymmetric strategy is presented in which the encoder and decoder are partitioned to perform distinct processing in separation tasks. The encoder analyzes features, and the output of the encoder is split into the number of speakers to be separated. The separated sequences are then reconstructed by the weight-shared decoder, which also performs cross-speaker processing. 
Without relying on speaker information, the weight-shared network in the decoder directly learns to discriminate features using a separation objective. In addition, to improve performance, traditional methods have extended the sequence length, leading to the adoption of dual-path models, which handle the much longer sequence effectively by segmenting it into chunks. To address this, we introduce global and local Transformer blocks that can directly handle long sequences more efficiently without chunking and dual-path processing. The experimental results demonstrated that this asymmetric structure is effective and that the combination of proposed global and local Transformer can sufficiently replace the role of inter- and intra-chunk processing in dual-path structure. Finally, the presented model combining both of these achieved state-of-the-art performance with much less computation in various benchmark datasets.

Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation
Hyung-Min Park Department of Electronic Engineering, Sogang University, Seoul, Republic of Korea
Ui-Hyeop Shin
Sangyoun Lee
Taehan Kim
[cmlguuq123,oeesy08822 taeaaan,,,,,,,aakk,,k t ac.kr
https///mgguugg.gggggrrbboooeoee Demo
Abstract
In speech separation, time-domain approaches have successfully replaced the üimeffreqeene domain with latent sequence feature from a learnable encoder. Conventionally, the feature is separated into speaker-specific ones at the final stage of the network. Instead, we propose a more intuitive strategy that separates features earlier by expanding the feature sequence the number of speakers as an extra dimension. To achieve this, an asymmetric strategy is presented which the encoder and decoder are partitioned to perform distinct processing in separation tasks. The encoder analyzes features, and the output of the encoder is split into the number of speakers be separated. The separated sequences then reconstructed by the weight-shared decoder, which also performs cross-speaker processing. Without relying on speaker information, the weight-sharee network the decoder directly learns to discriminate features using separation objective. In addition, to improve performance, traditional methods have extended the sequence length, leading to the adoption of dual- aual-path models, which handle the much longer sequence effectively by segmenting it into chunks. To address this, we introduce global and local Transformer blocks that can directly handle long sequences more efficiently without chunking and dual-path processing. The experimental results demonstrated that this asymmetric structure is effective and that the combination proposed global and local Transformer can sufficiently replace the role of inter- and intra-chunk processing in dual-path structure. Finally, the presented model combining both these achieved state-of-the-as performance with much less computation in various benchmark datasets.
Introduction
For the well-known cocktail party problem 14 3) single channel speech separation 30 has been
improved since the introduction of time-domain audio separation network (TasNet) [46 47 which processes the audio separation in the latent space instead of the short-time Fourier transform (STFT) domain, as shown in Figure Ma). In particular, in most speech separation methods, the process of separating the feature sequence for each speaker is typically positioned at the final stage of the network, as shown in Figure [[b), which we refer to as late split. Therefore, single feature sequence must encode all the information for all speakers be separated. In addition, experimental results have shown that TasNet employing a convolution- based audio encoder/decoderr performs better when the kernel length of the audio encoder is shortened [47 45), which requires modeling long sequence. Indeed, expanding the sequence channel and temporal dimensions is necessarily beneficial, since the separation process must include all the information for all speakers in the feature sequence.
As solution to modeling long sequences, DPRNN [45] was proposed using a dual path model, in which it segments long sequences into chunks and processes in terms of intra-chunk and inter-chunk
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Audio Encoder
Separator
Audio Decodar
He
(a) TasNet structure
Y;
V;
x
Main block
Split
Encoder
Split
Decoder
(b) Conventional separator design using late split
(c) Proposed separator design using an early split
Figure I: Block diagrams of (a) TasNet and separator designs the (b) conventional and proposed networks. The proposed network consists of separation encoder and reconstruction decoder based on weight sharing. After an encoder, separated features are independently processed by a decoder network.
to model the local and global contexts. As a result, due to promising performances in modeling long sequences, many TasNet-based approaches have adopted the dual-path model and repeatedly
achieved state-of-the-art performances in monaural speech separation 66 37
92
57 60
Meanwhile, some studies have tackled the high computational complexity of long sequences in the time domain approach and proposed using multi-scaled sequence models based on the recurrent or They reduced the computations to some extent, however, they still could not show competitive performance compared to the dual-path method.
stacked U-Net structure 70 32 42).
However, most studies have focused on handling long sequences rather than addressing the funda- mental inefficiency of TasNet's late split structure, where a single feature sequence must encode all speaker information without discrimination, creating an information bottleneck. Also, forcing the separator to generate all separated features at once before the audio decoder makes the task difficult and can easily lead to local minima during training. To alleviate this challenge, we propose more intuitive approach: expanding the feature sequence to include a dimension for the number speakers in advance, as early split. By splitting features earlier in the separator (Figure [c)), we adopt an asymmetric strategy where the encoder and decoder perform distinct roles. The encoder process single feature sequence before the split layer, similar to conventional separators. After splitting into multiple sequences, the decoder focuses on capturing discrininative characteristics between features using weight-sharing blocks [4 [1]. By employing this early split with a shared decoder (ESSD) structure, we ease the burden on the separator's 's encoder. This approach aligns with common practices in multi-channel speech processing, where processing is divided into stages. Coarse separation is achieved through spatial filtering [3 85 24), followed by post-enhancement to refine results 80 g 81
Furthermore, dual-path model itself also has redundancy because the segmentation process may increase the amount of computation by twice when the overlap between adjacent chunks is set to 50%. Also, the inter-chunk blocks in the dual-path model are inefficient because their role is mainly to capture the global context. Therefore, we design unit blocks for both global and local processing, integrating them effectively to replace the dual-path model and directly process long sequences without chunking. Both of global and local blocks are based on Transformer block structure (23) where multi-head self-attention (MHSA) module and feed-foward network (FFN) module are stacked. As global Transformer block, we modified MHSA module in Transformer block as an efficient gated attention (EGA) module to mainly capture the global dependency without redundancy. On the other hand, as a local block, the MHSA is replaced with convolutional local attention (CLA) to capture local contexts.
Consequently, we present the Eeparaation-Reconstrutiiiii Transformer (SepReformer) for more effi- cient time-domain separation. Based on the ESSD framework and efficient global and local Trans- former unit block, the SepReformer employs an asymmetric encoder-decoder structure with skip connections based on temporally multi-scaled sequence model. The encoder processes a single feature different temporal resolutions, and each intermediate feature is used for skip connection. The decoder then gradually reconstructs fine-graine grained information from the temporal bottleneck features, focusing on the discriminative characteristics of separated speech with auxiliary loss. To achieve this, the weight-shared decoder is trained to discriminate between the features separated by the encoder. In addition, a cross-speaker block is utilized in the decoder to facilitate information Furthermore, we design unit blocks for both global and local processing, integrating them effectively to replace the dual-path model and directly process long sequences without chunking. The experimental results demonstrated that the ESSD is
interaction between sequences, as described in 15 40

---

effective especially with a small network. Also, comprising the separator network with the proposed global and local blocks can sufficiently replace the inter- and intra-chunk processing in dual-path structure, suggesting effectiveness for long feature sequence processing in speech separation. As consequence, the proposed SepReformer that includes both of these achieved state-of-the-art (SOTA) performance with much less computation than before in various benchmark datasets.
2 Related Works
TasNet Conventional source separation has been performed in the STFT domain 30 36
In the time-frequeny representation. a separator is modeled to estimate mask values or direct output representations. Then, the inverse STFT (iSTFT) is operated for the output representations to obtain separated signals [30 (3). On the other hand, TasNet [46] replaces STFT with a 1D convolutional layer. Based on the encoder representation, mask values or direct output representations 65 59
are obtained in the separator. Then, the output representations are decoded by the audio decoder of ID transposed convolution instead of iSTFT. Also, unlike the STFT, the convolutional encoder turns out to work well in a much shorter kernel size. Therefore, TasNet requires the separator to process the much longer sequences. Therefore, instead of an LSTM-based separator [46), Conv-TasNet [47] is proposed based on temporal convolutional network (TCN) 71 39] to design a separator for longer sequence, showing impressive separation results.
Dual-path model for long sequence After Conv-TasNet, the dual-path model is extensively em- ployed handle long sequences. In the dual-path model, the sequence is segmented into smaller chunks, and the sequence is processed alternately as intra-chunk and inter-chunk, effectively inter- leaving between local and global contexts. This dual-path strategy has shown promising performance in TasNet and has been repeatedly adopted [9] 66 37 38 92 57 60 51 53 34]. Especially, it is shown that, compared to various efficient attention mechanisms 76 35 using the dual-path model with the original self-attention mechanism of Transformer (72) is effective for long sequence [67] in speech separation. However, modeling with the dual-path method can double computational costs due to the 50% overlap between adjacent chunks. The inter-chunk blocks in this model are somewhat redundant since they mainly capture global context. To reduce this redundancy, the quasi-dual-patte network (QDPN) (59 replaces inter-chunk processing with downsampling. Inspired by QDPN, we design EGA and CLA modules to capture the global and local contexts without chunking process. Multi-scale model for efficiency Instead of the dual-path model, based on U-Net structure [61].
some studies have suggested using multi-scaled sequence model 665 49 21
32
4p 7 SuDoRM-RF
model 770 used a stacked U-Net structure to reduce the computational cost. The SuDoRM-RF approach can be regarded Net sub-block. Although SuDoRM-RF reduces the computational cost, it still has the disadvantages of having fixed receptive field size and not considering the global context. More recently, TDANet 442 has efficiently improved performance with top-down attention and unfolding as in A-FCRNN 32
substitution of the TCN block in Conv-TasNe with U-ConyBlock U-
However, these conventional methods with multi-scaled sequences prefer stacked or recurrent struc- tures with U-Net sub-block to improve performance. Instead, we consider a single U-Net architecture and explicitly divide the roles of encoder and decoder as separation and reconstruction.
Discriminative learning Weight-sharing neural networks widely used in modern contrastive learn- including speaker verification [90,
588 17 On the other hand, some studies on speech separation proposed to exploit speaker identity as discriminative information to address the case that the similar voices are mixed [52 51 89] Therefore, we utilized the weight-shared network to reconstruct separated speech by extracting distinct speech representations for corresponding speakers. To separate the mixture, weight-shared decoder directly learns to focus discriminative features without the need for additionally designed, for example, speaker loss using an additional speaker embedding extractor. As result, based discriminative learning, the weight-shared network in the decoder strengthens the dominant speaker's 's components on each separated sequence, respectively.
ing [11 6 25
3 Method
3.1
Overall pipeline
When input mixture x E I the 1D convolution audio followed by GELU activation 28 encodes x to the input representation, as X E(x) E RRxx where F, and T denote the number of convolutional filter of encoder and the number of frames, respectively. The kernel and stride size are
3

---

Separation Encoder
Reconstruction Decoder
B
x 2
x
x By
x
Shared Global Transformer
Shared Local Transformer
Global Transfor Transformer former
Local Transformer
Upsample
Concat, Linear
CS Transformer
Upsample
Concat. + Linear
Down Conv.
Down Conv.
Output ayer
x
Speaker Spilt
T
T/2
T/2
T/2"
T/2
T
Figure 2: The architecture of the separator in the proposed SepReformer. The separator consists of three parts: separation encoder, speaker split module, and reconstruction decoder.
L and H, respectively. Then, the output representations Y, are estimated from the separator and decoded by the audio decoder, expressed as &, = D) E R N S S J. Following the recent works 660 79]. we design the separator to directly map the output signals instead of masking.
3.2
Architecture of separator
The detailed architecture of the separator the proposed SepReformer is illustrated Figure 2 The separator is constructed on the basis of the ESSD framework with separation encoder and
reconstruction decoder in temporally multi-scales U-Net structure.
Separation encoder The input representation is first projected to F dimension by the input layer. The input layer is composed of the linear layer and Layer Normalization (L.N) [1] applied each frame independently. In the encoder, the projected feature sequence is successively downsampled R times from the sequence length of T 7/2 The downsampling is performed by 1D depth-wise convolution (Dconv) layer with a stride of 2 and a kernel size of 5, followed by Batch Normalization (BN) and GELU activation (28). Each encoder stage processes the single sequence feature by Be stacks of global and local Transformer blocks.
Speaker split The encoded features in all stages of the encoder are expanded by the number of speakers J/ to transmit the speaker-wiss features from the encoder to the decoder. Therefore, the speaker split layer placed the middle, and it commonly separates the intermediate encoder features used for skip connections as well as
XF xT/2" Reshape
Poonv,
2JF x T/2"
Pconv, 2JF
Pcony, 2JF
the bottleneck feature. As shown Figure this module consists of
Fx
two linear layers with gated linear unit (GLU) activation [20] Each feature is then normalized by LN and processed by the decoder.
Figure 3: Speaker split module
Reconstruction decoder For temporal reconstruction, the upsampled sequence feature from the previous stage concatenated with the skip connection followed by linear layer. Then, Bp stacks of global and local Transformer blocks process the feature sequences as a weight-sharing network to discriminate between the separated features. By incorporating the separation objective function into the weight-sharing decoder, the network directly learns to capture the discriminative features. Then, the output of the last decoder stage is projected back F, dimension by output layer. The output layer consists of two linear layers with GLU activation.
Cross-speaker (CS) Transformer During the discrimination process by the weight-sharing decoder, speech elements can be mistakenly clustered into other speaker channels. As a result, would be beneficial to attend to each other in order to effectively recover the distorted speech elements. Therefore, improve the interaction of contexts between speakers within the decoder, we incorporate Transformer-bast CS module as in [1] 40). Based on MHSA module without positional encoding. CS block performs an attention operation on speaker dimension while temporal dimension is processed independently. Therefore, the CS block learns to identify the interfering components of the opposing sequences within the same temporal frame. For convenience, we call ESSD with CS as separation-ani ietrrationaad-recoossii recoonsrrttttooonn (SepRe) method.

---

Dropout
Dropout
Pconv,
PConv, F
Trropout
GELU
GCFN
Dconv(3), 3F
Dconv(3), 3F
Dropout
GCFN
PCanv,
LN
Pconv,
Poonv,
LN
Linear,
DC)
MHSA w/ pos.
EGA
PConv,
PConv,
LN
(a) Global Transformer block
(b) Local Transformer block
Figure 4: Block diagrams of global and local Transformer for sequence processing. + and f in EGA denote downsampling with average pooling and upsampling with nearest interpolation. Note that the point-wise convolution (Pconv) layer performs equivalent operation to the lincar layer channel mixing. The hidden dimension of GCFN is to 3F after GLU maintain similar parameter size to the FFN with hidden size of 4F. Therefore, while FFN has parameter size 8F², GCFN has slightly larger size of about 9
3.3 Global and local Transformer for long sequences
Instead of the dual-path model based on chunking, we directly process a long sequence using global and local processing blocks, similar to QDPN (5 or Conformer [27]. In particular, global and local blocks replace inter- and intra-chunk processing, respectively. The design of the blocks follows
29 This structure consists Transformer block structure to ensure structural effectiveness [23 688
86
of two sub-modules: temporal mixing and frame- wise channel mixing. These modules stacked together with pre-norm residual unit [75] (55 and LayerScale [69] to facilitate faster training of deep networks. Also, in all residual units, apply dropout [64] for regularization.
Gated convolutional feed forward network (GCFN) Instead of using the conventional feed- forward network (FFN) 23 722 for channel mixing, improve by incorporating temporal Dconv with small kernel size of and substituting GELU with GLU activation [20) as shown in Figure a). This GCFN effectively process channel features by considering the adjacent frame context. Several studies also have demonstrated the effectiveness of these enhancements in FFN 63
Global Transformer with efficient global attention (EGA) In Figure aa), the global block consists of an EGA module for temporal mixing and GCFN. The EGA module is based on the MHSA with relative positional encoding [19]. However, to reduce the computation and focus global information in the attention layer, the downsampled sequence is processed and upsampled downsampled to T/2R, which is equal the
back. Sequences T/2" at all stages 0 length the bottleneck. To compensate for downsampling, the upsampled features are multiplied by the gate value obtained from an additional branch with linear layer and sigmoid function a., The simple strategy allows the effective capture of global contexts while maintaining local contexts.
R
Local Transformer with convolutional local attention (CLA) For the local block, we design CLA module based on ID temporal convolution with a large kernel of K in Figure ()). Inspired by [27]84), the CLA module first processes the feature with the Pconv layer and GLU activation to facilitate capturing local contexts attentively. After the temporal Dconv, two Pconv layers are used. They have hidden dimension of 2/" and employ BN and GELU activation.
3.4 Boosting discriminative learning by multi-loss
The objective function is given as scale-invariant signal-to-noise ratio (SI-SNR) 62 46 defined as
||,,8 [|,s,; - Sy :7)
C= Σ min 20 log
(I)
where 7, = $/ss,//sss) and denotes 1.2-norm. The clipping value of limits the influence Notably, the output of the decoder stages can be trained for progressive reconstruction the feature sequences are already separated in the decoder stages the
of the best training prediction [89 83
progressive multi-stage strategy [54] 9 88 [6]. In particular, shight-sharing decoder in each stage
5

---

can be trained clearly for discriminative learning with stage-specitt separation objective. This multi- loss strategy is also considered to guide intermediate features in audio separation [33
S 59
60
40.
Therefore, the source signal can be estimated as ,,,r Dr [X O M E RI*N when M, E R.XT estimated with additional output layers for J,r- and the nearest upsampling. denotes element- wise multiplication, and (-) is an auxiliary audio decoder, which is also additional required with additional output layers. Therefore, we can set the auxiliary objective function as
-- E min 20 1000 Ih $3.rll2 )7
(2)
where Y/rr = S,,,-/
Note that, when calculating the output from intermediate features, we opt for masking instead of direct estimation because the temporal resolutions of the feature sequences are deficient. Then, the multi-loss can be set to L = (1 a) + a >x= L R. Moreover, we alternatively calculate the intermediate loss L, using the magnitude values of sj and sy in the STFT domain as it provided more stable training and no actual separated signals are required from the intermediate outputs.
4
Experimental Settings
4.1
Dataset
We evaluated our proposed SepReformer on WSJO-2Mix [30]. WHAM! 82)
WHAMR! [50], and LibriMix [18), which are popular datasets for monaural speech separation. To ensure generality, the mixtures the test set were generated by the speakers that were not seen during training. For all the datasets, networks were trained with 4-s-long segments at a 8-kHz sampling rate while the model processes inputs of varying lengths in the evaluation.
WSJO-2Mix WSJO-2Mix is the most popular dataset to benchmark the monaural speech separation task. It contains 30, 10, and 5 hours for training, validation, and evaluation sets, respectively. Each mixture was artificially generated by randomly selecting different speakers from the corresponding set and mixing them random relative signal-to-noise ratio (SNR) between -5 and 5 dB. WHAM//WHAME
WHAM//WHAMR! is a poisy/noisy-reverbeee dataset. In the WHAM! dataset, speeches were mixed with noise recorded in scenes such as cafes, restaurants, and bars. The noise was added to get mixtures at SNRs uniformly sampled between -6dB and 3dB, making the mixtures more challenging than those in the WSJO-2Mix.
version of the WSJO-2Mix
Libri2Mix In Libri2Mix dataset, the target speech in each mixture was randomly selected from a subset of LibriSpeech's s train-100 [56] for faster training. Each source was mixed with uniformly sampled Loudness Units relative to Full Scale (LUFS) to get mixture at SNR between -25 and -33 dB. We used the clean version as in previous studies [9] 42].
4.2 Training and model configuration
We trained the proposed SepReformer for maximum of 200 epochs with an initial learning rate of 1.0e-3 We used warm-up training scheduler for the first epoch, and then the learning rate decayed by factor of 0.8 if the validation loss did not improve in three consecutive epochs. As optimizer, Adam W
[43 was used with weight decay of 0.01, and gradient clipping with a maximum L2-norm of 5 was applied for stable training. All models were trained with Permutation Invariant Training (PIT) [36]. When the multi-loss in Subsection 3.4 was applied, the a was set to 0.4, and after 100 epochs, it decayed by factor of 0.8 at every five epochs. was set to 30 as in [89]. SI-SNRi and SDRi [73] were used as evaluation metrics. Also, we compared the parameter size and the number of multiply-accumulatt operations (MACs) for 16000 samples. The number of heads in MSHA was commonly set to 8, and the kernel size K in the local block was set to 65. Also, we evaluated our model in various model sizes as follows:
SepReformer- T/B/L: F' 64/ 128/256, F. = 256, L 16, 4. 4
SepReformer-SMM S/MM: F = 64/ F. = 256, L = 8, H 2, R
We used longer encoder length L = 32 in the Large model when evaluating the WHAMR dataset to account for reverberation. Note that we did not train the models multiple times, as the deviations in the results are negligible below the significant digits. All experiments were conducted on a server with GeForee RTX 3090 x 6.

---

Case
MACs
(G/s)
Param.
(M)
SI-SNRi
(dB)
Case
MI.
Param. SISNRi
(M)
(dB)
late split + origin dec.
5.0/18.3
2.8/11.6
19.0/21.6
late split+origin dec.
11.6
21.2
late spiggn dec.
13.2
21.6
late split + large dec.
9.0/33.7
4.9/20.1
19.7/22.0
early split sharedd dec.
11.6
22.4
early split multi dec.
7.9/29.5
4.5/18.4
19.8/22.1
early spait+ dec.
12.2
23.1
early split+ shared dec.
7.9/29.5
2.8/11.6
21.3/23.1
early split+ dec. CS
14.2
22.6
early split seared
dec. CS
10.4/39.8
3.5/14.2
22.4/23.8
early split+ shared dec. CS V
14.8"
23.8
(a) Decoder design.
(b) Effects of multi-loss.
Table Experimental evaluation of SepRe method on the WSJ0-2Mix dataset. MI. denotes the multi-loss. In (a), all the methods were trained with MI,, and the numbers in the left and right of the '/" symbol were obtained for the tiny and base models, respectively. In (b), when MI. was used for training, we indicated the numbers of parameters including the additional output layer for auxiliary output for šj. which were denoted with asterisk Note that additional output layers were not required during inference.
Case
ESSD CS ML Param. (M) SI-SNRi (dB)
Case
ESSD CS ML Param, (M) SI-SNRi (dB)
(origin.)
5.1
15.3
(origin.)
26.0
20.4
5,4
17.5
2
27.1
21.3
5.5
17.8
27.2
22.0
(SepRe)
5.7
19.2
(SepRe)
28.0
21.6
(SepRe)
5.7
19.5
(SepRe)
28.0
22.7
(a) Conv-TasNet with SepRe method.
(b) Sepformer with SepRe method.
Table 2: Application of SepRe to other networks. From the original separator of Conv-TasNet and Sepformer. applied the SepRe method with multi-loss (ML) and evaluated on the WSJO-2Mix dataset.
5
Results
5.1 Ablation studies of SepRe method
Decoder Design In Table [a), evaluated various decoder structures (See Appendix A for detailed structures) to validate the effectiveness of weight-sharing decoder structure. As shown in Table aa), the computations increases about twice by using large decoder in late split or using early split methods. In particular, the model using multiple decoders after an early split yielded performance comparable to that of using a large decoder after late split. In contrast, by sharing a decoder after an early split, the separation result increased significantly, suggesting that the ESSD structure effectively discriminates between the separated features. This impact was more noticeable in the tiny models by showing increase of 1 1.5dB. Applying CS to ESSD) improved the performance especially on the tiny model, leading to the SepRe mechanism. Although changing from a late split structure an ESSD structure can increase computation if the channel size is kept constant, reducing the channel size allows us still achieve better performance. This adjustment significantly improves computational efficiency in relation to performance. This is particularly evident when comparing the base model with late split to the tiny model with the proposed ESSD + CS (SepRe) structure. The latter model achieves higher performance, with an SI-SNRi of 22.4 dB compared to 21.6 6 dB, while using fewer computations and a smaller model size, clearly demonstrating the efficiency of the model architecture.
Effects of multi-loss Furthermore, we experimented with the effects of multi-loss on various decoder structures in Table [bb). Compared to late split, the case with an carly split increased more significantly with multi-loss because an early split structure could be trained with a clearer objective for discriminative learning using an intermediate loss at each stage. In particular, while applying only CS without multi-loss resulted in marginal improvement, combining CS with multi-loss led to substantial gain. The results demonstrated that stage-specific objective functions induce each CS CS-equipped weight-sharing decoder stage to effectively learn simultaneously how to discriminate between and attend to each other. As result, our proposed SepRe method using ESSD and CS significantly improved separation performance by applying stage-specific objective functions and inducing progressive reconstruction of separated feature sequences.
7

---

(Be, BD)
Param,
(M)
SI-SNRi (dB)
Case
Param. SI-SNRi
(M)
(dB)
Case
Param.
(M)
SI-SNRi
(dB)
(1,4)
14.6
23.6
FEN [23]
13.3
23.0
MHSA w/ d.s & u.s
13.9
23.3
(2.
14.2
23.8
FFN w/ Deonv
13.4
23.4
(3, 2)
13.8
22.8
EGA w/o linear gate 13.9
23.2
FFN w/ GLU
14.1
23.3
(4,1)
13.4
21.7
EGA
14.2
23.8
GCFN
14.2
23.8
(a) Depth encoder-decoder.
(b) EGA module design.
(c) FFN module design.
Table 3: Ablation studies for unit blocks SepReformer-B the WSJ0-2Mix dataset. Various configura-
tions of Bg and BD were evaluated assess relative importance of encoder and decoder. Also, we validated the proposed EGA and GCFN modules.
Separator
Long sequence model
Param. (M)
MACs (G/s)
SI-SNRi (dB)
Nen 47
TCN [71
5.1
10.5
13.6
DPRNN
Duth-path- BLSTM
2.6
88.5
18.8
SuDoRM-RF 770
Multi-scale + Convolution
10.1
18.9
Sepformer
66
Dual-path + Transformer
26.0
86.9
20.4
TDANet 42
Multi-scale + Transformer
2.3
9.1
18.5
MossFormer(S)
92
GAU [33
10.8
44.0
20.9
S4M 77
Multi-scale SSM 26
3.6
38.4
20.5
Ours
Global-Local Translormer
11.9
43.1
21.3
Ours + U-Net
Multi-scale + Global-Lock Transformer
11.6
18.3
21.2
Table 4: Comparison with various long sequence models in speech separation of WSJ0-2Mix. MS denotes multi-scale. For model, global and local blocks repeated 22 times with F 128.
5.2 Effects the SepRe method other networks
To validate the general applicability of the SepRe method, we incorporated the SepRe method with multi-loss into the original separators of Conv-TasNe [47] and Sepformer [66] and conducted experi- ments on WSJO-2MIX. The experimental results in Table demonstrated a significant performance improvement when ESSD was applied for both networks. Also, applying CS and multi-loss in addition to the ESSD framework improved the performance further, which confirms the effectiveness of SepRe with loss.-loss.
5.3
Ablation studies of unit blocks
Depth of encoder-decoder In Table 3ja, we experimented the depth of encoder and decoder to determine the optimal configuration terms of the number block repetition Bg and Bp. Generally, experimental results showed that using more blocks in the reconstruction decoder had a greater impact on performance improvement than in the separation encoder. demonstrate that it is more important to discriminate the features more elaborately in weight-sharing decoder than to analyze the features in encoder in speech separation. In particular, optimal performance was achieved with Bg 2 and Bn 3, which were used the common configuration for subsequent experiments.
EGA module design Next, we validated our proposed EGA module by ablating its components (see Appendix B for detailed structures) in Table (().. First of all, using vanilla MHSA on a long sequence without chunking was infeasible due the extremely large computational requirements. Therefore, one approach was to perform downsampling before applying MHSA, similar to the method used QDPN [59]. However, this naive approach had the drawback of losing detailed frame-wise wise information. Although another consideration was to simply multiply the features to reflect the fine- grained frame-wise information, this method still could not significantly improve performance. In contrast, the optimal performance a sigmoid function. As a result, it is shown that our proposed global Transformer with EGA module and local Transformer with CLA module have effectively replaced conventional sequence models with smaller computations.
achieved by estimating gate values based on a linear layer and
FEN module design Also, by improving the design of FFN with Dcony and GLU activation, could achieve the significant improvement of performance with slight increase of parameters as shown in Table
Comparison with various long sequence models Table we evaluated the network by stacking proposed global-local Transformer blocks to assess the performance of modeling a long sequence.

---

WSJ0-2Mix
WHAM!
Libri2Mix
Params. (M)
MACs (G/s)
System
SI-SNRi
SDRi
SI-SNRi
SDRi
SI-SNRi
SDRi
(dB)
(dB)
(dB)
(dB)
(dB)
(dB)
Conv-TasNNet 47
5.1
10.5
15.3
15.6
12.7
12.2
12.7
SuDoRM RF
70
6.4
10.1
18.9
13.7
14.1
14.0
14.4
TDANet
18.5
18.7
15.2
15.4
17.4
17.9
Sandglasset
38
2.3
28.8
20.8
21.0
S4M 77
3.6
38.4
20.5
20.7
16.9
17.4
SepReformer-1
3.5
10.4
22.4
22.6
17.2
17.5
19.7
20.2
SepReformer-S
4.3
21.3
23.0
23.1
17.3
17.7
20.6
21.0
DPRNN
45
2.6
88.5
18.8
19.0
13.7
14.1
16.1
16.6
DPTNet
102.5
20.2
20.3
14.9
15.3
17.1
Sepformer
26.0
86.9
20.4
20.5
14.7
16.8
16.5
17.0
WaveSplit
89
29.0
21.0
21.2
16.0
16.5
16.6
17.2
A-FRCNN
32
6.1
125.0
18.3
18.6
14.5
14.8
16.7
17.2
SFSRNet
60
59.0
124.2
22.0
22.1
ISCIT 51
58.4
252.2
22.4
22.5
16.4
16.8
QDPN
59
200.0
22.1
TF-GridNet 79
14.5
460.8
23.5
23.6
SepReformer-B
14.2
39.8
23.8
23.9
17.6
18.0
21.6 22.0
21.9
SepReformer-
17.3
81.3
24.2
24.4
17.8
18.1
22.2
(a) Comparison SepReformer existing models.
WSJ0-2Mix
WHAM!
WHAMR!
Params.
MACs
System
SI-SNRi (dR)
SDRi (dR)
SI-SNRi
SDRi (dR)
SI-SNRi
SDRi
(M)
(G/s)
(dR)
(dR)
(dR)
Sepformer
67
26.0
86.9
22.3
22.5
16.4
16.7
14.0
13.0
WaveSplit
29.0
21.0
21.2
13.2
12.2
SFSRNet
60
59.0
466.2
24.0
24.1
ISCIT
51
58.4
252.2
24.3
24.4
16.9
17.2
QDPN
50
200.0
23.6
14.4
Mossformer(L)
42.1
22.8
17.3
16.3
(ossformer2(L)
X
55.7
24.1
18.1
17.0
Separate And Diffuse
48
23.9
ZSepReformer-I.
59.4
155.5
25.4
25.6
18.4
18.7
17.2
16.0
(b) Comparison of SepReformer-L. to existing large models with DM.
Table Evaluation on various benchmark dataset of WSJO-2MIX, WHAM!, WHAMR! and Libri2Mix. "1" denotes networks use additional speaker information.
Note that did not apply the ESSD structure and multi-loss to separator in this experiment. could observe that the network based on aual-path sequence models requires high computation resources in terms of MACs while multi-scale sequence models more efficient. The recently proposed Mossformer based on efficient gate attention unit (GAU) mechanism [33] showed improved performance with relatively smaller computations compared the networks with dual-path model. In particular, the proposed model showed improved separation performance with similar MACs, which demonstrated the capacity can sufficiently replace the dual-path models with fewer computations. Furthermore, by combining the U-Net structure into global-local Transformer blocks, the network became more efficient with the similar separation performance.
model for long sequence. It also suggested that the proposed block
5.4 Comparison with existing models
Finally, we compared our SepReformer models with existing separation models on various benchmark datasets in Table 5 Although we evaluated SepReformer trained with standard pairs from the training set in Table 5la), SepReformer-L. was trained with dynamic mixing (DM) 889 66] for data augmentation and compared to other existing large models with DM in Table (tb). When traind with DM, we set an initial learning rate of 2.0e and fixed during first 50 epoch. Table a), with almost the smallest computational loads in terms of MACs, our tiny model showed the best performance except for TF-GrrdNett in the WSI0-2Mix dataset which was powerful model recently proposed. It demonstrated the efficiency of the SepRe method in speech separation. Also, SepReformer-M

---

26
SepReformer-L.
DM
25
Scpieffrmee-B
SepReformer-M
ISCIT (large)
DM Net DM
24
SepReforner-
SI-SNRI (dB) on WSJ0-2MM MM-
23
SepReformer-T
Mossformen(() + DM
Tr-GNeeNet
Sepformer
Mossformert
21
Sandglassel
DP
ISCTT
S4M-tiy
S4M
DPRNN A-ERCNN
SuDORM-RE
TDANet
16
Conv-TasNet
10
20
50 MACs (GHS)
Figure 5: Si-SNRi results on WSJO-2Mix versus MACs (G/s) for the conventional methods and the proposed SepReformer. The check mark the circle indicates the use of DM method training. The radius circle proportional to the parameter size of the networks.
without DM in Table 5 a) showed competitive separation performance on WSJO-2Mix compared the large models with data augmentation in Table(5(b). In partieular, SepReformer-L with DM achieved a state-of-fe sific-of-thhe ff-the-art art performance of 252544 dB SI-SNRi on the WSJO-2Mix dataset, demonstrating significant improvement over other conventional methods.
In Table 5 a), the smallest SepReformer-1 T among the proposed models even showed significant im- provements WHAM! and Libri2Mix datasets compared to the conventional methods. It suggested that the proposed SepRe method can efficiently applied to a speech separation task in general. WHAM! and WHAMR! datasets,
Also, SepReformer-L with DM showed the SOTA performance as well as WSJO-2Mix, which demonstrated that the proposed method can be trained effectively in large model. Figure compares the separation performance of various existing methods in terms of SI-SNRi versus MACs on WSJO-2MIX. From the figure, we can observe the significant effectiveness of the proposed SepReformer in the speech separation task with high computational efficiency. Especially, it is noteworthy that the SepReformer-1 T models outperformed the conventional Sepformer trained with DM with 10 times smaller computations.
Conclusion
this work, we introduced the SepRe method, in which the asymmetric encoder and decoder perform separation and reconstruction, respectively. The encoder analyzes and separates a feature sequence, and the separated sequences are reconstructed by a weight-sharing network and a cross-speaker network. We demonstrated that the SepRe method can be applied to conventional separators in general and utilizing multi-loss significantly improves the performance. Moreover, we replaced the dual-path model with presented global and local Transformer blocks to address long sequence. The separator using the presented unit blocks has shown enhanced separated results efficiently, and combining U-Net structure exploit the multi-scale sequence model has further increased the efficiency. Finally, not only did our presented SepReformer outperform the most conventional methods speech separation even with almost the smallest computational resources, but our large models achieved SOTA performance with large margins compared to the conventional models on various speech separation datasets.
Limitations and future work. Our study focuses on 2-speaker mixture situation to assess our models in various model sizes and in the extensive datasets including noise and reverberation. Consequently, we believe that further investigation is needed to validate for more than 2-speaker mixture scenarios. Also, an important future direction is to separate mixtures for an unknown number of speakers it is impractical to assume that the number of speakers to be separated is known in advance. Finally, although we experimentally validated our SepRe method, we believe that further investigation is necessary to figure out its underlying mechanism.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-202--122209992222 00989), Development of Artificial Intelligence Technology for Multi-speaker Dialog Modeling), and in part by the National Research Foundation of Korea (NRF) and the Commercializztioon Promotion Agency for R&D Outcomes (COMPA) grant funded by the Korea government (MSIT) (RS-2023- 00237117).
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[2] Iz Beltagy. Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020.
[3] Albert S Bregman. Auditory scene analysis: The perceptual organization of sound. MIT press, 1994.
[4] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume Morgan-Kaufmann Kaufmann, 1993.
[5] Jaeuk Byun and Jong Won Shin. Monaural speech separation using speaker embedding from preliminary separation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2753 2763, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning visual features by contrasting cluster assignments. In Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol- ume 33, pages 9912 9924, Curran Associates, Inc., 2020.
[7] Chen Chen, Chao-Han Huck Yang, Kai Li, Yuchen Hu, Pin-Jui Ku, and Eng Siong Chng. A neural state-space model approach to efficient speech separation, 2023.
[8] Hangting Chen, Yi Yang, Feng Dang, and Pengyuan Zhang. Beam-Guided TasNet: An Iterative Speech Separation Framework with Multi-Channel Output. In Proc. Interspeech 2022, pages 866-870, 2022.
191 Jingjing Chen, Qirong Mao, and Dong Liu. Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation. In Proc. Interspeech 2020, pages 2642-2646, 2020.
[10] Sanyuan Chen, Yu Wu, Zhuo Chen, Jian Wu, Jinyu Li. Takuya Yoshioka. Chengyi Wang. Shujie Liu, and Ming Zhou. Continuous speech separation with conformer. ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5749-5753, 2021.
[11] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-160777 PMLR, 13 18 Jul 2020.
[12] Zhuo Chen, Yi Luo, and Nima Mesgarani. Deep attractor network for single-microphone speaker separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 346-250. IEEE Press, 2017.
[13] Zhuo Chen, Takuya Yoshioka, Liang Lu, Tianyan Zhou, Zhong Meng, Yi Luo, Jian Wu, Xiong Xiao, and Jinyu Li. Continuous speech separation: Dataset and analysis. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7284-7288. 2020.
[14] E. Colin Cherry. Some Experiments on the Recognition of Speech, with One and with Two Ears. The Journal of the Acoustical Society of America, 25(5):975-979, 06 2005.
[15] Srikanth Raj Chetupalli and Emanuël Habets. Speech Separation for an Unknown Number of Speakers Using Transformers With Encoder-Decode Attractors. In Proc. Interspeech 2022, pages 5393-5397, 2022.
[16] Sung-Jin Cho, Seo- Non Ji, Jun-Pyo Hong. Seung- Won Jung, and Sung Ko. Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4641-4650. October 2021.

---

[17] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee-Soo Heo, Soyeon Choe, Chiheon Ham. Sunghwan Jung, Bong-Jin Lee, and lcksang Han. In Defence of Metric Learning for Speaker Recognition. In Proc. Interspeech, pages 2977-2981, 2020.
[18] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. LibriMix: An open-source dataset for generalizable speech separation, 2020.
[19] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988 Florence, Italy, July 2019. Association for Computational Linguistics.
[20] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 933-941. PMLR, 06-11 Aug 2017.
[21] Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis R. Bach. Music source separation in the waveform domain. CoRR, 1t//191111325.. 2019.
[22] David Biaz-Guerra, Antonio Miguel, and Jose R. Beltran. gpuRIR: python library room impulse response simulation with GPU acceleration. Multimedia Tools and Applications, 80(4):5653 5671, Fehruary 2021.
[23] Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[24] Hakan Erdogan, John R. Hershey, Shinji Watanabe, Michael I. Mandel, and Jonathan Le Roux. Improved Mask Prediction Networks. In Proc. Interspeech 2016, pages
MVDR Beamforming Using 1981-1985, 2016.
[25] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. leannnseeeeaannng Advances neural information processing systems,
33:21271- 12284, 2020.
[26] Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
[27] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmennte Transformer for Speech Recognition. In Proc. Interspeech 2020, pages 5036-5040, 2020.
[28] Dan Hendrycks and Kevin Gimpel. Bridging nonlincarities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606. (0010,08155.0 2016.
[29] Hyun-Jun Heo, Ui-Hyeop Shin, Ran Lee, YoungJu Cheon, and Hyung-Min Park. Next-TDNN: Modernizing multi-scale temporal convolution backhone for speaker verification. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11186-11198 2024.
[30] John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 31-35, 2016.
[31] Jahn Heymann, Lukas Drude, Aleksej Chinaev, and Reinhold Hacb-Umbach. BLSTM supported gev beamformer front-end for the 3rd chime challenge. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 441-451, 2015.
[32] Xiaolin Hu, Kai Li, Weiyi Zhang. Luo, Mean-Marie Lemercier, and Timo Gerkmann. Speech separation using an asynchronous fully recurrent convolutional neural network. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22509-22522. Curran Associates, Inc., 2021.
12

---

[33] Weizhe Hua. Zihang Dai, Hanxiao Liu, and Quoe Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors. Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9099 9017- PMLR, 17-23 Jul 2022.
[34] Xilin Jiang, Cong Han, and Nima Mesgarani. Dual-path mamba: Short and long-term bidirectional selective structured state space models for speech separation, 2024.
[35] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient Transformer. In International Conference on Learning Representations, 2020.
[36] Morten Kolhæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen. Jensen. Multitalker speech separation with utterance- level permutation invariant training of deep recurrent neural networks, IEEF/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1901-1113 2017.
[37] Max W. Y. Lam, Jun Wang, Dan Su, and Dong Yu. Effective low-cost globally attentive locally recurrent networks. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 801-808, 2021.
audio separation using
[38] Max W. Y. Lam, Jun Wang, Dan Su, and Dong Yu. Sandglasset: A light multi-graaullann self-attentive network for time-domain speech separation. In ICASSP 2021 2021 IEEE International Conference on Acousticx, Speech and Signal P Processing (ICASSP), pages 5759-5763, 2021.
[39] Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolutional networks for action segmentation and detection. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognittin (CVPR), July 2017.
[40] Younglo Lee, Shukjae Choi, Byeong-Yedd Kim, Zhong-Qiu Wang, and Shinji Watanabe. Boosting unknown-number speaker separation with transformer decoder- attractor, 2024.
[41] Andong Li, Wenzhe Liu, Chengshi Zheng, and Xiaodong Li. Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022- 2022 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6487-6491, 2022.
[42] Kai Li, Runxuan Yang, and Xiaolin Hu. An efficient decoder architecture with tow- attention for speech separation. In The Eleventh International Conference on Learning Representations, 2023.
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019.
[44] Yi Luo, Zhuo Chen, and Nima Mesgarani. Speaker- independent speech separation with deep attractor network. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(4).787-796, 2018.
[45] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path RNN: Efficient long sequenee modeling for time- domain single-channel speech separation. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46-50, 2020.
[46] Yi Luo and Nima Mesgarani. TasNet: Time-domain audio separation network for real-time, single-channe speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 696-700, 2018.
[47] Yi Luo and Nima Mesgarani. Conv-TasNet: Surpassing ideal time-freyuency magnitude masking for speech separation. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2256):125611266 2019.
[48] Shahar Lutati, Eliya Nachmani, and Lior Wolf. Separate and diffuse: Using pretrained diffusion model for hetter source separation. In The Twelfth International Conference on Learning Representations, 2024.
[49] Craig Macartney and Tillman Weyde. Improved speech enhancement with the wave-u-net. arXiv preprint 111. 2018
[50] Matthew Maciejewski, Gordon Wichern, Emmett McQuinn, and Jonathan Le Roux. WHAMR!: Noisy and reverberant single-channel speech separation. In ICASSP 2020- 2020 IEEE International Conference on Acousticx, Speech and Signad Processing (ICASSP), pages 696-700, 2020.
[51] Zhaoxi Mu, Xinyu Yang, and Wenjing Zhu. Multi-dimensiooall and multi-scale modeling for speech separation optimized by discriminative learning. In ICASSP 2023 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pages 1-5, 2023.
13

---

[52] Hannah Raphaelle Muckenhirn, Ignacio Lopez Moreno, John Hershey, Kevin Wilson, Prashant Sridhar, Quan Wang, Rif A. Saurous, Ron Weiss, Ye Jia, and Zelin Wu. Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking. In ICASSP 2019, 2018.
[53] Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7164-7175. PMLR, 13-18 Jul 2020.
[54] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep sulti-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[55] Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self- attention. In Jan Nichues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loie Barrault, Lucia Specia, and Marcello Federico, editors, Pruceedings of the 16th International Conference on Spoken Language Trunslation, Hong Kong, November 2-3 2019. Association for Computational Linguistics.
[56] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015.
[57] Shuangging Qian, Lijjan Gao, Hongjie Jia, and Qirong Mao. Efficient monaural speech separation with multiscale time-delay sampling. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6847-6851, 2022.
[58] F A Rezaur rahman Chowdhury, Quan Wang. Ignacio Lopez Moreno, and Li Wan. Attention-base models for text-dependent speaker verification. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5359-5363, 2018.
[59] Joel Rixen and Matthias Renz QDPN Quasi-dual-path Network for single-channel Speech Separation. In Proc. Interspeech 2022, pages 5353-5357, 2022.
[60] Joel Rixen and Matthias Renz. SFSRnet: Super-resolution for single-channel audio source separation.
Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11220 -1200, Jun. 2022.
[61] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-A Assisted Intervention- MMCCOA/ 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[62] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey. SDR half-baked or well done? In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 626 006-630,222 2019.
[63] Noam Shazeer. GLU variants improve transformer. CORR, abs/2002.05202, 2020.
[64] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1953) jan 2014.
[65] Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-U-Net: sulti-scale neural network for end-to- end audio source separation. In Emilia Gómez, Xiao Hu, Fric Humphrey, and Emmanouil Benetos, editors, Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, September 23-27, 2018, pages 334-340, 2018.
[66] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21-25, 2021.
[67] Cem Subakan, Mirco Ravanelli, Samuele Cornell, François Grondin, and Mirko Brouzi. Exploring self- attention mechanisms for speech separation. JEEF/ACM Transactions on Audio, Speech, and Language Processing, 31:2169-2180, 2023.
[68] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy, MLP-Mixer: An alI-MLP architecture for vision. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22261224272. Curran Associates, Inc., 2021.
14

---

[69] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 32-42, October 2021.
[70] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo RM RFF: Efficient networks for universal audio source separation. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP). pages 1-6, 2020.
[71] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals. Alexander Graves. Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Arxiv, 2016.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[73] E. Vincent, R. Gribonval, and C. Fevotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462 1469, 2006.
[74] Thilo von Neumann, Keisuke Kinoshita, Christoph Boeddeker, Mare Delcroix, and Reinhold Haeb- Umbach. Sa-sdr: A novel loss function for separation of meeting style data. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6022-6026, 2022.
[75] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, Florence, Italy, July 2019. Association for Computational Linguisties.
[76] Sinong Wang, Belinda 7., Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2000.00466, 2020.
[77] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general U-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17683-17693, June 2022.
[78] Zhong-Qin Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Integrating full- and sub-band modeling for speech separation. /EEF/ACM Transactions on Audio, Speech, and Language Processing, 31:3221-3236, 2023.
[79] Zhong-Qin Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeer Yeol Kim, and Shinji Watanabe. TF-Gridnet: Making free-enyy domain models great again for monaural speaker separation. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.202. 2023.
[80] Zhong- Qiu Wang, Peidong Wang, and DeLiang Wang. Complex spectral mapping for single- and multi- channel speech enhancement and robust ASR. JEEE/ACM Transactions on Audio, Speech, and Language P Processing, 28: 18:117881111 1778-1787, 2020.
[81] Zhong-Qia Wang, Peidong Wang, and DeLiang Wang. Multi-microphone complex spectral mapping for utterance-wist and continuous speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 19:2001-2014, 2021.
[82] Gordon Wichern, Joe Antognini. Michael Flynn, Licheng Richard Zhu, Emmeu McQuian. Dwight Crow, Ethan Manilow, and Jonathan Le Roux. WHAM!: Extending Speech Separation to Noisy Environments. In Proc. Interspeech 20/9, pages 1368-1372, 2019.
[83] Scott Wisdom, Hakan Erdogan, Daniel P. W. Ellis, Romain Serizel. Nicolas Turpault, Eduardo Fonseca, Justin Salamon, Prem Seetharaman, and John R. Hershey, What's all the FUSS about free universal sound separation data? CoRR, abs/2011 10.0200 00000,0 2020.
[84] Zhanghao Wu Zhijian Liu*, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In International Conference on Learning Representations, 2020.
[85] Takuya Yoshioka, Hakan Erdogan, Zhuo Chen, and Fil Alleva. Mutti-microphone neural speech separation for far field multi-talked speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
2018
15

---

[86] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si. Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 10819-10829, June 2022.
[87] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer high-resolution image restoration. CVPR, 2022.
[88] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuun Yang, and Ling Shao. Multi- stage progressive image restoration, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
June 2021.
[89] Neil Zeghidour and David Grangier. Wavesplit: End-to-end speech separation by speaker clustering. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 29:2840-2849, 2021.
[90] Chunlei Zhang, Kazuhito Koishida, and John L. Hansen. Text-independent speaker verification based on triplet convolutional neural network embeddings. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(9):
2018.
[91] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[92] Shengkui Zhao and Bin Ma. Mossformer: Pushing the performance limit of monaural speech separation using gated single-head transformer with convolution-augmennee joint self-attentions. In ICASSP 2023 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[93] Shengkui Zhao, Yukun Ma. Chongjia Ni, Chong Zhang, Hao Wang, Trung Hieu Nguyen, Kun Zhou, Jiaqi Yip. Dianwen Ng, and Ma. Mossformer2: Combining transformer and mn-free recurrent network for enhanced time-domain monaural speech separation, 2023.
16

---

x R
X
X R
x
Block
Block
Split
Block
Split
Block
Encoder stage Decoder stage
Encoder stage
Decoder stage
(a) late split
(h) early split multiple decoder
X
R
x
Block
Split
Block
Block
Block
Split
Block
Block
cs Block
Encoder stage
Shared weight
Encoder stage
Shared Weight
Decoder stage
Decoder stage
(c) an early split + shared decoder (ESSD)
(d) an early split shared decoder CS (SepRe)
Figure Block diagrams various decoder designs experimented Table of subsection 5.1 all cases, the encoder and decoder consists of R stages and the blocks were stacks of global and local Transtormer block in our cases.
Dropout
Drapaut
GCFN
Dropout
Lineer,
MHSA w/ pos.
MHSA w/ pos.
MHSA w/ pos.
LN
EGA w/o gate
w/
Figure 7: The block diagram ablation studies for EGA in Table 3(b).
Appendix supplementl material
This appendix organized follows:
Appendix A describes the various decoder designs in Table 1 a)).
Appendix B illustrates cases of ablating EGA module in bbble
Appendix C experiments by comparing two speaker split layer designs.
Appendix D interprets the discriminative learning mechanism in reconstruction decoder.
Appendix E evaluates perceptual quality on WHAMR dataset.
Appendix F experiments on real-recorded reverberant mixture for overlapped speech recognition.
A Architecture of various decoder design
In Figure 6 the block diagrams of various decoder designs are illustrated, which were experimented on in Table In Table aa), the case of late split, corresponding to the first and second rows, processes single feature sequence both the encoder and decoder, forming a symmetric encoder-decoder structure. In contrast, the case of an early split with multiple decoders, shown in Figure b[b), has each decoder block processing the separated sequences from the encoder. Thus, Figure b) can be seen as a special case of late split with a large decoder of 2F, where the large decoder performs group operation with the number of groups equal to the number of speakers J. On the other hand, in the ESSD method illustrated in Figure the decoder shares weights to process the early split feature sequences. Therefore, even without interaction between the separated feature sequences, the decoder can learn discriminative characteristics by sharing the weights. Furthermore, in the proposed SepRe method shown in Figure 6 d), the decoder learns to attend to each other using the CS block additionally recover the deviated elements.
17

---

Separation Encodier
Reconstruction Decoder
Separation Encoder
Reconstruction Decoder
Input Layer
Encoder Stage
Encoder Stage
Bottlenack Stage
Decoder Stage
Decoder PecoderSStgee Stage
Output Layer
Input
Enooder Stage
Encoder Stage
Bottleneok Stage
Decoder Stage N
Decoder er Stage
Output syer
Split
Split
Split
Split
(a) Shared speaker split
(b)
Multiple speaker split
Figure 8: The block diagram of shared and multiple speaker split layer in SepReformer architecture.
Params. (M)
SI-SNRi (dB)
SDRi (dB)
Speaker Split
Weight-shared layer
55.3
25.2
25.3
Multiple layer
59.4
25.1
25.2
Table 6: Comparison of shared and multiple speaker split layer based on SepReformer-L with DM on the WSJ0-2Mix, WHAM!, and WHAMR! dataset.
B Illustration of ablation for EGA module
Figure] 7 we drew block diagram ablation studies for Table In the fisrt row of Table b), MHSA is simply performed with downsampling and upsampling to reduce the sequence length. In the second row, the upsampled output sequences of MHSA with downsampling is multiplied to the input features before downsampling order reflect detailed temporal information. However, simply multiplying the input features does not improved the performances. reflect the frame-level details to the upsampled feature, we added linear layer and sigmoid function to make gate values, Icading to our proposed EGA module.
C Comparison of shared and multiple split layers
designing the SepReformer architecture, a key consideration whether to use a shared speaker split layer across all feature stages as illustrated in Figure (ja), or to implement distinct speaker split layers for each stage in Figure b). Our hypothesis is that shared speaker split layer would enable the reconstruction decoder process consistently separated feature sequences. Conversely. assigning unique split layer each stage in the separation encoder could account for stage-specific feature variations, as shown Figure ()). Therefore, we conducted comparative experiments on these two configurations.
Upon comparing the two approaches of the multiple and the shared split layer. interestingly. different tendencies are observed depending on the type of mixture, whether it was an anechoic clcan mixture or noisy or, furthermore, reverrreverberant mixture, in terms of SI-SNRi and SDRi performance (as shown in Table 6). For the WSJO-2Mix dataset, which comprises simple anechoic mixtures, the shared split layer method produced results that were comparable to, or slightly better than, those our hypothesis. However, for the WHAM dataset, containing noisy mixtures, the multiple split layer approach demonstrated improved results, with the performance gap still sustained on the WHAMR dataset, which includes noisy and reverberant mixtures. Based these findings, opted for the shared split layer approach in most cases, as it provided competitive results while reducing model's parameter count. SepReformer-L with DM in Table we employed multiple split layers to enhance performance, particularly on the WHAM and WHAMR datasets.
multiple split layer approach, in with
other hand,
D Visualization of discriminative learning
To analyze the roles of each transformer block within the shared decoder of the ESSD framework with multi-loss application, we calculate the cosine similarities between the separated features from Z1 to Z4 shown in Figure First, we observe that Z1 has higher cosine similarity compared the others, indicating that the separated features initially share similar characteristies before being processed by weight sharing blocks. However, after the weight-sharing global block, the similarities in Z2 are generally lower compared to Z1.
This result suggests that the weight-shan sharing global blocks capture discriminative characteristics. making the features more dissimilar in terms of cosine similarities. The further decrease in similarity in Z3 compared to
18

---

Froquancy (Hz))
2
1k
Time (s)
(a) Spectrogram of speaker
Prequaccy (H2)
2k
1k
4
Time (s)
(b) Spectrogram of speaker
Cosine Similrityy
21
22
23
Time (s)
(c) Cosine similarity between split features time
Z1
Z2
Z3
Z4
CS Block
Global Block
Local Block
Shared Weight
Shared Weight
(d) Block pipelines
Figure 9: Plot of cosine similarities for the two separated features in the first decoder stage using sample mixture WSJO-2Mix dataset.
Separator
S1-SNRi (dB)
SDRI (dB)
PESQ
eSTOI
No Processing
-6.1
-3.5
1.41
0.317
TF-GridNet (78
10.6
11.7
2.75
0.793
SepReformer-L
11.0
12.85
2.78
0.798
Table 7: Perceptual evaluation by PESQ and eSTOI on WHAMR! dataset.
Z2 and Z1 demonstrates the effect of the weight-sharing structure of the local block. As features pass through subsequent local blocks, more region-specific traits are refined. The local block, which handles these localized characteristics. further enhances the distinctiveness of the local features, resulting in decreased similarity.
In contrast, features processed through the CS block exhibit increased similarity, unlike the weight-sharing structure. This increase in similarity can understood as the separated features attending and becoming more similar to each other, as the CS structure designed to cross-ree reference information between the speech features. During this process, the CS block preserves distinct features and restores degraded information by leveraging mutual information. As the weight-sharing structure emphasizes unique characteristics, the split features deviate from the original characteristies of the speech within the frames, where the influence of each speaker's information similar. This deviation occurs because emphasizing features in such frames can increase interference from other speakers' speech components, potentially distorting and degrading these features. Therefore, the CS block after the weight-sharing block is expected to recover the deviated features by attending to each other within the frames.
19

---

Overlap Ratio (%)
Separator
0S
10
30
40
No Processing
11.8
11.7
18.8
27.2
35.6
43.3
BLSTM [13
15.8
14.2
18.9
25.4
31.6
35.5
Conformer 10
12.9
12.2
15.1
20.1
24.3
27.6
DPRNN +7
10.6
10.4
12.7
16.6
20.8
23.5
SepReformer-B 9.6
10.2
10.6
12.5
13.8
16.1
Table 8: WERs (%) of utterance- wise evaluation the LibriCSS dataset for the baseline without any processing for input data acquired at the center microphone and separation by LSTM, Conformer, DPRNN, and proposed SepReformer.
E Evaluation of speech perception measures
While SI-SNRi is often the primary metric for speech separation tasks, it's important also report perceptual metrics like PESQ and eSTOI, especially for datasets like WHAMR! that include noise and reverberation. Table shows that SepReformer-L demonstrates effective performance compared to IF-GridNet [78). with a PESQ of 2.79 and an eSTOI of 0.799, slightly improving upon TF- Griddeee''ss scores of 2.75 and 0.793. No Processing bascline performs poorly, as expected, with a PESQ of 1.41 and eSTOI of 0.317, reflecting low perceptual quality and intelligibility without any enhancement. GridNNeeet shows notable results with SI-SNRi of 10.6 dB, while SepReformer-L shows slightly better performance across both perceptual and signal-Ievel metries. This suggests that SepReformer-L is also effective in maintaining perceptual quality and intelligibility in challenging environments, making it a relevant model for real-world applications.
Evaluation on real-recorded mixture
To further validate the applicability of the proposed model in real-world environments, we evaluated word error rate (WER) the LibriCSS [ datasct using baseline speech recognition model. The dataset consists of recordings that simulate real meeting scenarios, allowing
the separation performance of model as pre-processing step for overlapped speech recognition. The LibriCSS data set enables evaluation of varying levels of overlap, ranging from 0% to 40%, which helps not only in measuring separation performance but also in verifying the model's ability to preserve speech quality when overlap is minimal. Accordingly, the proposed model was trained using aggreeagg SDR (SA-S) 3DS))R 74] instead of conventional averaged SDR, with varying overlap ratios to account for different levels of overlap during training. For comparison, additionally trained and evaluated the DPRNN model as a competitive baseline. Since LibriCSS dataset does not provide a training set, we trained the model using LibriSpeech [56) for speech source with noise simulated using colored noise and reverberation based room impulse response (RIR) simulations using gpuRIR 22 Although the evaluation dataset includes both reverberation and moderate background noise, excluded dereverheration during training to ensure stability. Thus, the model was trained focus solely on speech separation and noise reduction.
Referring to Table S we can observe that the word error rate (WER) increases as the overlap ratio (OV) rises for the unprocessed input. Even in cases with no overlap, the haseline results show around 10% WER due to inherent distortions from reverberations and background noises. The BLSTM [ and Conformer [10 models, which are based STFT and real-valued masking, show some improvement higher OV ratios, but their performance tends degrade compared to input when the OV is low, indicating unstable results. This instability could be attributed to the negative effects of attempting to remove reverberation as well. In contrast, the results of the DPRNN model demonstrate consistent improvements over the unprocessed input across all overlap ratios. The proposed SepReformer model further improves performance. This suggests that the speech separation was performed effectively. However, since reverberation was not removed, the results show less improvement the OVO condition, which can be interpreted as the model successfully preserving the original speech without unnecessary distortion.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section. 3 and 5.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of work performed by the authors?
Answer: [Yes]
Justification:
See Section. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well sell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to- teecchtt-textt system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
21

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We described the detailed model architectures and explained tranining and model configuration in Section 4 and Supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.- with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Not only did we include the implmentation code in supplemental material, but also we are planning release our code for our main experiments in public soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttpa://nips.co///
Saaddes/CoeSuubni
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ //nnis/ cc/public/ qu.ides/booeeuubllesssoooll..... for more details.
22

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed methad and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental Setting/Detann Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sce Section and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Scction
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL. if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For cach experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
23

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::/..ttpp////.pppp.p.paaaa/a
Answer: [Yes]
Justification:
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [NA]
Justification: We concluded that source separation technology does not have negative social impacts since it does not create new data.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path any negative applications, the authors should point out. example, it legitimate to point out that an improvement in the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
24

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yss)
Justification:
Guidelines:
The answer NA mcans that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should he provided. For popular datasets, paperswithcode com/ datasets has curated licenses for datasets. Their licensing guide help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information is not available online, the authors encouraged to reach out the asset's creators.
13. New Assets
Question: Are new introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submis- sions via structured templates. This includes details about training, license, limitations, etc. used.
The paper should discuss whether how consent was obtained from people whose asset
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
25

---

Guidelines:
answer means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
26


Reviews:
[{"review_id": "Vab0DhJvBO", "reviewer": "Reviewer_Gfro", "summary": "The authors propose a neural network architecture to separate a speech mixture containing 2 speakers.  The proposed U-net based architecture replaces the inter and intra chunk processing - a popular method for speech separation - with global and local attention mechanisms. They further propose a mechanism to reduce the computational cost of the model. The model does the speech separation early in the network and uses a decoder with a low parameter count - all thanks to its weight sharing strategy - to do the speech separation.  Evaluation on simulated data shows improvement over the state of the art methods.", "questions": "1. Looking at Figure 2, I’m wondering if it’s really a good idea to use the same split network for all ‘r’ values.  Would it be better to have a different ‘spk’ split for each ‘r’ ?\n\n2. I believe Eq(1) should be a minimum of the Si-SNR and \\tau since you refer to it in line 201?\n\n3. In line 213, shouldn’t the multi loss be (1-\\alpha / R) L + \\alpha \\sum_r L_r/R ?\n\n4. It is not very clear  why the authors refer to the decoder as a Siamese decoder. Is this because they share the same weights across all speakers? This is typically how a speech separation network is structured. I don't see any contrastive learning loss functions as part of the training loss to discriminate the speakers.", "limitations": "Lack of evaluation on real data is a limitation of this work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The Efficient Global Attention (EGA) component of the model, as discussed in Section 3.3, appears to be a cost-effective method for using the global context. This is achieved by initially subsampling to a reduced number of frames, thereby reducing the computational cost associated with transformers, and subsequently upsampling to a larger number of frames. The anticipated loss incurred from the downsampling process is compensated through the implementation of a gating mechanism.\n\n2. Results show all the proposed mechanics, namely: early split, multi-loss, shared decoder parameters, EGA module design gives improvement in Si-SDR \n\n3. Interestingly, when the proposed methods were applied to existing architectures such as conv-tasnet and Sepformer, an improvement in SI-SNR was observed. \n\n4. Appendix D, also shows the computation effectiveness of the model\n\n5. Attached samples clearly showed the quality of separated speech using the proposed network.", "weaknesses": "1. The authors have shown good results on a bunch of datasets, and the separated audios in the supplemental file are of high quality. But all these results are based on simulated data, so it makes you wonder how well the model would do with real data. It’d be great if the authors could show how the model performs on the Chime-6 dataset or Libricss, maybe through WER metrics or even objective speech perception measures after separation. Or at least, they could show us some samples after separation on real data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The authors have shown good results on a bunch of datasets, and the separated audios in the supplemental file are of high quality. But all these results are based on simulated data, so it makes you wonder how well the model would do with real data. It’d be great if the authors could show how the model performs on the Chime-6 dataset or Libricss, maybe through WER metrics or even objective speech perception measures after separation. Or at least, they could show us some samples after separation on real data.**\n\nA1: Thank you for your valuable feedback. We appreciate your point regarding the importance of evaluating the model on real-recorded datasets. We will include word error rate (WER) evaluation results for the LibriCSS dataset in the appendix.\n\nAdditionally, we previously evaluated an early version of SepReformer on LibriCSS, similar to the proposed Ours+U-Net in Table 5. Although we cannot provide the WER from the latest SepReformer at this moment as it requires additional training with background diffuse noise and reverberation, we can share a comparison of an early version of SepReformer with DPRNN, using the Librispeech dataset and Room-Impulse-Response (RIR) simulations with background noise.\n\n| WER on LibriCSS |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Condition |  |  |  |  |  |  |\n| Overlap Ratio in %  | 0S | 0L | 10 | 20 | 30 | 40 |\n| Oracle | 4.9 | 5.1 | - | - | - | - |\n| Input | 11.8 | 11.7 | 18.8 | 27.2 | 35.6 | 43.3 |\n| DPRNN | 10.6 | 10.4 | 12.7 | 16.6 | 20.8 | 23.5 |\n| Early version of SepReformer | 9.8 | 10.1 | 10.9 | 12.5 | 14.4 | 17.5 |\n\nPlease note that SepReformer is expected to perform better than the early version. Both DPRNN and our model were trained for stable separation and denoising without dereverberation.\n\nFurthermore, as our work focuses on speech, reporting metrics such as PESQ and STOI for datasets like WHAMR!, which include noise and reverberation, would be valuable. We will include PESQ and STOI results for the WHAMR! dataset in the appendix.\n\n| WHAMR! | SI-SDR (dB) | SDR (dB) | PESQ | eSTOI |\n| --- | --- | --- | --- | --- |\n| Unprocessed | -6.1 | -3.5 | 1.41 | 0.317 |\n| TF-GridNet[1] | 10.6 | 11.7 | 2.75 | 0.793 |\n| SepReformer-L | 11.0 | 12.5 | 2.77 | 0.796 |\n\n**Q2: Looking at Figure 2, I’m wondering if it’s really a good idea to use the same split network for all ‘r’ values. Would it be better to have a different ‘spk’ split for each ‘r’ ?**\n\nA2: Thanks for your insightful comment. In our early version, we initially used different split layers for each stage. We then considered using a shared split layer, assuming that consistently separated feature sequences could benefit the reconstruction decoder. Interestingly, we observed more stable convergence and no significant difference in separation performance. To simplify our model and reduce the number of parameters, we opted for the shared split layer. However, as you suggested, we will add experiments comparing the two approaches in the appendix, as exploring shared split layers for skip connections in the U-Net structure is indeed worth investigating.\n\n**Q3: I believe Eq(1) should be a minimum of the Si-SNR and \\tau since you refer to it in line 201?**\n\nA3: We really appreciate for your detailed comment and apologize for the typo. As you commented, Eq(1) should be a minimum of the Si-SNR and \\tau. We will correct it.\n\n**Q4: In line 213, shouldn’t the multi loss be (1-\\alpha / R) L + \\alpha \\sum_r L_r/R ?**\n\nA4: If all the losses at each stage, including $\\mathcal{L}$ should be considered equally, the multi-loss $\\hat{\\mathcal{L}}$ can be set to $(1-\\alpha) \\mathcal{L} /R + \\alpha \\sum_r \\mathcal{L}_r/R$. However, we thought the final loss of $\\mathcal{L}$ alone is important as much as summation of all the auxiliary loss $\\mathcal{L}_r$. Therefore, we set the multi-loss as $(1-\\alpha) \\mathcal{L} + \\alpha \\sum_r \\mathcal{L}_r/R$.\n\n**Q5: It is not very clear why the authors refer to the decoder as a Siamese decoder. Is this because they share the same weights across all speakers? This is typically how a speech separation network is structured. I don't see any contrastive learning loss functions as part of the training loss to discriminate the speakers.**\n\nA5: Thank you for your valuable comment, and we apologize for the confusion caused by misuse of the term. As you pointed out, we named the weight-sharing part across all speakers the Siamese decoder. We did not use an additional loss to discriminate the speakers using speaker identities. Instead, we trained the model directly with a separation objective based on  the Permutation Invariant Training (PIT) loss, which is conventional in speech separation network.\n\nMost speech separation networks are designed with a single feature sequence that is addressed before a late split layer. However, in our approach, given early separated feature sequences, whose similarity is relatively high (as indicated by grey line in Figure6(c)), the weight-sharing blocks can learn to discriminate speech sources (as indicated by orange and blue lines in Figure6(c)). This suggests that the discriminative learning is enhanced by the weight-sharing (or what we thought of as Siamese) structure. We believe that separation loss with PIT itself can operate as a kind of discriminative learning with ESSD structure, given the early split feature sequence. \n\nNevertheless, we acknowledge that using the term ‘Siamese’ must be misleading since we did not apply a typical contrastive loss. Moreover, we concluded that prematurely using the term \"Siamese\" could be risky, as it may hinder considering scenarios involving more than two individuals in the future. Therefore, we will replace the term ‘Siamese’ with ‘weight-shared’ in the revised paper.\n\n[1] Z. -Q. Wang, S. Cornell, S. Choi, Y. Lee, B. -Y. Kim and S. Watanabe, \"TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation,\" in IEEE/ACM TASLP, vol. 31, pp. 3221-3236, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X5WFgYHBQX", "reviewer": "Reviewer_aFMn", "summary": "The paper proposes SepReformer, an efficient time-domain separation network. The model is an encoder-decoder architecture that splits the output features of the encoder based on the number of speakers before feeding them to the decoder. Both encoder and decoder networks are comprised of transformer blocks that capture global and local characteristics of the signal in different time-scales. The proposed approach achieves state of the art results in 3 well-established datasets.", "questions": "1. How does the early split strategy affect the overall computational efficiency compared to late split and other conventional methods?\n\n2. Why was the alpha value set to 0.4, and have you experimented with other values to determine the optimal setting?\n\n3. How does the model handle varying lengths of input sequences, and what is the maximum sequence length it can effectively process?\n\n4. Are there any limitations or failure modes of the proposed method that have been identified during experimentation?", "limitations": "1. The paper does not discuss potential failure modes or limitations observed during experimentation,", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Paper well-written and easy to follow.\n2. Efficient architecture that produces state of the art results.\n3. Thorough experimentation and ablation analysis of their approach.", "weaknesses": "1. Testing on clean data. It would be interesting to see how the proposed model performs on noisy datasets.\n2. Evaluation of the approach to the two-speaker separation problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Testing on clean data. It would be interesting to see how the proposed model performs on noisy datasets.**\n\nA1: Thank you for your comment.  We performed experiments on noisy and noisy-reverberant datasets in Table 4 to show our models generalizability power. Please refer to it.\n\n**Q2: Evaluation of the approach to the two-speaker separation problem.**\n\nA2: We appreciate your concern regarding the extension of our method to scenarios with more than two speakers. However, we are confident that our ESSD mechanism will perform effectively even with more speakers compared to conventional methods.\n\nOur ESSD method naturally extends the feature dimension and requires increased computation proportional to the number of sources to be separated. This approach is more reasonable compared to conventional methods, which do not adjust the feature dimension based on the number of speakers. This limitation in conventional methods is significant because the separation difficulty and network requirements vary with the number of sources.\n\nTo validate this, we are conducting experiments on the WSJ-{3,4,5}MIX datasets, which include scenarios with more than two sources. We expect our model to show SOTA performance in these multi-source scenarios. Specifically, in WSJ-3MIX, our SepReformer-B showed  an SI-SNRi of 23.5 dB, which is near SOTA performance, and we believe the performance will increase further for larger models. Although these results were not included in the paper, the experimental results and their discussion will be included in our future work, as mentioned in the conclusion of Section 6.\n\n**Q3: How does the early split strategy affect the overall computational efficiency compared to late split and other conventional methods?**\n\nA3: Thank you for your valuable comment. First, even with the same number of parameters, using the ESSD structure will approximately increase the computation by factor of the number of speakers in the decoder. Therefore, simply changing a late split structure to an ESSD structure can potentially increase the computation significantly if the channel size is kept constant. Adding a Cross-Speaker module would further increase the computation slightly. \n\nHowever, with the ESSD structure, we can reduce the channel size while still achieving higher performance, which would significantly improve computational efficiency relative to performance. As shown by comparing the second and third rows in the table below, even with a substantial reduction in channel size and computation, the performance improved.\n\nComparing with traditional methods, the late split method in the first row shows competitive performance against models like Conv-TasNet, SuDoRM-RF, and DPRNN, achieving better performance with less computation using the Global-Local Transformer. The ESSD + CS (SepRe) structure in the third row demonstrated comparable or better performance with significantly reduced computation. Additionally, the proposed base model outperforms TF-GridNet with much smaller computation. This highlights the efficiency of the proposed SepRe method.\n\nWe will make sure to clearly highlight these points in the revised paper to provide a better understanding of how the early split strategy impacts computational efficiency.\n\n| Case | F(scale) | Params. (M) | MACs (G/s) | SI-SNRi (dB) |\n| --- | --- | --- | --- | --- |\n| Ours + Late Split + origin dec | 64(Tiny) | 3.3 | 5.1 | 19.0 |\n| Ours + Late Split + origin dec | 128(Base) | 11.6 | 18.3 | 21.6 |\n| Ours + Early Split + shared dec +CS | 64(Tiny) | 3.7 | 10.4 | 22.4 |\n| Ours + Early Split + shared dec + CS | 128(Base) | 14.2 | 39.8 | 23.8 |\n| Conv-TasNet | - | 5.1 | 10.5 | 15.3 |\n| SuDoRM-RF | - | 6.4 | 10.1 | 18.9 |\n| DPRNN | - | 2.6 | 88.5 | 18.8 |\n| Sepformer | - | 26.0 | 86.9 | 20.4 |\n| SFSRNet | - | 59.0 | 466.2 | 22.0 |\n| ISCIT | - | 58.4 | 252.2 | 22.4 |\n| TF-GridNet | - | 14.5 | 460.8 | 23.5 |\n\n**Q4: Why was the alpha value set to 0.4, and have you experimented with other values to determine the optimal setting?**\n\nA4: We did not perform experiments with other values for the alpha parameter. While further tuning might potentially improve performance, we did not consider this aspect critical because the alpha value also decays as the training epoch proceeds. Therefore, we did not conduct extensive experiments to optimize this parameter.\n\n**Q5: How does the model handle varying lengths of input sequences, and what is the maximum sequence length it can effectively process?**\n\nA5: Thank you for this valuable question. During training, we fixed the input length to 4 seconds for efficiency. However, during evaluation, the model processes inputs of varying lengths in one batch, similar to most existing studies. We apologize for not explicitly stating this in the paper and will ensure to clarify this point in the revised version. The average length of test samples in the WSJ0-2Mix dataset is about 6 seconds, with a maximum length of about 15 seconds. The model can effectively separate sources within this range without significant issues. For sequences longer than 10-20 seconds, we can split the input into smaller segments of 5-10 seconds for processing.\n\n**Q6: Are there any limitations or failure modes of the proposed method that have been identified during experimentation?**\n\nA6: As mentioned in the conclusion, our method still struggles with cases involving varying numbers of speakers due to the fixed split layer. We believe this approach is important because, in practice, identifying the number of speakers in advance can be cumbersome. Our model still has limitations in this regard because it relies on a split layer with fixed input and output shapes. This means the network requires prior knowledge of the number of speakers to be separated, and individually trained models must have corresponding split layers to address different numbers of speakers. We will make sure to clarify these potential limitations in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOsa17YeHc", "reviewer": "Reviewer_UNNK", "summary": "This paper presents a novel approach to time-domain speech separation, departing from the conventional chunk-based dual-path processing. The authors introduce an asymmetric encoder-decoder architecture, where the encoder analyzes features and splits them based on the number of speakers. A Siamese decoder reconstructs the separated sequences, learning to discriminate features without explicit speaker information. The use of global and local Transformer blocks for long-sequence processing eliminates the need for chunking, contributing to a more efficient and effective model.", "questions": "--", "limitations": "--", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces an innovative asymmetric encoder-decoder framework for time-domain speech separation, deviating from the standard chunk-based dual-path models. \n2. Efficient Feature Discrimination: The Siamese decoder enables the model to learn to discriminate features directly without relying on explicit speaker information, leading to a more streamlined and potentially more robust separation process.\n3. The proposed model achieves good performance on benchmark datasets while requiring significantly less computation than previous approaches. This demonstrates the potential for this method to be applied in real-world applications where computational resources may be limited.", "weaknesses": "The paper presents some interesting ideas, but their novelty and significance are questionable:\n\n1. Transformer Usage: While the use of Transformer blocks is highlighted, similar architectures have been successfully employed in previous works like Sepformer, raising questions about the uniqueness of this contribution.\n2. Limited Evaluation: The experimental results primarily focus on two-speaker separation, which is considered a relatively solved problem in the current state-of-the-art. The absence of evaluations on more challenging scenarios with three or more speakers limits the generalizability and impact of the findings.\n3. Incomplete Comparison: The paper's claims of achieving state-of-the-art results are undermined by the lack of comparison with other important papers in the field. Notably, models like the \"DIFFUSION-BASED SIGNAL REFINER FOR SPEECH SEPARATION\" have reported superior performance (SI-SDR of 23.1dB), raising concerns about the validity of the SOTA claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Transformer Usage: While the use of Transformer blocks is highlighted, similar architectures have been successfully employed in previous works like Sepformer, raising questions about the uniqueness of this contribution.**\n\nA1: Thank you for your comment. We agree that the usage of Transformer blocks is now very common in the speech separation field. Sepformer model primarily replaces the LSTM in a dual-path structure, similar to DPRNN, with classic Transformer blocks. However, our proposed method does not use a dual-path approach. Instead, we designed Global and Local Transformers with EGA and CLA, replacing the simple multi-head self-attention (MHSA) to handle global and local modeling more effectively. Additionally, we have enhanced the vanilla FFN by integrating it with a gating mechanism to create the GCFN, which efficiently models adjacent frames. Therefore, our approach is substantially different from the straightforward application of Transformer, and we believe these enhancements represent significant contributions. Experimental results of Table3(b), Table3(c), and Table 5 show the effectiveness of proposed Local and Global Transformer alone, without SepRe method.\n\n**Q2: Limited Evaluation: The experimental results primarily focus on two-speaker separation, which is considered a relatively solved problem in the current state-of-the-art. The absence of evaluations on more challenging scenarios with three or more speakers limits the generalizability and impact of the findings.**\n\nA2: We appreciate your concern regarding the extension of our method to scenarios with more than two speakers. Although we focused on experiments using datasets including noisy and noisy-reverberant mixture to show our model’s generalizability in this paper, we are also confident that our ESSD mechanism will perform effectively with even more than two speakers compared to the conventional methods.\n\nOur ESSD method naturally extends the feature dimension and requires increased computation proportional to the number of sources to be separated. Meanwhile, conventional methods do not adjust the feature dimension based on the number of speakers. This limitation in the conventional methods is significant because the separation difficulty and network requirements vary with the number of sources.\n\nTo validate this, we are conducting on experiments on the WSJ-{3,4,5}MIX datasets, which include scenarios with more than two sources. We expect that our model will still show SOTA performance in these multi-source scenarios. Specifically, in the WSJ-3MIX, our SepReformer-B showed an SI-SNRi of 23.5 dB, which is comparable to the SOTA performance, and we expect that the score will increase further for a larger model. Although these results were not included in the paper, the experimental results and their discussion will be included in our future work, as mentioned in the conclusion of Section 6.\n\n**Q3: Incomplete Comparison: The paper's claims of achieving state-of-the-art results are undermined by the lack of comparison with other important papers in the field. Notably, models like the \"DIFFUSION-BASED SIGNAL REFINER FOR SPEECH SEPARATION\" have reported superior performance (SI-SDR of 23.1dB), raising concerns about the validity of the SOTA claim.**\n\nA3: Thank you for your comment. We acknowledge the importance of comprehensive comparisons in validating state-of-the-art (SOTA) claims.\n\nThe paper you mentioned [1] evaluates the separation performance based on the WSJ0-2MIX dataset, reporting an SI-SDRi of 23.1 dB. To address your concern, we have indeed compared our results with competitive models in Table 4 of our paper.\n\n- Our SepReformer-S model, which has a smaller model size and lower computational requirements than many recent powerful networks, shows an SI-SDRi of 23.0 dB, which is comparable to the performance of the diffusion-based model you mentioned.\n- Furthermore, our SepReformer-L model demonstrates an SI-SDRi of 25.1 dB, which to the best of our knowledge, represents state-of-the-art performance with a significant margin over existing models.\n\nWe recognize the diffusion-based model’s performance as reported in [1] and [2], with results of 23.1 dB and 23.9 dB, respectively. However, it is important to note that these models did not include evaluations on noisy and noisy-reverberant datasets, and they did not provide details on model size and computational resources. Therefore, we chose not to include them in our comparison table. Nevertheless, we agree that it is still valuable to provide the results of the diffusion-based model in Table 4 according to your comment. Therefore, we will include the result of [2] which shows more competitive result than [1].\n\nThank you again for your valuable feedback.\n\n[1] Hirano, Masato, et al. \"Diffusion-based Signal Refiner for Speech Separation.\" *arXiv preprint arXiv:2305.05857,* 2023.\n\n[2] Lutati, Shahar, Eliya Nachmani, and Lior Wolf. \"Separate and diffuse: Using a pretrained diffusion model for better source separation.\" The Twelfth International Conference on Learning Representations. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBOOaEzafF", "reviewer": "Reviewer_y9eH", "summary": "This paper proposes a novel transformer based architecture, sepreformer, for speech separation based on a siamese decoder network that operates on separated speech signals in the encoded space.", "questions": "Are there any experiments conducted on more than 2 speakers? Are there any metrics considered beyond SNR?", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "A major strength of the paper is it's state of the art performance on a variety of datasets, showing as high as 25dB on the WSJ 2 mix. In addition, sepreformer is quite computationally efficient when compared with other baseline methods. The supplementary materials include wav files which show the fidelity of the reconstructed examples. The ablation studies provide good insights into the contributions of different component, such as the depth of the encoder decoder. Details of the method are provided with high degree of reproducibility and description.", "weaknesses": "The first major weakness is the presentation quality of the paper. Starting with the abstract, which goes right into a discussion of feature length and computation without setting up the problem and overview. I think a lot of the high level picture is missing, especially related to the particular insights that help this method work better.\n\nBeyond the presentation, a main concern I have of this method is extending to beyond 2 sources. The experiments focus on two-speaker separation. It's unclear how well the method would generalize to scenarios with more than two speakers, which is an important real-world use case. It's also particularly important here because the authors propose separating the sources first and then using a siamese network for decoding the features back to speech. With more than 2 speakers, it's quite likely that the method would have a harder time as the features for each source get separated out earlier in the process.\n\nThe experiments and comparisons primarily focuses  on SI-SNRi. I'd like to see other metrics like PESQ or STOI considered as SNR does not always represent perceptual quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The first major weakness is the presentation quality of the paper. Starting with the abstract, which goes right into a discussion of feature length and computation without setting up the problem and overview. I think a lot of the high level picture is missing, especially related to the particular insights that help this method work better.**\n\nA1: Thank you for your feedback on the presentation quality of the paper. We recognize that the abstract lacks a detailed problem formulation and overview, which can make it challenging for readers to grasp the context and significance of our work.\n\nIn the revised paper, we will include an earlier explanation of the TasNet concept in the abstract to enhance clarity. Also, we agree that TasNet overview should be more detailed. Therefore, we will expand on the concept of TasNet and the task of speech separation in the introduction.\n\nMoreover, we agree that we did not provide sufficient explanation with high-level picture why proactively expanding the speaker dimension with early split (ESSD method) is more advantageous than the late split. To show effectiveness of the proposed method, we will clarify the task and challenge of speech separation, emphasizing that it requires generating more information than what is provided in the original input data. Also, we will clarify that\n\n1. generating the separated feature at once just before the audio decoder (late split) can be overwhelming for the separator.\n2. and, Instead, splitting the feature early and using a reconstructing decoder can ease the task for the encoder of separator.\n\nPlease refer to Figure 6 in the appendix to demonstrate these points.\n\nBy clarifying these points, we aim to improve the presentation quality clarity and make the high-level picture and insights of our method more accessible to the readers.\n\n**Q2: Beyond the presentation, a main concern I have of this method is extending to beyond 2 sources. The experiments focus on two-speaker separation. It's unclear how well the method would generalize to scenarios with more than two speakers, which is an important real-world use case. Are there any experiments conducted on more than 2 speakers?**\n\nA2: We appreciate your concern about extending our method to more than two speakers. We agree that demonstrating effectiveness in multi-source scenarios is crucial for real-world applicability.\n\nWhile our current paper focuses on two-speaker separation, we recognize the importance of exploring multi-speaker scenarios. We plan to include detailed experimental results and discussions on this topic in our future research, as noted in the conclusion of Section 6. We are confident in the scalability of our method.\n\nOur ESSD mechanism is designed to handle multiple speakers by naturally extending the feature dimension and increasing computation in proportion to the number of sources. This contrasts with conventional methods, which do not adjust the feature dimension based on the number of speakers, often limiting their effectiveness as the number of sources increases. \n\nWe are currently working on experiments using the WSJ-{3,4,5}MIX datasets, which include scenarios with more than two sources. Preliminary results are promising; for example, our SepReformer-B achieved an SI-SNRi of 23.5 dB on WSJ-3MIX, nearing state-of-the-art performance. We anticipate further improvements with larger models and additional research.\n\n**Q3: The experiments and comparisons primarily focuses on SI-SNRi. I'd like to see other metrics like PESQ or STOI considered as SNR does not always represent perceptual quality. Are there any metrics considered beyond SNR?**\n\nA3: We primarily focused on SI-SNRi because most studies on speech separation (unlike speech enhancement) evaluated separation performance using only signal-level metrics of SNR, particularly for simple instantaneous mixtures without background noise and reverberation. However, we agree that reporting additional metrics such as PESQ and STOI for datasets like WHAMR!, which include noise and reverberation, would be valuable. The table below shows PESQ and eSTOI results, demonstrating competitive performance compared to the recent powerful TF-GridNet model [1].\n\n|  | SI-SDR (dB) | SDR (dB) | PESQ | eSTOI |\n| --- | --- | --- | --- | --- |\n| Unprocessed | -6.1 | -3.5 | 1.41 | 0.317 |\n| TF-GridNet[1] | 10.6 | 11.7 | 2.75 | 0.793 |\n| SepReformer-L | 11.0 | 12.5 | 2.77 | 0.796 |\n\nTherefore, we will include the PESQ and STOI for the WHAMR! dataset in the appendix.\n\nAdditionally, evaluating speech recognition accuracy for separated speech using real data like LibriCSS [2] would be meaningful for real-world applications. We will include WER evaluation results for the LibriCSS dataset in the appendix. Although we cannot provide WER results now due to the need for additional training with background noise and reverberation, we include perviously evaluated comparison results of an early version of SepReformer with DPRNN for reference:\n\n| WER on LibriCSS |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n| Condition | Overlap Ratio in % |  |  |  |  |  |\n|  | 0S | 0L | 10 | 20 | 30 | 40 |\n| Oracle | 4.9 | 5.1 | - | - | - | - |\n| Input | 11.8 | 11.7 | 18.8 | 27.2 | 35.6 | 43.3 |\n| DPRNN | 10.6 | 10.4 | 12.7 | 16.6 | 20.8 | 23.5 |\n| Early version of SepReformer | 9.8 | 10.1 | 10.9 | 12.5 | 14.4 | 17.5 |\n\nNote that the performance of SepReformer is expected to be better than shown.\n\n---\n\n[1] Z. -Q. Wang, S. Cornell, S. Choi, Y. Lee, B. -Y. Kim and S. Watanabe, \"TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation,\" in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3221-3236, 2023\n\n[2] Z. Chen et al., \"Continuous Speech Separation: Dataset and Analysis,\" ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020, pp. 7284-7288", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their helpful comments and suggestions. We sincerely appreciate the time and effort they have dedicated to reading and commenting on the paper. Below, please find our point-by-point response to all the comments. We believe that it is greatly improved by incorporating the reviewers' suggestions and comments.

Meta-review: The paper presented sepformer which uses global and local transformers with EGA and CLA to effectively model the global and local information for speaker separation. The proposed method has shown better results comparing to SoTA. The authors also provided additional results to address reviewers’ concerns. However, the presentation of the paper still needs improvements and the novelty is good but not breakthrough. Hence I recommend an accept.
Decision: Accept (poster)