Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation

Authors: ui-hyeop  shin
sangyoun  lee
taehan  kim
hyung-min  park
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:02:34

Abstract: In speech separation, time-domain approaches have successfully replaced the time-frequency domain with latent sequence feature from a learnable encoder. Conventionally, the feature is separated into speaker-specific ones at the final stage of the network. Instead, we propose a more intuitive strategy that separates features earlier by expanding the feature sequence to the number of speakers as an extra dimension. To achieve this, an asymmetric strategy is presented in which the encoder and decoder are partitioned to perform distinct processing in separation tasks. The encoder analyzes features, and the output of the encoder is split into the number of speakers to be separated. The separated sequences are then reconstructed by the weight-shared decoder, which also performs cross-speaker processing. 
Without relying on speaker information, the weight-shared network in the decoder directly learns to discriminate features using a separation objective. In addition, to improve performance, traditional methods have extended the sequence length, leading to the adoption of dual-path models, which handle the much longer sequence effectively by segmenting it into chunks. To address this, we introduce global and local Transformer blocks that can directly handle long sequences more efficiently without chunking and dual-path processing. The experimental results demonstrated that this asymmetric structure is effective and that the combination of proposed global and local Transformer can sufficiently replace the role of inter- and intra-chunk processing in dual-path structure. Finally, the presented model combining both of these achieved state-of-the-art performance with much less computation in various benchmark datasets.

Main paper: 
 introduction: 
 For the well-known cocktail party problem [14, 3] , single channel speech separation [30] has been improved since the introduction of time-domain audio separation network (TasNet) [46, 47] , which processes the audio separation in the latent space instead of the short-time Fourier transform (STFT) domain, as shown in Figure 1 (a). In particular, in most speech separation methods, the process of separating the feature sequence for each speaker is typically positioned at the final stage of the network, as shown in Figure 1 (b), which we refer to as late split. Therefore, a single feature sequence must encode all the information for all speakers to be separated. In addition, experimental results have shown that TasNet employing a convolution-based audio encoder/decoder performs better when the kernel length of the audio encoder is shortened [47, 45] , which requires modeling a long sequence. Indeed, expanding the sequence in channel and temporal dimensions is necessarily beneficial, since the separation process must include all the information for all speakers in the feature sequence. As a solution to modeling long sequences, DPRNN [45] was proposed using a dual path model, in which it segments long sequences into chunks and processes in terms of intra-chunk and inter-chunk to model the local and global contexts. As a result, due to promising performances in modeling long sequences, many TasNet-based approaches have adopted the dual-path model and repeatedly achieved state-of-the-art performances in monaural speech separation [9, 66, 37, 38, 92, 57, 60, 51] . Meanwhile, some studies have tackled the high computational complexity of long sequences in the time domain approach and proposed using multi-scaled sequence models based on the recurrent or stacked U-Net structure [70, 32, 42] . They reduced the computations to some extent, however, they still could not show competitive performance compared to the dual-path method. However, most studies have focused on handling long sequences rather than addressing the fundamental inefficiency of TasNet's late split structure, where a single feature sequence must encode all speaker information without discrimination, creating an information bottleneck. Also, forcing the separator to generate all separated features at once before the audio decoder makes the task difficult and can easily lead to local minima during training. To alleviate this challenge, we propose a more intuitive approach: expanding the feature sequence to include a dimension for the number of speakers in advance, as an early split. By splitting features earlier in the separator (Figure 1 (c)), we adopt an asymmetric strategy where the encoder and decoder perform distinct roles. The encoder process a single feature sequence before the split layer, similar to conventional separators. After splitting into multiple sequences, the decoder focuses on capturing discriminative characteristics between features using weight-sharing blocks [4, 11] . By employing this early split with a shared decoder (ESSD) structure, we ease the burden on the separator's encoder. This approach aligns with common practices in multi-channel speech processing, where processing is divided into two stages. Coarse separation is achieved through spatial filtering [31, 85, 24] , followed by post-enhancement to refine results [80, 8, 81, 41] . Furthermore, dual-path model itself also has redundancy because the segmentation process may increase the amount of computation by twice when the overlap between adjacent chunks is set to 50%. Also, the inter-chunk blocks in the dual-path model are inefficient because their role is mainly to capture the global context. Therefore, we design unit blocks for both global and local processing, integrating them effectively to replace the dual-path model and directly process long sequences without chunking. Both of global and local blocks are based on Transformer block structure [23] where multi-head self-attention (MHSA) module and feed-foward network (FFN) module are stacked. As a global Transformer block, we modified MHSA module in Transformer block as an efficient gated attention (EGA) module to mainly capture the global dependency without redundancy. On the other hand, as a local block, the MHSA is replaced with convolutional local attention (CLA) to capture local contexts. Consequently, we present the Separation-Reconstruction Transformer (SepReformer) for more efficient time-domain separation. Based on the ESSD framework and efficient global and local Transformer unit block, the SepReformer employs an asymmetric encoder-decoder structure with skip connections based on a temporally multi-scaled sequence model. The encoder processes a single feature at different temporal resolutions, and each intermediate feature is used for skip connection. The decoder then gradually reconstructs fine-grained information from the temporal bottleneck features, focusing on the discriminative characteristics of separated speech with auxiliary loss. To achieve this, the weight-shared decoder is trained to discriminate between the features separated by the encoder. In addition, a cross-speaker block is utilized in the decoder to facilitate information interaction between sequences, as described in [15, 40] . Furthermore, we design unit blocks for both global and local processing, integrating them effectively to replace the dual-path model and directly process long sequences without chunking. The experimental results demonstrated that the ESSD is effective especially with a small network. Also, comprising the separator network with the proposed global and local blocks can sufficiently replace the inter-and intra-chunk processing in dual-path structure, suggesting effectiveness for long feature sequence processing in speech separation. As a consequence, the proposed SepReformer that includes both of these achieved state-of-the-art (SOTA) performance with much less computation than before in various benchmark datasets. 
related works: 
 TasNet Conventional source separation has been performed in the STFT domain [30, 36, 12, 44] . In the time-frequency representation, a separator is modeled to estimate mask values or direct output representations. Then, the inverse STFT (iSTFT) is operated for the output representations to obtain separated signals [30, 36] . On the other hand, TasNet [46] replaces STFT with a 1D convolutional layer. Based on the encoder representation, mask values or direct output representations [65, 59, 40] are obtained in the separator. Then, the output representations are decoded by the audio decoder of 1D transposed convolution instead of iSTFT. Also, unlike the STFT, the convolutional encoder turns out to work well in a much shorter kernel size. Therefore, TasNet requires the separator to process the much longer sequences. Therefore, instead of an LSTM-based separator [46] , Conv-TasNet [47] is proposed based on a temporal convolutional network (TCN) [71, 39] to design a separator for longer sequence, showing impressive separation results. Dual-path model for long sequence After Conv-TasNet, the dual-path model is extensively employed to handle long sequences. In the dual-path model, the sequence is segmented into smaller chunks, and the sequence is processed alternately as intra-chunk and inter-chunk, effectively interleaving between local and global contexts. This dual-path strategy has shown promising performance in TasNet and has been repeatedly adopted [9, 66, 37, 38, 92, 57, 60, 51, 53, 34] . Especially, it is shown that, compared to various efficient attention mechanisms [76, 2, 35] , using the dual-path model with the original self-attention mechanism of Transformer [72] is effective for long sequence [67] in speech separation. However, modeling with the dual-path method can double computational costs due to the 50% overlap between adjacent chunks. The inter-chunk blocks in this model are somewhat redundant since they mainly capture global context. To reduce this redundancy, the quasi-dual-path network (QDPN) [59] replaces inter-chunk processing with downsampling. Inspired by QDPN, we design EGA and CLA modules to capture the global and local contexts without chunking process. Multi-scale model for efficiency Instead of the dual-path model, based on U-Net structure [61] , some studies have suggested using multi-scaled sequence model [65, 49, 21, 32, 42, 7] . SuDoRM-RF model [70] used a stacked U-Net structure to reduce the computational cost. The SuDoRM-RF approach can be regarded as a substitution of the TCN block in Conv-TasNet with U-ConvBlock as U-Net sub-block. Although SuDoRM-RF reduces the computational cost, it still has the disadvantages of having a fixed receptive field size and not considering the global context. More recently, TDANet [42] has efficiently improved performance with top-down attention and unfolding as in A-FCRNN [32] . However, these conventional methods with multi-scaled sequences prefer stacked or recurrent structures with U-Net sub-block to improve performance. Instead, we consider a single U-Net architecture and explicitly divide the roles of encoder and decoder as separation and reconstruction. Discriminative learning Weight-sharing neural networks widely used in modern contrastive learning [11, 6, 25] including speaker verification [90, 58, 17] . On the other hand, some studies on speech separation proposed to exploit speaker identity as discriminative information to address the case that the similar voices are mixed [52, 51, 89] . Therefore, we utilized the weight-shared network to reconstruct separated speech by extracting distinct speech representations for corresponding speakers. To separate the mixture, weight-shared decoder directly learns to focus discriminative features without the need for additionally designed, for example, speaker loss using an additional speaker embedding extractor. As a result, based on discriminative learning, the weight-shared network in the decoder strengthens the dominant speaker's components on each separated sequence, respectively. 
overall pipeline: 
 When input mixture x ∈ R 1×N , the 1D convolution audio encoder, followed by GELU activation [28] , encodes x to the input representation, as X = E(x) ∈ R Fo×T ,where F o and T denote the number of convolutional filter of encoder and the number of frames, respectively. The kernel and stride size are L and H, respectively. Then, the J output representations Y j are estimated from the separator and decoded by the audio decoder, expressed as ŝj = D(Y j ) ∈ R 1×N , 1 ≤ j ≤ J. Following the recent works [60, 79] , we design the separator to directly map the output signals instead of masking.𝑇𝑇/2 𝑅𝑅-1 T F 𝑇𝑇/2 𝑅𝑅 T 𝑇𝑇/2 𝑅𝑅 𝑇𝑇/2The detailed architecture of the separator of the proposed SepReformer is illustrated in Figure 2 .The separator is constructed on the basis of the ESSD framework with a separation encoder and a reconstruction decoder in temporally multi-scaled U-Net structure.The input representation is first projected to F dimension by the input layer. The input layer is composed of the linear layer and Layer Normalization (LN) [1] applied to each frame independently. In the encoder, the projected feature sequence is successively downsampled R times from the sequence length of T to T /2 R . The downsampling is performed by a 1D depth-wise convolution (Dconv) layer with a stride of 2 and a kernel size of 5, followed by Batch Normalization (BN) and GELU activation [28] . Each encoder stage processes the single sequence feature by B E stacks of global and local Transformer blocks. Speaker split The encoded features in all stages of the encoder are expanded by the number of speakers J to transmit the speaker-wise features from the encoder to the decoder. Therefore, the speaker split layer is placed in the middle, and it commonly separates the intermediate encoder features used for skip connections as well as the bottleneck feature. As shown in Figure 3 , this module consists of two linear layers with gated linear unit (GLU) activation [20] . Each feature is then normalized by LN and processed by the decoder.Reconstruction decoder For temporal reconstruction, the upsampled sequence feature from the previous stage is concatenated with the skip connection followed by linear layer. Then, B D stacks of global and local Transformer blocks process the J feature sequences as a weight-sharing network to discriminate between the separated features. By incorporating the separation objective function into the weight-sharing decoder, the network directly learns to capture the discriminative features. Then, the output of the last decoder stage is projected back to F o dimension by an output layer. The output layer consists of two linear layers with GLU activation.Cross-speaker (CS) Transformer During the discrimination process by the weight-sharing decoder, speech elements can be mistakenly clustered into other speaker channels. As a result, it would be beneficial to attend to each other in order to effectively recover the distorted speech elements. Therefore, to improve the interaction of contexts between speakers within the decoder, we incorporate a Transformer-based CS module as in [15, 40] . Based on MHSA module without positional encoding, the CS block performs an attention operation on speaker dimension while temporal dimension is processed independently. Therefore, the CS block learns to identify the interfering components of the opposing sequences within the same temporal frame. For convenience, we call ESSD with CS as a separation-and-reconstruction (SepRe) method.Instead of the dual-path model based on chunking, we directly process a long sequence using global and local processing blocks, similar to QDPN [59] or Conformer [27] . In particular, global and local blocks replace inter-and intra-chunk processing, respectively. The design of the blocks follows a Transformer block structure to ensure structural effectiveness [23, 68, 86, 29] . This structure consists of two sub-modules: temporal mixing and frame-wise channel mixing. These modules are stacked together with a pre-norm residual unit [75, 55] and LayerScale [69] to facilitate faster training of deep networks. Also, in all residual units, we apply dropout [64] for regularization. Gated convolutional feed-forward network (GCFN) Instead of using the conventional feedforward network (FFN) [23, 72] for channel mixing, we improve it by incorporating temporal Dconv with a small kernel size of 3 and substituting GELU with GLU activation [20] as shown in Figure 4(a) . This GCFN can effectively process channel features by considering the adjacent frame context. Several studies also have demonstrated the effectiveness of these enhancements in FFN [63, 87, 77] .Global Transformer with efficient global attention (EGA) In Figure 4 (a), the global block consists of an EGA module for temporal mixing and GCFN. The EGA module is based on the MHSA with relative positional encoding [19] . However, to reduce the computation and focus on global information in the attention layer, the downsampled sequence is processed and upsampled back. Sequences T /2 r at all stages 0 ≤ r ≤ R -1 are downsampled to T /2 R , which is equal to the length in the bottleneck. To compensate for downsampling, the upsampled features are multiplied by the gate value obtained from an additional branch with a linear layer and sigmoid function σ. The simple strategy allows the effective capture of global contexts while maintaining local contexts.Local Transformer with convolutional local attention (CLA) For the local block, we design a CLA module based on 1D temporal convolution with a large kernel of K in Figure 4(b) . Inspired by [27, 84] , the CLA module first processes the feature with the Pconv layer and GLU activation to facilitate capturing local contexts attentively. After the temporal Dconv, two Pconv layers are used.They have a hidden dimension of 2F and employ BN and GELU activation.The objective function is given as scale-invariant signal-to-noise ratio (SI-SNR) [62, 46] defined asEQUATIONwhere γ j = ŝT j s j /∥s j ∥ 2 2 and ∥ • ∥ 2 denotes L2-norm. The clipping value of τ limits the influence of the best training prediction [89, 83] . Notably, the output of the decoder stages can be trained for progressive reconstruction as the feature sequences are already separated in the decoder stages as in the progressive multi-stage strategy [54, 91, 88, 16] . In particular, weight-sharing decoder in each stage can be trained clearly for discriminative learning with stage-specific separation objective. This multiloss strategy is also considered to guide intermediate features in audio separation [53, 5, 59, 60, 40] . Therefore, the source signal can be estimated as ŝj,r = D r (X⊙M j,r ) ∈ R 1×N when M j,r ∈ R Fo×T is estimated with additional output layers for L j,r and the nearest upsampling. ⊙ denotes an elementwise multiplication, and D r (•) is an auxiliary audio decoder, which is also additional required with additional output layers. Therefore, we can set the auxiliary objective function asEQUATIONwhere γ j,r = ŝT j,r s j /∥s j ∥ 2 2 . Note that, when calculating the output from intermediate features, we opt for masking instead of direct estimation because the temporal resolutions of the feature sequences are deficient. Then, the multi-loss can be set to L = (1 -α)L + α R r=1 L r /R. Moreover, we alternatively calculate the intermediate loss L r using the magnitude values of s j and ŝj in the STFT domain as it provided more stable training and no actual separated signals are required from the intermediate outputs. 
dataset: 
 We evaluated our proposed SepReformer on WSJ0-2Mix [30] , WHAM! [82] , WHAMR! [50] , and LibriMix [18] , which are popular datasets for monaural speech separation. To ensure generality, the mixtures in the test set were generated by the speakers that were not seen during training. For all the datasets, networks were trained with 4-s-long segments at a 8-kHz sampling rate while the model processes inputs of varying lengths in the evaluation. WSJ0-2Mix WSJ0-2Mix is the most popular dataset to benchmark the monaural speech separation task. It contains 30, 10, and 5 hours for training, validation, and evaluation sets, respectively. Each mixture was artificially generated by randomly selecting different speakers from the corresponding set and mixing them at a random relative signal-to-noise ratio (SNR) between -5 and 5 dB. WHAM!/WHAMR! WHAM!/WHAMR! is a noisy/noisy-reverberant version of the WSJ0-2Mix dataset. In the WHAM! dataset, speeches were mixed with noise recorded in scenes such as cafes, restaurants, and bars. The noise was added to get mixtures at SNRs uniformly sampled between -6dB and 3dB, making the mixtures more challenging than those in the WSJ0-2Mix. Libri2Mix In Libri2Mix dataset, the target speech in each mixture was randomly selected from a subset of LibriSpeech's train-100 [56] for faster training. Each source was mixed with uniformly sampled Loudness Units relative to Full Scale (LUFS) to get a mixture at an SNR between -25 and -33 dB. We used the clean version as in previous studies [9, 42] .We trained the proposed SepReformer for a maximum of 200 epochs with an initial learning rate of 1.0e -3 . We used a warm-up training scheduler for the first epoch, and then the learning rate decayed by a factor of 0.8 if the validation loss did not improve in three consecutive epochs. As optimizer, AdamW [43] was used with a weight decay of 0.01, and gradient clipping with a maximum L2-norm of 5 was applied for stable training. All models were trained with Permutation Invariant Training (PIT) [36] . When the multi-loss in Subsection 3.4 was applied, the α was set to 0.4, and after 100 epochs, it decayed by a factor of 0.8 at every five epochs. τ was set to 30 as in [89] . SI-SNRi and SDRi [73] were used as evaluation metrics. Also, we compared the parameter size and the number of multiply-accumulate operations (MACs) for 16000 samples. The number of heads in MSHA was commonly set to 8, and the kernel size K in the local block was set to 65. Also, we evaluated our model in various model sizes as follows: Note that we did not train the models multiple times, as the deviations in the results are negligible below the significant digits. All experiments were conducted on a server with GeForce RTX 3090 × 6. 1 : Experimental evaluation of SepRe method on the WSJ0-2Mix dataset. ML denotes the multi-loss. In (a), all the methods were trained with ML, and the numbers in the left and right of the '/' symbol were obtained for the tiny and base models, respectively. In (b), when ML was used for training, we indicated the numbers of parameters including the additional output layer for an auxiliary output for ŝj, which were denoted with asterisk * . Note that the additional output layers were not required during inference. we applied the SepRe method with multi-loss (ML) and evaluated on the WSJ0-2Mix dataset.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We described the detailed model architectures and explained tranining and model configuration in Section 4 and Supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closedsource models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
ablation studies of sepre method: 
 Decoder Design In Table 1 (a), we evaluated various decoder structures (See Appendix A for detailed structures) to validate the effectiveness of weight-sharing decoder structure. As shown in Table 1 (a), the computations increases about twice by using large decoder in late split or using early split methods. In particular, the model using multiple decoders after an early split yielded a performance comparable to that of using a large decoder after a late split. In contrast, by sharing a decoder after an early split, the separation result increased significantly, suggesting that the ESSD structure effectively discriminates between the separated features. This impact was more noticeable in the tiny models by showing increase of 1.5dB. Applying CS to ESSD improved the performance especially on the tiny model, leading to the SepRe mechanism. Although changing from a late split structure to an ESSD structure can increase computation if the channel size is kept constant, reducing the channel size allows us to still achieve better performance. This adjustment significantly improves computational efficiency in relation to performance. This is particularly evident when comparing the base model with late split to the tiny model with the proposed ESSD + CS (SepRe) structure. The latter model achieves a higher performance, with an SI-SNRi of 22.4 dB compared to 21.6 dB, while using fewer computations and a smaller model size, clearly demonstrating the efficiency of the model architecture.Effects of multi-loss Furthermore, we experimented with the effects of multi-loss on various decoder structures in Table 1 (b). Compared to a late split, the case with an early split increased more significantly with multi-loss because an early split structure could be trained with a clearer objective for discriminative learning using an intermediate loss at each stage. In particular, while applying only CS without multi-loss resulted in a marginal improvement, combining CS with multi-loss led to a substantial gain. The results demonstrated that stage-specific objective functions induce each CS-equipped weight-sharing decoder stage to effectively learn simultaneously how to discriminate between and attend to each other. As a result, our proposed SepRe method using ESSD and CS significantly improved separation performance by applying stage-specific objective functions and inducing progressive reconstruction of separated feature sequences. To validate the general applicability of the SepRe method, we incorporated the SepRe method with multi-loss into the original separators of Conv-TasNet [47] and Sepformer [66] and conducted experiments on WSJ0-2MIX. The experimental results in Table 2 demonstrated a significant performance improvement when ESSD was applied for both networks. Also, applying CS and multi-loss in addition to the ESSD framework improved the performance further, which confirms the effectiveness of SepRe with multi-loss.Depth of encoder-decoder In Table 3 (a), we experimented the depth of encoder and decoder to determine the optimal configuration in terms of the number of block repetition B E and B D . Generally, experimental results showed that using more blocks in the reconstruction decoder had a greater impact on performance improvement than in the separation encoder. It demonstrate that it is more important to discriminate the features more elaborately in weight-sharing decoder than to analyze the features in encoder in speech separation. In particular, optimal performance was achieved with B E = 2 and B D = 3, which were used as the common configuration for subsequent experiments.EGA module design Next, we validated our proposed EGA module by ablating its components (see Appendix B for detailed structures) in Table 3 (b). First of all, using vanilla MHSA on a long sequence without chunking was infeasible due to the extremely large computational requirements. Therefore, one approach was to perform downsampling before applying MHSA, similar to the method used in QDPN [59] . However, this naive approach had the drawback of losing detailed frame-wise information. Although another consideration was to simply multiply the features to reflect the finegrained frame-wise information, this method still could not significantly improve performance. In contrast, the optimal performance was achieved by estimating gate values based on a linear layer and a sigmoid function. As a result, it is shown that our proposed global Transformer with EGA module and local Transformer with CLA module have effectively replaced conventional sequence models with smaller computations. FFN module design Also, by improving the design of FFN with Dconv and GLU activation, we could achieve the significant improvement of performance with slight increase of parameters as shown in Note that we did not apply the ESSD structure and multi-loss to our separator in this experiment. We could observe that the network based on dual-path sequence models requires high computation resources in terms of MACs while multi-scale sequence models are more efficient. The recently proposed Mossformer based on efficient gate attention unit (GAU) mechanism [33] showed improved performance with relatively smaller computations compared to the networks with dual-path model. In particular, the proposed model showed improved separation performance with similar MACs, which demonstrated the capacity as a model for a long sequence. It also suggested that the proposed block can sufficiently replace the dual-path models with fewer computations. Furthermore, by combining the U-Net structure into global-local Transformer blocks, the network became more efficient with the similar separation performance.Finally, we compared our SepReformer models with existing separation models on various benchmark datasets in Table 5 . Although we evaluated SepReformer trained with standard pairs from the training set in Table 5 (a), SepReformer-L was trained with dynamic mixing (DM) [89, 66] for data augmentation and compared to other existing large models with DM in was a powerful model recently proposed. It demonstrated the efficiency of the SepRe method in speech separation. Also, SepReformer-M without DM in Table 5 (a) showed competitive separation performance on WSJ0-2Mix compared to the large models with data augmentation in Table 5 (b).In particular, SepReformer-L with DM achieved the SOTA performance of 25 dB of SI-SNRi on WSJ0-2Mix, showing significantly improved performance over other conventional methods.In Table 5 (a), the smallest SepReformer-T among the proposed models even showed significant improvements on WHAM! and Libri2Mix datasets compared to the conventional methods. It suggested that the proposed SepRe method can be efficiently applied to a speech separation task in general. Also, SepReformer-L with DM showed the SOTA performance on WHAM! and WHAMR! datasets, as well as WSJ0-2Mix, which demonstrated that the proposed method can be trained effectively in a large model. Figure 5 compares the separation performance of various existing methods in terms of SI-SNRi versus MACs on WSJ0-2MIX. From the figure, we can observe the significant effectiveness of the proposed SepReformer in the speech separation task with high computational efficiency. Especially, it is noteworthy that the SepReformer-T models outperformed the conventional Sepformer trained with DM with 10 times smaller computations.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Not only did we include the implmentation code in supplemental material, but also we are planning to release our code for our main experiments in public soon. 
conclusion: 
 In this work, we introduced the SepRe method, in which the asymmetric encoder and decoder perform separation and reconstruction, respectively. The encoder analyzes and separates a feature sequence, and the separated sequences are reconstructed by a weight-sharing network and a cross-speaker network. We demonstrated that the SepRe method can be applied to conventional separators in general and utilizing multi-loss significantly improves the performance. Moreover, we replaced the dual-path model with presented global and local Transformer blocks to address a long sequence.The separator using the presented unit blocks has shown enhanced separated results efficiently, and combining a U-Net structure to exploit the multi-scale sequence model has further increased the efficiency. Finally, not only did our presented SepReformer outperform the most conventional methods in speech separation even with almost the smallest computational resources, but our large models achieved SOTA performance with large margins compared to the conventional models on various speech separation datasets.Limitations and future work. Our study focuses on 2-speaker mixture situation to assess our models in various model sizes and in the extensive datasets including noise and reverberation. Consequently, we believe that further investigation is needed to validate for more than 2-speaker mixture scenarios. Also, an important future direction is to separate mixtures for an unknown number of speakers as it is impractical to assume that the number of speakers to be separated is known in advance. Finally, although we experimentally validated our SepRe method, we believe that further investigation is necessary to figure out its underlying mechanism. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 4 and supplemental material.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Section 4Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This appendix is organized as follows:• Appendix A describes the various decoder designs in Table 1 (a).• Appendix B illustrates the cases of ablating EGA module in Table 3 (b)• Appendix C experiments by comparing two speaker split layer designs.• Appendix D interprets the discriminative learning mechanism in reconstruction decoder.• Appendix E evaluates perceptual quality on WHAMR dataset.• Appendix F experiments on real-recorded reverberant mixture for overlapped speech recognition.In Figure 6 , the block diagrams of various decoder designs are illustrated, which were experimented on in Table 1 . In Table 1 (a), the case of a late split, corresponding to the first and second rows, processes a single feature sequence in both the encoder and decoder, forming a symmetric encoder-decoder structure. In contrast, the case of an early split with multiple decoders, shown in Figure 6 (b), has each decoder block processing the separated sequences from the encoder. Thus, Figure 6 (b) can be seen as a special case of a late split with a large decoder of 2F , where the large decoder performs group operation with the number of groups equal to the number of speakers J. On the other hand, in the ESSD method illustrated in Figure 6 (c), the decoder shares weights to process the early split feature sequences. Therefore, even without interaction between the separated feature sequences, the decoder can learn discriminative characteristics by sharing the weights. Furthermore, in the proposed SepRe method shown in Figure 6 (d), the decoder learns to attend to each other using the CS block to additionally recover the deviated elements. In Figure 7 , we drew a block diagram of ablation studies for EGA in Table 3 (b). In the fisrt row of Table 3 (b), MHSA is simply performed with downsampling and upsampling to reduce the sequence length. In the second row, the upsampled output sequences of MHSA with downsampling is multiplied to the input features before downsampling in order to reflect detailed temporal information. However, simply multiplying the input features does not improved the performances. To reflect the frame-level details to the upsampled feature, we added the linear layer and sigmoid function to make gate values, leading to our proposed EGA module.In designing the SepReformer architecture, a key consideration is whether to use a shared speaker split layer across all feature stages as illustrated in Figure 8 (a), or to implement distinct speaker split layers for each stage as in Figure 8 (b). Our hypothesis is that a shared speaker split layer would enable the reconstruction decoder to process consistently separated feature sequences. Conversely, assigning a unique split layer to each stage in the separation encoder could account for stage-specific feature variations, as shown in Figure 8 (b). Therefore, we conducted comparative experiments on these two configurations.Upon comparing the two approaches of the multiple and the shared split layer, interestingly, different tendencies are observed depending on the type of mixture, whether it was an anechoic clean mixture or a noisy or, furthermore, noisy-reverberant mixture, in terms of SI-SNRi and SDRi performance (as shown in Table 6 ). For the WSJ0-2Mix dataset, which comprises simple anechoic mixtures, the shared split layer method produced results that were comparable to, or slightly better than, those of the multiple split layer approach, in line with our hypothesis. However, for the WHAM dataset, containing noisy mixtures, the multiple split layer approach demonstrated improved results, with the performance gap becoming more pronounced on the WHAMR dataset, which includes noisy and reverberant mixtures. Based on these findings, we opted for the shared split layer approach in most cases, as it provided competitive results while reducing the model's parameter count. On the other hand, for the SepReformer-L with DM in Table 5 (b), we employed multiple split layers to enhance performance, particularly on the WHAM and WHAMR datasets.To analyze the roles of each transformer block within the shared decoder of the ESSD framework with multi-loss application, we calculate the cosine similarities between the separated features from Z1 to Z4 as shown in Figure 9 . First, we observe that Z1 has higher cosine similarity compared to the others, indicating that the separated features initially share similar characteristics before being processed by the weight-sharing blocks. However, after the weight-sharing global block, the similarities in Z2 are generally lower compared to Z1. This result suggests that the weight-sharing global blocks capture discriminative characteristics, making the features more dissimilar in terms of cosine similarities. The further decrease in similarity in Z3 compared to Z2 and Z1 demonstrates the effect of the weight-sharing structure of the local block. As features pass through subsequent local blocks, more region-specific traits are refined. The local block, which handles these localized characteristics, further enhances the distinctiveness of the local features, resulting in decreased similarity.In contrast, features processed through the CS block exhibit increased similarity, unlike the weight-sharing structure. This increase in similarity can be understood as the separated features attending to and becoming more similar to each other, as the CS structure is designed to cross-reference information between the speech features. During this process, the CS block preserves distinct features and restores degraded information by leveraging mutual information. As the weight-sharing structure emphasizes unique characteristics, the split features can deviate from the original characteristics of the speech within the same frames, where the influence of each speaker's information is similar. This deviation occurs because emphasizing features in such frames can increase interference from other speakers' speech components, potentially distorting and degrading these features. Therefore, the CS block after the weight-sharing block is expected to recover the deviated features by attending to each other within the frames. While SI-SNRi is often the primary metric for speech separation tasks, it's important to also report perceptual metrics like PESQ and eSTOI, especially for datasets like WHAMR! that include noise and reverberation. Table 7 shows that SepReformer-L demonstrates effective performance compared to TF-GridNet [78] , with a PESQ of 2.79 and an eSTOI of 0.799, slightly improving upon TF-GridNet's scores of 2.75 and 0.793. No Processing baseline performs poorly, as expected, with a PESQ of 1.41 and eSTOI of 0.317, reflecting low perceptual quality and intelligibility without any enhancement. TF-GridNet shows notable results with SI-SNRi of 10.6 dB, while SepReformer-L shows slightly better performance across both perceptual and signal-level metrics. This suggests that SepReformer-L is also effective in maintaining perceptual quality and intelligibility in challenging environments, making it a relevant model for real-world applications.To further validate the applicability of the proposed model in real-world environments, we evaluated word error rate (WER) on the LibriCSS [13] dataset using baseline speech recognition model. The dataset consists of recordings that simulate real meeting scenarios, allowing us to assess the separation performance of model as a pre-processing step for overlapped speech recognition. The LibriCSS data set enables the evaluation of varying levels of overlap, ranging from 0% to 40%, which helps not only in measuring separation performance but also in verifying the model's ability to preserve speech quality when overlap is minimal. Accordingly, the proposed model was trained using source-aggregated SDR (SA-SDR) [74] instead of conventional averaged SDR, with varying overlap ratios to account for different levels of overlap during training. For comparison, we additionally trained and evaluated the DPRNN model as a competitive baseline. Since LibriCSS dataset does not provide a training set, we trained the model using LibriSpeech [56] for speech source with noise simulated using colored noise and reverberation based on room impulse response (RIR) simulations using gpuRIR [22] . Although the evaluation dataset includes both reverberation and moderate background noise, we excluded dereverberation during training to ensure stability. Thus, the model was trained to focus solely on speech separation and noise reduction.Referring to Table 8 , we can observe that the word error rate (WER) increases as the overlap ratio (OV) rises for the unprocessed input. Even in cases with no overlap, the baseline results show around 10% WER due to inherent distortions from reverberations and background noises. The BLSTM [13] and Conformer [10] models, which are based on STFT and real-valued masking, show some improvement at higher OV ratios, but their performance tends to degrade compared to the input when the OV is low, indicating unstable results. This instability could be attributed to the negative effects of attempting to remove reverberation as well. In contrast, the results of the DPRNN model demonstrate consistent improvements over the unprocessed input across all overlap ratios. The proposed SepReformer model further improves performance. This suggests that the speech separation was performed effectively. However, since reverberation was not removed, the results show less improvement in the OV0 condition, which can be interpreted as the model successfully preserving the original speech without unnecessary distortion.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The paper presented sepformer which uses global and local transformers with EGA and CLA to effectively model the global and local information for speaker separation. The proposed method has shown better results comparing to SoTA. The authors also provided additional results to address reviewers’ concerns. However, the presentation of the paper still needs improvements and the novelty is good but not breakthrough. Hence I recommend an accept.

Reviews: review: paper_summary: paper_summary: The authors propose a neural network architecture to separate a speech mixture containing 2 speakers.  The proposed U-net based architecture replaces the inter and intra chunk processing - a popular method for speech separation - with global and local attention mechanisms. They further propose a mechanism to reduce the computational cost of the model. The model does the speech separation early in the network and uses a decoder with a low parameter count - all thanks to its weight sharing strategy - to do the speech separation.  Evaluation on simulated data shows improvement over the state of the art methods.
strength_weakness: strength_weakness: 1. The authors have shown good results on a bunch of datasets, and the separated audios in the supplemental file are of high quality. But all these results are based on simulated data, so it makes you wonder how well the model would do with real data. It’d be great if the authors could show how the model performs on the Chime-6 dataset or Libricss, maybe through WER metrics or even objective speech perception measures after separation. Or at least, they could show us some samples after separation on real data.
questions: questions: 1. Looking at Figure 2, I’m wondering if it’s really a good idea to use the same split network for all ‘r’ values.  Would it be better to have a different ‘spk’ split for each ‘r’ ?

2. I believe Eq(1) should be a minimum of the Si-SNR and \tau since you refer to it in line 201?

3. In line 213, shouldn’t the multi loss be (1-\alpha / R) L + \alpha \sum_r L_r/R ?

4. It is not very clear  why the authors refer to the decoder as a Siamese decoder. Is this because they share the same weights across all speakers? This is typically how a speech separation network is structured. I don't see any contrastive learning loss functions as part of the training loss to discriminate the speakers.
limitations: limitations: Lack of evaluation on real data is a limitation of this work.
paper_summary: paper_summary: The paper proposes SepReformer, an efficient time-domain separation network. The model is an encoder-decoder architecture that splits the output features of the encoder based on the number of speakers before feeding them to the decoder. Both encoder and decoder networks are comprised of transformer blocks that capture global and local characteristics of the signal in different time-scales. The proposed approach achieves state of the art results in 3 well-established datasets.
strength_weakness: strength_weakness: 1. Testing on clean data. It would be interesting to see how the proposed model performs on noisy datasets.
2. Evaluation of the approach to the two-speaker separation problem.
questions: questions: 1. How does the early split strategy affect the overall computational efficiency compared to late split and other conventional methods?

2. Why was the alpha value set to 0.4, and have you experimented with other values to determine the optimal setting?

3. How does the model handle varying lengths of input sequences, and what is the maximum sequence length it can effectively process?

4. Are there any limitations or failure modes of the proposed method that have been identified during experimentation?
limitations: limitations: 1. The paper does not discuss potential failure modes or limitations observed during experimentation,
paper_summary: paper_summary: This paper presents a novel approach to time-domain speech separation, departing from the conventional chunk-based dual-path processing. The authors introduce an asymmetric encoder-decoder architecture, where the encoder analyzes features and splits them based on the number of speakers. A Siamese decoder reconstructs the separated sequences, learning to discriminate features without explicit speaker information. The use of global and local Transformer blocks for long-sequence processing eliminates the need for chunking, contributing to a more efficient and effective model.
strength_weakness: strength_weakness: The paper presents some interesting ideas, but their novelty and significance are questionable:

1. Transformer Usage: While the use of Transformer blocks is highlighted, similar architectures have been successfully employed in previous works like Sepformer, raising questions about the uniqueness of this contribution.
2. Limited Evaluation: The experimental results primarily focus on two-speaker separation, which is considered a relatively solved problem in the current state-of-the-art. The absence of evaluations on more challenging scenarios with three or more speakers limits the generalizability and impact of the findings.
3. Incomplete Comparison: The paper's claims of achieving state-of-the-art results are undermined by the lack of comparison with other important papers in the field. Notably, models like the "DIFFUSION-BASED SIGNAL REFINER FOR SPEECH SEPARATION" have reported superior performance (SI-SDR of 23.1dB), raising concerns about the validity of the SOTA claim.
questions: questions: --
limitations: limitations: --
paper_summary: paper_summary: This paper proposes a novel transformer based architecture, sepreformer, for speech separation based on a siamese decoder network that operates on separated speech signals in the encoded space.
strength_weakness: strength_weakness: The first major weakness is the presentation quality of the paper. Starting with the abstract, which goes right into a discussion of feature length and computation without setting up the problem and overview. I think a lot of the high level picture is missing, especially related to the particular insights that help this method work better.

Beyond the presentation, a main concern I have of this method is extending to beyond 2 sources. The experiments focus on two-speaker separation. It's unclear how well the method would generalize to scenarios with more than two speakers, which is an important real-world use case. It's also particularly important here because the authors propose separating the sources first and then using a siamese network for decoding the features back to speech. With more than 2 speakers, it's quite likely that the method would have a harder time as the features for each source get separated out earlier in the process.

The experiments and comparisons primarily focuses  on SI-SNRi. I'd like to see other metrics like PESQ or STOI considered as SNR does not always represent perceptual quality.
questions: questions: Are there any experiments conducted on more than 2 speakers? Are there any metrics considered beyond SNR?
limitations: limitations: No

Comments: title: Thanks for addressing the comments.
comment: I thank the authors for providing their rebuttal which address my concerns. I will retain the Accept score.
Thank you for your responses. After considering your rebuttal, I maintain my score.