Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors have proposed a deep learning-based framework for speech separation.
The main claim is the effectiveness of the top-down attention mechanism that works across multiple scales to recover the signal.
strength_and_weaknesses: Overall the paper is easy to follow, and the topic addressed is interesting.

- The proposed attention mechanism has similar ideas to the UNet autoencoder and other similar generative models in the area of empirical deep learning that combine evidence from multiple scales. The authors have not discussed such ideas in this paper.
- There is no theoretical discussion on the impact of individual modules and their interaction for overall performance gains. e.g., why adopt such a complicated pipeline?
Encoder: is Resnet style model with additional pooling layers at each layer. I see no advantage of this over a regular ResNet style architecture which also combines information from a previous block/layer operating at a different scale.
- The use of the transformer layer is also not clear. It seems like an unnecessary complication. Transformers are for sequence modeling, but I don't see such a requirement here. Also, the input positional encoding and training of a transformer using masking is very important for its effectiveness.



clarity,_quality,_novelty_and_reproducibility: Clarity is fine.
Novelty is acceptable.
Work is reproducible from the author's code.
summary_of_the_review: Please see my detailed comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new top-down attention framework for the task of speech separation. The knowledge is from human brain mechanism. The authors show that their method can achieve better separation results with a rather smaller model compared with recent approaches.
strength_and_weaknesses: Strengths: the authors work towards realistic app scenario for the speech separation task, which is a good point. The top-down attention is another good point which can be inspired by human brain knowledge.

Weaknesses: Beyond model size, the computational complexity is also unavoidable for practical uses on low-resource devices. It’s quite curious how the model behaves on the low latency (causal) mode, whether it is potential for a real-time processing speed. The design of TDNet also needs more discussions, seems the current one uses more existing architectures which can be interpreted as a top-down process. It would be better to analyze what the model learns in such a top-down process while previous methods did not. Or more analysis should be provided to show the insightful contribution of such a top-down network design.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly presented, with a full review of related research, the proposed idea is clarified. But there are a few flaws on writing, like in page 6, sec. 4, dataset name should be WHAM! instead of WAHM!. The authors should ensure the correctness of paper writing to avoid misunderstanding.

For the task of speaker separation, this work shows a worthwhile trial of approaching the target on a limited memory cost, which is less discussed in previous methods. This is good. And the use of top-down processes for the goal makes sense, which is surely inspired by related research on neurosciences. But a more thorough discussion/comparison of possible top-down attention options would be more persuasive and stronger for supporting the authors’ claim. Especially when there are abundant corpus of literature about top-down bottom-up attention in multiple AI fields, to name a few [1-3]. Principles can be learned from these works for a comprehensive assessment of the top-down bottom-up architecture.

Details about the proposed methods are sufficiently introduced by the paper, which is friendly to the readers for reproduction.

[1] Pinto Y, van der Leij A R, Sligte I G, et al. Bottom-up and top-down attention are independent[J]. Journal of vision, 2013, 13(3): 16-16.

[2] Jaiswal S, Fernando B, Tan C. TDAM: Top-Down Attention Module for Contextually Guided Feature Selection in CNNs[J].

[3] Wu T, Zhu S C. A numerical study of the bottom-up and top-down inference processes in and-or graphs[J]. International journal of computer vision, 2011, 93(2): 226-252.
summary_of_the_review: This paper provides a good trial on applying top-down bottom-up processes with MHSA on the task of speaker separation. The motivation of reducing model complexity is aligned with top-down bottom-up mechanism in human perception. However, the paper merely focus on intuitive design of network modules, without insightful & thorough analysis of the proposed top-down attention solution, especially compared with many related trials in other application fields. It is necessary to add more analysis for a wider interest in the ICLR community instead of solving a specific downstream task of speaker separation. As an efficient solution, it would be better if causal version performance can be shown for the potential of real-time processing. Paper writing should be carefully examined to avoid mistakes.

Although there are several flaws as mentioned, this work still proposes a novel architecture based on a clear intuition and achieves SOTA on more challenging scenarios of speaker separation task. It would be inspiring of this work in the field of speech processing. Hence I would suggest a weak accept of this paper to ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, a lightweight blind source separation model (TDANet) is presented. The model is inspired from the brain’s top-down attention architecture.   The separation is carried out using an embedder, a separation and a decoder (with the  same parameters for all speakers) on the time samples. 
The model is consisted with down and up sample units, global attention, and transformer to output acoustic features on this information and LA layers in the decoder.

strength_and_weaknesses: Experimental results show that although this model is small it gets competitive results to SOTA models which are larger less efficient. Furthermore, they show that each part of the architecture is essential for the best results
clarity,_quality,_novelty_and_reproducibility: see in the review
summary_of_the_review: Major concerns :

The bio-inspiration in this paper is emphasized many times although it is not clear at all to the reader. A short description can help understand the motivation for this approach. 

The main idea of this paper is to build an efficient model for speech separation. The minimum number of parameters in the proposed method is 2.3 M. It is compared to other method with more parameters. Interestingly, the SuDoRM-RF0.5x and SuDoRM-RF0.25x with less parameters are not mentioned – please explain.

B equals to 16, it is not clear if this is considered in the complexity measurement. 

The authors chose L=4ms and only 3 1d conv layers. The other models are using the same parameters ? if not, it is interesting to understand the contribution of these parameters to the method efficiency and performance.

The new LRS2-2mix dataset – It seems that the random method can chose for the same scenario 2 reverberated signals from different enclosures. Please elaborate on this.
 
Minor concerns :

To make it easier to follow, I think that it will be better to add the parameters names (G_m,F_i for instance) to the architecture figure. 

In the experimental study, only the improvement is reported. To better understand the performance of the model, the noisy signal should also be mentioned as a reference. For instance, if the noisy input’ SI-SDR is -10, 10 dB improvement does not improve the intelligibility so much.

The SI-SDR measurement reported in the training method (eq. 8,9) is different from the original paper (I guess it is only a typo but if not, we cannot compare the results to the other methods)

The Sepformer model on the wham! Is reported in their paper with sisdri=16.4 while in this paper it is only 14.4 – please check

It is not clear what is the frequency sample used (8KHz/16KHz ?)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Paper Summary:
 
The authors propose a bio-inspired method neural network architecture for speech separation, where a top-down attention and a local attention module are introduced to the U-Net based separation network. The proposed method has shown a better performance but with a significant less computation cost, than several recent separation networks. 

Novelty:
 
Reviewers have raised questions on the novelty of the paper, compared with baseline UNET. As the author pointed out, the main difference, i.e. the major contribution of the paper lies in the top down attention mechanism to speech separation. In additional experiment part, the authors further verifies the effectiveness of the top-down attention(TDA) by incorporating the SuDoRM-RF network with the TDA module. Together with the result for the result of the proposed model, method seems convincing. 
 
 
Effectiveness:
 
Some reviewers raised the question for causal processing, which is answered by additional experiment results added to the appendix. 
A question on the comparison with baseline model was raised, where the baseline models have different hyper-parameter setting in window length. This question was answered by additional experiment results.
 
Clarity:
 
The clarity of the paper is good. Some reviewers raised comments on insufficient explanation in bio-logical neural processing, which is later improved by the modification of the paper in corresponding section.
 
After additional experiment results and updates in description. The novelty, effectiveness and clarity seem good for publication in ICLR 2023   

justification_for_why_not_higher_score: The paper is a little speech-specific. it will be great that the work/idea can be generalized to more domains. 
justification_for_why_not_lower_score: The authors actively addressed all review comments by adding new experiment results. Overall the paper quality is good. The proposed method is effective. Compared to other existing methods, the proposed one has much less computational cost. Note that the database challenge from one reviewer may not be an issue because the authors used standard databases on the task.
Decision: Accept: poster