Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new speech separation architecture based on asynchrony fully recurrent CNN (A-FRCNN). The paper first describes FRCNN variants inspired by biological systems and discusses implementing such techniques for speech separation. Based on this discussion, the paper proposes a novel A-FRCNN speech separation technique. The experimental results show the effectiveness of the proposed method in two speech separation benchmarks (WHAM! and Librimix), achieving state-of-the-art performance compared with DPRNN. This new architecture can be applied to other similar problems, e.g., speech enhancement, and the contribution of this paper in terms of the technical novelty, experimental effectiveness, and potential wide applications is significant. My major concern about this paper is the clarity of the presentation. I could not fully understand the definition of the stage, bottom-up, and top-down operations until the actual speech separation discussions in Section 3.2. These concepts are not familiar, especially for speech separation researchers, and I recommend that the authors provide concrete examples of these concepts and elaborate the discussions.

Other comments:
1. Is it possible to use a transformer as a node instead of CNN? Recently, transformer-based processing becomes popular in speech separation.
2. Please discuss the effect of the reverberation. The real condition always has to deal with the reverberation.
3. Section 3.3 is a bit trivial, and it can be removed (or reduced). There are several important discussions in the appendix, and they can be put in the main document instead.
main_review: originality:
The paper's concept of formulating speech separation problems with FRCNN is very novel for me. 

quality:
Proposed A-FRCNN with several considerations of the implementation options and experimental validations make this paper very strong. Also, the paper has various experiments, including the ablation study, the careful comparisons with u-net, and the comparisons of the other methods with public benchmarks, which makes the experimental conclusions very valid.

clarity:
As I mentioned before, it is difficult to read the first part of the paper as it does not fully define/introduce the node, stage, bottom-up, top-down, lateral connections. I recommend the authors provide more concrete examples of them.

significance:
The paper's architecture and discussions can be applied to other machine learning problems (at least other speech processing problems, including speech enhancement). Thus, the impact of the paper would not be limited to speech separation and can have more general impacts in the machine learning community.
limitations_and_societal_impact: The paper explicitly discusses the deficiency of the current techniques for more realistic scenarios.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents biologically inspired design of recurrent convolutional neural networks with bottom-up, lateral, and top-down connections. In contrast to a ‘synchronous’ update of all nodes according to connectivity, the biologically motivated unrolling of recurrent connections and information flow in time results in an ‘asynchronous’ update where different nodes get updated at different time steps with bottom-up, lateral, and top-down information flow.  These architectures are termed A-FRCNN (asynchronous fully recurrent convolutional neural networks).  A-FRCNN architectures can equivalently be viewed as feed-forward networks with specific weight tying.  These models are applied to the task of speech separation and state of the art results are obtained on two data sets.
main_review: Overall, the paper is well written with clear descriptions and explanations and presents strong results.  A few clarifying questions / comments:

* At inference time I’m assuming (but couldn’t confirm in the paper) the A-FRCNN architecture is applied as well, or do the A-FRCNN architectures only guide how the FRCNN model is trained and then at inference time the fully connected FRCNN model is used in synchronous mode?  If it is the former, that is A-FRCNN architecture is used for both training and inference, then the model architecture (and not just parameter updates at training time) is different from FRCNN, in which case is it fair to call the proposed approach asynchronous updating scheme of FRCNN?

* Given the biological inspiration for A-FRCNN, a control (aside from the two controls presented in the paper) that seems missing is a model that is similar to S-FRCNN but is without all the skip connections (thus restricting information flow to strictly one layer in one time step).  Did the authors try this model, or any specific reason for not trying it?  Similarly could there be variants where skip connections (both bottom-up and top-down) will skip in the time dimension (a connection between node N and N+2 will result in an unfolded model that connects node N at time T to N+2 at time T+2)?

* For a given FRCNN network, the proposed approach of asynchronous updates and corresponding network design essentially adds another dimension of possibilities to explore.  Can there be a data driven learning based approach to determine optimal design, or at least guide towards that?  This seems important, because as the results with various asynchronous configurations show, the gains are quite sensitive to the model architecture (and hence it is quite possible that substantially better configurations remain undiscovered).

* In Section 3.2.4 please clarify how is the mask M_i generated for i^th speaker.

* In terms of speech separation background, there have been strong graphical model based approaches such as
Steven Rennie, John R. Hershey and Peder A. Olsen
"Single Channel Multi-talker Speech Recognition: Graphical Modeling Approaches,"
IEEE Signal Processing Magazine, Special issue on Graphical Models, November 2010.

limitations_and_societal_impact: Limitations adequately addressed, and no negative societal impact I can think of.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new separation module for extracting multi-scale features and aggregating them for improving the performance of source separation models. The proposed architecture consists of an asynchronous update mechanism which operates at different time-scales and fuses the features in parallel at the latter stage. The authors claim that the inspiration behind their proposed model is the non-simultaneous stimulus of human neurons when they get triggered from some sensory activity. The authors provide quantitative results on widely used benchmarks datasets and show improvement over strong baselines. Finally, the authors conduct ablations studies to show the importance of each update scheme inside the proposed separation module block.
main_review: ### Pros
This is a well written paper, the authors have put significant effort into presenting in detail their proposed architecture both in text and with explanatory figures showing the differences between different update mechanisms. Moreover, the parallel with the biological aspect of how the neurons work is quite interesting and is something that is missing from many current models in the audio source separation community. The experiments seems quite extensive and solid by using multiple baselines and evaluating different variations of the proposed architecture. This is definitely a quality paper in terms of the experimental framework, the evaluation and the derivations. Overall, I believe that the paper has great potential if some concerns are addressed and could contribute significantly to the field of source separation.

### Limitations / Criticism
However, the paper has several limitations which downplay the true potential of the paper. Please see below my concerns and questions to the authors in descending order of importance (0 is the most important):

0. The authors claim that the inspiration of the proposed architecture comes from the morphology of the human neural activations. However, the submitted code is weirdly similar to the original implementation of the SuDo RM-RF [24] model which seems to be the backbone structure of the proposed models (in fact after checking with an online tool, the code for control 1 and 2 has > 80% **identical** lines with this file https://github.com/etzinis/sudo_rm_rf/blob/master/sudo_rm_rf/dnn/models/improved_sudormrf.py but also the proposed A-FRCNN has the same level of similarity). Although, using open source code is encouraged (**if it’s accredited appropriately**), in this specific case, it downplays the inspiration from the biological aspect and one could misinterpret the proposed architecture as an ad-hoc variation of a SuDo RM-RF [24] model.
1. The authors could build the inspiration of their module architecture by first identifying the limitations of the SuDo-RM-RF [24] and then identifying biologically plausible solutions. Moreover, I believe that such story would make more sense and it would definitely avoid the negative affect on the originality of the paper, especially to a reader who is knowledgeable wrt the SuDo-RM-RF [24] architecture and can immediately see the similarities of the two works. Also, not that important, but why did the authors prefer to use the Asteroid [18] implementations for training all the models when they had access to the original repository of SuDo-RM-RF [24] and other models?
2. The authors need to delve deeper into the details of their biologically inspire model. Specifically, I can find only two references wrt the biological inspiration [1, 3], whereas I can see whole paragraphs (Lines 34-45) where multiple things stated should be backed up by the corresponding citations. By doing so, and in combination with the limitations of used separation models, the authors can pass a clear message to the reader on why such biological mechanisms should be employed towards building better architectures.
3. Since the authors refer to the efficiency of their model (e.g. Line 13) and they also use the term “light” for their proposed architecture, I have the following suggestions. It is quite convenient to put one number (e.g. the number of parameters or the inference time) and compare the efficiency. However, the authors propose a complex architecture that might severely enlarge the actual memory footprint of a forward pass of the network (or backward) mainly because of the intermediate activations tensors. Moreover, the same could also apply for the FLOPs or any other computational aspect. The authors should also report several other factors like actual memory requirements, time consuming on training on GPU / CPU and floating point operations and compare with the SuDo-RM-RF [24]. Based on that, the group communication mechanism [1*] has been shown to be able to significantly reduce the number of trainable parameters by dividing the channels into groups and processing them independently (with shared parameters across the groups). The authors could also integrate it to their proposed method and compare their numbers since it has also been shown beneficial for the SuDo-RM-RF [24] and DPRNN [14] models in [1*, 2*].
4. The authors should also compare against stronger baselines with respect to the upper bound of the separation performance. A stronger SOTA architecture could be the transformer architecture proposed in [3*], which also has an open source repository.
5. Lines 221-222: “Scale invariance is ensured by normalizing and zero-mean before the calculation.” There is no need to normalize the targets and the estimates since scale-invariant signal-to-noise ratio (SI-SNR) [4*] is by itself scale-invariant. The weirdly noted term  $ A_{\operatorname{target}} $ in equation 7 ensures that the magnitude of the projection of the estimated source $ \hat{s} $ onto the target $ s $ would be irrelevant to SI-SDR (since $ A_{\operatorname{target}} $  is a vector it should probably be lowercase).
6. The citation for the SI-SDR in Line 217 is incorrect, it should be [4*] instead of [26] which is the SDR paper. 
7. Differences smaller than <0.1 dB are not either significant nor hearable, thus I would suggest to round up all those performance numbers to a one decimal precision (it would also make the Tables less cluttered).
8. Lines 82-83. "For instance, Sudo-RM-RF [24] uses a U-Net structure [20], obtaining good results on smaller-sized models". This is not true, SuDo-RM-RF uses a repetitive U-ConvBlock structure. Now, each U-ConvBlock uses a U-Net architecture. "Smaller sized models" is also loosely defined considering the efficiency analysis I also referred above (See point 3.) where the number of parameters is not the major concern.  

Typos that I found:
Line <>: 
159: Therefore → Therefore,
164: increase → increases
264: wham! → WHAM!

I am willing to increase my score if most of the most important above concerns are addressed by the authors since I truly believe that the paper has a great potential to change the way how we design neural network architectures for source separation. 

[1] Mark Bear, Barry Connors, and Michael A Paradiso. Neuroscience: Exploring the Brain. Jones & Bartlett Learning, LLC, 2020.

[3] Peter Dayan and Laurence F Abbott. Theoretical Neuroscience: Computational and Mathemati362 cal Modeling of Neural Systems. The MIT Press, 2001.

[14] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46–50, 2020.

[18] Manuel Pariente, et. al.. Asteroid: the PyTorch-based audio source separation toolkit for researchers. In Proceedings of the International Speech Communication Association (Interspeech), 2020.

[24] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo rm-rf: Efficient networks for universal audio source separation. In Proceedings of the Machine Learning for Signal Processing (MLSP), pages 1–6, 2020.

[26] Emmanuel Vincent, Rémi Gribonval, and Cédric Fevotte. Performance measurement in blind audio source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 14(4):1462–1469, 2006.

[1*] Luo Y, Han C, Mesgarani N. Ultra-lightweight speech separation via group communication. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021 Jun 6 (pp. 16-20). IEEE.

[2*] Tzinis E, Wang Z, Jiang X, Smaragdis P. Compute and memory efficient universal sound source separation. arXiv preprint arXiv:2103.02644. 2021.

[3*] Subakan C, Ravanelli M, Cornell S, Bronzi M, Zhong J. Attention is all you need in speech separation. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021 Jun 6 (pp. 21-25). IEEE.

[4*] Le Roux J, Wisdom S, Erdogan H, Hershey JR. SDR–half-baked or well done?. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2019 May 12 (pp. 626-630). IEEE.
limitations_and_societal_impact: The authors do vaguely discuss a few limitations of their work in the conclusion section but they can also include any of the points that I raised here (if they do not wish to address them in the rebuttal).

In my opinion, this work raises some potential negative societal impacts, especially environmental ones, that the authors do not discuss. For instance, the hours required to train all those models could be at least be stated in the appendix and how the authors improve upon the training time of the baseline models.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper introduce a novel method to separate mixture of  sounds. The paper introduce a new neural architecture called Fully Recurrent Convolutional Neural Network (FRCNN) which use neural network with lateral connections. The main variant of the proposed method is the A-FRCNN which use asynchronous updating. The main novel parts are: (1) novel bottleneck structure (2) removing some connection that seems to be redundant. 
main_review: The paper has interesting ideas however if found that it has marginal contribution to the community due to the following:

1. The proposed architecture is very similar to the previous one in the literature ״Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex״ [https://arxiv.org/pdf/1604.03640.pdf]

2. The improvement is marginal for WHAM! dataset - less then 1dB. For Libri2Mix the results in the paper is weird: in all of previous papers the DPRNN method get better results then Conv-TasNet. Here, the author provide 14.1dB for DPRNN and 14.7dB for Conv-TasNet. I'm sure that with the right training procedure the DPRNN will get better results. Therefore, i don't think that the proposed method improve the results by more then 2dB. 

3. The comparison is updated for 2 years ago, the DPRNN paper published at 2019. Furthermore, the paper luck of comparison with state-of-the-art results from previous publications such as Wavsplit [https://arxiv.org/pdf/2002.08933.pdf], Gated DPRNN [https://arxiv.org/pdf/2003.01531.pdf], SepFormer [https://arxiv.org/pdf/2010.13154v2.pdf], Sandglasset [https://arxiv.org/pdf/2010.13154v2.pdf]



limitations_and_societal_impact: The authors discuss the limitation of the proposed method.
review_title: null

Meta-review: This paper proposes to use a bio-inspired asynchronous fully recurrent convolutional neural network (A-FRCNN) for speech separation. In contrast to the conventional synchronous update, the authors argue that with asynchronous updates via the bottom-up, top-down and lateral connections in the network, the model can fuse information flows at various time scales for improved learning.  Experiments on WHAM! and Librimix demonstrate the effectiveness of the proposed approach.  The authors show that A-FRCNN can deliver high-performance results on these two datasets compared to some of the existing best performing models under the synchronous update mechanism.  While some of the results obtained by A-FRCNN are state of the art and some are slightly underperform,  A-FRCNN also offers high efficiency with fewer parameters.  The work is well motivated.  In the rebuttal, the authors also cleared numerous concerns raised by the reviewers.   Overall, this bio-inspired approach is novel and interesting.   I would recommend accept.  The authors should revise the paper accordingly based on suggestions in the reviews and discussion. 
Decision: Accept (Poster)