Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a two-stage model for multi-channel speech enhancement. The first stage predicts STFT domain filters for each microphone that will take them close to a reference microphone's clean signal, by estimating time-varying RTFs. This is called an alignment estimator. The second stage predicts further filters for each channel and filtered channels are summed up to obtain the estimated signal which needs to be close the target signal for the reference microphone. This second step can be seen as a typical deep beamforming network.

The model for each stage is a W-net model which is like a U-net model but uses complex matrix multiplications and convolutions.
strength_and_weaknesses: Strengths:
1. The idea seems novel.
2. It seems the two-stage model could work well by dividing the beamformer calculation into two separate stages and having individual targets (RTFs) for filters of the first stage.
3. It is nice that the method seems to be able to handle time-varying environments, but it has not been tested on such environments.

Weaknesses:
1. The experimentation is only on simulated data. Would the method work well for real data?
2. Comparison with another sequential multichannel method [Wang 2021] would have been nice. In this method, a beamformed signal based on an initial neural estimate is fed into a second stage (and even more stages) which refines the neural network output, hence the model makes use of spatial information implicitly.
3. The burden on the second stage is still high since it still needs to do some spectral denoising as well as spatial denoising.
4. The training uses a single room configuration which is limited. Does it generalize to unseen rooms?
5. U-net seems better than W-net itself (for mics > 2) in Figure 2. What if we used U-nets in AFNet instead of W-nets? It has been observed that using complex weights in a network is usually not beneficial, and it is better to use real valued weights within the neural network.

[Wang 2021] Wang, Zhong-Qiu, et al. "Sequential multi-frame neural beamforming for speech separation and enhancement." 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021.
clarity,_quality,_novelty_and_reproducibility: The paper seems novel in its approach. It also reads well and seems reproducible.
summary_of_the_review: The paper is nice and innovative but it has some flaws in experimentation as listed in the weaknesses above.

Some specific issues are highlighted below:

1. In Figure 1, one of the align filters are always all ones (the one corresponding to the reference mic) and does not need to be estimated. It should be noted.
2. In Section 3, the discussion assumes a time-varying single frame STFT domain filtering to describe room impulse response filtering. However, typically RIRs are much longer than a single frame and we may assume they are time-invariant if there is no motion.
3.  Instead of SSNR, maybe report SI-SNR which is more commonly used. If using SSNR, maybe mention the segment length.
4. In Section 4.2.3, it seems obvious that alignment masks would give more spatially diverse coefficients when there is an RTF loss on them. When there is no such loss, it is not clear what those initial filters mean and how they contribute to overall system.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes an autoencoder based multi-channel speech enhancement framework. Overall the work is applied and mostly empirical in nature. The proposed framework is based on existing well-established ideas from empirical deep learning. There are no theoretical insights and the contribution in terms of the combination of existing ideas is relatively marginal.
strength_and_weaknesses: The Paper is very hard to read, where I found lots of details, equations, symbols being intermingled within inline text.

- I couldn't find the motivation behind the proposed framework. Why the choices are made, how they impact the learning process, and what extra they provide over existing studies.
- This is another classical example of an empirical deep-learning paper where authors have tried to combine various existing ideas without any theoretical backing.
- The link between speech production/perception as well as multichannel setting with the choices made is clearly missing.

- Authors mentioned that their work is [Inspired by the alignment concept in signal processing]. What is this alignment process? no reference is provided, neither any discussion on why this is important.
- The alignment and filter network are very similar. Why not just train a single big network with the same capacity?
  In experiments, authors have shown results with individual networks empirically, but it is unclear if the model capacity is similar or not.
-  ILD and ITD correspondence to the magnitude and phase spectra is exploited using a complex neural network. 
While working in complex STFT domain using conventional methods is well understood, the same is not the case with DNNs. In-fact their are a many works which have shown that the same task can be performed entirely in time domain by using learned filterbanks modelling raw waveform directly in an end-to-end fashion.
Complex networks are very difficult to optimize in general, especially for speech/audio tasks.
- Authors are suggested to avoid MSE loss as existing works have highlighted the drawbacks and shown better performance by using STOI or its variants for enhancement tasks. 
- I fail to understand the rationale behind split training. Why not joint training right from scratch? May be some visualization based on the loss landscape of the overall model (in complex and real settings) and what the network is learning in terms of filter responses/saliency maps or geometric properties of network weights is highly encouraged for explainability.


Experiments: 
- lots of details are missing, which makes it very hard to reproduce the experiments
- given the empirical nature of the paper, Authors should submit the code and pre-trained models.
- details about existing works with which comparison is made are missing. Why these were chosen? I only see FaSNet closely related to this work.
clarity,_quality,_novelty_and_reproducibility: Not novel enough or at par with the standards of ICLR.
summary_of_the_review: While the problem addressed is interesting, the proposed ideas are just incremental and not novel enough for the ICLR main conference.
Authors are recommended to resubmit to a suitable workshop, but the question about the theoretical contribution is the main weakness of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an align-and-filter structure for neural network-based multi-channel speech enhancement.
The paper claims that the proposed decomposition of a filter into an element-wise product of alignment gain and a filtering gain is critical for an improved performance. Furthermore, it is claimed that training with an RTF loss is critical for the success of the proposed method.
strength_and_weaknesses: I think the basic idea is sensible and relevant: adding the RTF-based loss to improve construction of a multi-channel filter. The presented results indicate that this has a considerable effect on the proposed method.

Some weaknesses which should be addressed
- "To the best of our knowledge, the only work that incorporates RTFs for SE is by Wang & Wang (2018), in which, however, the RTF estimation is used as an intermediate step to assist the conventional MVDR beamformer only." -- Many DNN-based speech enhancement papers estimate RTFs for speech enhancement. For example, "ADL-MVDR: All deep learning MVDR beamformer for target speech separation" is a relatively recent example which estimates a steering vector, which is an RTF. It is true that, in this instance, the vector is used to estimate the filter.
- Furthermore, very relevant work is the following paper: The PCG-AIID System for L3DAS22 Challenge (https://arxiv.org/abs/2202.10017). The system used there is using a structure very similar to the one used here (W-like, multichannel intermediate output and single-channel final output).
- I think it should be clarified that the AlignNet is not doing only temporal alignment, but also level adjustment. Even more critical, the results from B.1 should be included in the main text. One of the main questions for me was if the magnitude equalization is really important for this work, and I expected the phase to be the important part. B.1 clearly gives the answer: phase adjustment is sufficient. It's therefore not clear why is this not included in the main text.
- The above brings another questions. If phase alignment is enough, would it be suffice to have a phase alignment loss only for the final output mask, without using the intermediate mask? I would assume that this would be sufficient, and that having alignment in the middle of the W structure is not necessarily critical. However, it would be good to get clarification from the authors, and to understand if the two-stage structure is critical (or it's suffice to have a skip connection and a combined loss at the output).
- Leaky ReLU is used for real and imaginary components independently. However, the slope is not defined anywhere in the paper. This is interesting, since it may mean the complex gain values are relatively strongly pushed to be in the first quadrant, which is not very intuitive.
- Table 1 -- These results indicate that the proposed FilterNet fails on the original unaligned signals. Operating on aligned signals is expected to perform better, but FilterNet on unaligned signals fails completely. This is quite surprising, since multi-channel systems proposed in other papers work well without the two-stage process (align+filter). I'm curious if this failure could be related to the choice of nonlinearity (above).
- Section 4.2.2 / Table 2 -- It would be interesting to connect these results to Table 1, e.g., what's the gap of the AFnet compared to perfectly adjusted signals from Table 1.
- Some typos, e.g., "Nerual BF"

clarity,_quality,_novelty_and_reproducibility: The paper is relatively well written, with some typos and inconsistencies.
The novelty is relatively small, but sufficient and relevant.
Appendix has several relevant results, which should be included in the main text.
Components of the datasets are publicly available. However, training details seem to be omitted, and it may be relatively difficult to reproduce the exact results without the access to the code for data preparation.
summary_of_the_review: The paper claims that the proposed two-stage network with an RTF-based loss is a better alternative to single-stage multi-channel processing systems. Experimental results are promising.
However, there are several open questions which need to be addressed before publishing the paper.

---

The authors addresses most of my comments in their responses, and I've updated the score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new deep-learning-based method for multichannel speech enhancement.
The proposed method so called AFnet uses a W-net 2D architecture which considers as the input a multi-channel time-frequency representation and  which predict both the alignment mask and the filtering mask which minimize the reconstruction error.
strength_and_weaknesses: Strength:
-well written paper
-reproducible results (code and dataset)
-technically correct
-Convincing results


Weaknesses
-Not very original method
clarity,_quality,_novelty_and_reproducibility: This work is similar to previous work (2 of the 3 refs. are not cited in the current paper) and it corresponds to a slight modifications of the Yoyama et al, 2020 paper.

-KOYAMA, Yuichiro et RAJ, Bhiksha. Exploring Optimal DNN Architecture for End-to-End Beamformers Based on Time-frequency References. arXiv preprint arXiv:2005.12683, 2020.
-KIM, Hansol, KANG, Kyeongmuk, et SHIN, Jong Won. Factorized MVDR Deep Beamforming for Multi-Channel Speech Enhancement. IEEE Signal Processing Letters, 2022, vol. 29, p. 1898-1902.
-KOYAMA, Yuichiro et RAJ, Bhiksha. Exploring Optimal DNN Architecture for End-to-End Beamformers Based on Time-frequency References. arXiv preprint arXiv:2005.12683, 2020.

 Despite the better results in comparison to the state of the art, the authors should further justify their changes in the proposed method architecture through a theoretical study, ablation study, etc.

To my opinion, this is the main weakness of this paper.


summary_of_the_review: This is a well-written paper with reproducible results and a suitable evaluation methodology with freely available codes and dataset.

However, the proposed work is only an incremental contribution of a previously published work from Yoyama et al, 2020.

The authors could provide more arguments to explain in what their contribution is significant in comparison to the state of the art from a  theoretical point of view.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present an algorithm for multichannel speech enhancement. The algorithm, which the authors call align-and-filter net, is a 2-stage system, wherein, the first stage computes an approximate relative transfer function (RTF) between the multiple microphones with respect to a reference microphone, with the goal of aligning the multiple channels. The 2nd stage computes the enhancement filters. The first stage is trained specifically to estimate RTFs, and the second stage on enhancement. Results show gains on simulated datasets, but smaller gains on more realistic sets like Chime3, when comparing models trained w/ and w/o RTF loss.

The reviewers agreed that the idea of using 2 stages, with the first stage specifically focusing on RTF estimation is novel. The authors also present a range of results (including in appendix) that show consistent, but marginal gains compared to a similar model without RTF loss (the closest baseline, since they are trained in similar settings). 

The reviewers also pointed out prior works in this space. There are multistage approaches for enhancement in the literature (although not in the same form), which the authors have not sufficiently compared against. The authors also do not compare against more recent techniques that use beamforming together with neural nets (like predicting a mask for covariance computation). In their rebuttal, the authors argue that such approaches have to compute an inverse of the matrix, which can be ill-defined. While true, such approaches are widely used in practice. The reviewers also pointed out that the size of the matrix to invert is usually not large, as they are done independently for each channel (at least when the number of microphone channels is not huge). There are also several prior approaches that aim to estimate RTFs. This dilutes the overall novelty, although the specific form in which the authors put the system together is novel. Together with the observation that the improvements using the proposed algorithm are marginal, there is consensus among reviewers that the paper does not meet the high acceptance threshold.

justification_for_why_not_higher_score: Limited novelty, missing experiments and marginal improvements, as was discussed in the meta review.
Decision: Reject