Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies masking-based audio-visual source separation, which predicts a complex spectral mask for the audio mixture for each speaker conditioning on the mixture speech and the video of the target speaker. The authors proposed BFRNet, which is composed of a audio-visual source separation model that predicts a mask for each speaker, and a filter-and-recovery network (FRNet) that refines predicted masks to a) remove non-target residual speech and b) to recover target speech removed from the initially predicted mask.
strength_and_weaknesses: Strengths

- BFRNet yields better performance on VoxCeleb2, LRS2, LRS3 compared to prior works
- The mask refinement module, FRNet, is complementary to other masked-based audio-visual speech separation models according to the experiments in Table 2 and 3.
- Ablation studies confirm that both the filter and the recovery module contribute to the improvement. It also shows video facilitates the filter network, and using the cleaned mask to the recovery module improves.

Weaknesses
- While the empirical results are strong, this paper can be improved by providing more explanations of why the FRNet further improves the base audio-visual speech separation module. 
  - Why does the filter network remove the non-target speakers’ voice better than the base separation module? The filter network only takes predicted mask and video features as the input. It does not even take the mixture audio as the input. It is surprising that it can tell where the noise is by just looking at a mask.
  - Similarly, why does the recovery network learn what to keep better than the base audio-visual source separation network?
  - It is possible that adding FRNet improves because now the entire model is bigger than the one without. To justify the FRNet carries benefits other than increasing the model capacity, the authors are recommended to compare with a stronger base model (e.g., Basic Audio-Visual Speech Separator) that have a deeper encoder/decoder where the parameter counts match roughly that of the BFRNet.

clarity,_quality,_novelty_and_reproducibility: - Notation can be improved
  - The authors describe that “[we] feed $V_i$ and $Au_i$ to AV-Fusion module to obtain a[n] enhanced feature.” Why is the audio feature $Au_i$ speaker dependent? According to the diagram it is the output from the encoder that takes mixture spectrogram X as input.
  - In Sec 3.3 Recovery Net, why are M_{1, \cdots, S} referred to as “coarse” masks? Do they have different temporal or frequency resolution?
  - The paper can use more proofreading. There are still some typos.
- Experiments are good but can be improved as suggested in the Weakness section.
- The idea of refining masks for source separation is novel
- The proposed method appears to easy to reproduce


summary_of_the_review: Experimental results are strong compared to the prior work. The main novelty is a module for mask refinement, which can improve several existing audio-visual speech separation models as demonstrated by the authors. More controlled experiments are required to justify the gain does not simply result from the increase in the number of parameters.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents an audio-visual speaker separation method suitable for multi-speaker scenarios. An audio-visual speaker separator is first introduced which is followed by a filter-recovery network which aims to further improve the separation quality. The latter can also be combined with other separator networks and it is shown that it can improve their performance as well. The proposed approach is tested on standard benchmarks and it  outperforms other state-of-the-art methods.  
strength_and_weaknesses: Strengths

Several results and an ablation study is presented which convincingly show that the proposed approach outperforms other existing approaches.

The proposed FRNet can improve the performance of other speech separation approaches and this is a nice contribution.

Weaknesses

First of all, writing can be improved. Although the text is understandable there are several errors/typos. More details in the next section.

The impact of facenet/lipnet is missing in the ablation study. It would be good to show how important each of them is. Since most of the information is contained in the lip movements the contribution of facenet might be small.

The authors emphasise that one of the main contributions is that the proposed method can separate mixtures with a variable number of speakers simultaneously during training. Isn’t this something the existing approaches can already do? And also the number of speakers needs to be known in advance. It’s not clear why this is an important contribution.

It is not explained why the authors chose a two-step approach instead of an end-to-end approach, i.e., why not integrating the filter and recovery networks in the speech separation model and optimise them jointly?

The proposed audio-visual speech separation network (without the FRNet) is very similar to other existing audio-visual approaches. Why is this model better? This is also not clear.

Some details are not explained, e.g. the lambda coefficient in eq. 13 is set to 0.5, how is this value chosen? Why is training performed for 19 epochs? Why 2.55 seconds are used for training? It seems these numbers have been chosen via an optimisation stage but it is not explained how this was performed.

clarity,_quality,_novelty_and_reproducibility: It is easy to follow the paper but proofreading is needed. A non-exhaustive list of typos from the introduction  is the following (but can be found in all sections):
Firstly, Filter module -> Firstly, the filter module
the Recovery module use -> uses
most of works -> most works
Specifically, Specifically -> specifically

Fig. 2 is a bit confusing. It’s only understandable after reading the text. It would be better if all the necessary information to fully understand it is contained in the figure or caption.

The main novelty is the introduction of the FRnet. It is shown that if combined with other existing speech separation networks leads to improved performance.

It is not possible to reproduce all the results without help from the authors. The test sets are generated by random sampling.

summary_of_the_review: Overall, this is an interesting contribution but there are several weakness as explained above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper points out that there are two prominent issues in complex multi-speaker separation results: 1) There exist some noisy voice pieces belonging to other speakers; 2) Part of the target speech is missing. A Filter-Recovery Network (FRNet) is hence proposed to solve these problems. The authors also emphasize that their single model can separate voices for a variable number of speakers, which is simply achieved by proper training data setting (i.e., including mixtures with a different number of speakers). Overall, the model design is quite interesting with a good performance improvement.
strength_and_weaknesses: This paper first points out that there are two issues in complex multi-speaker separation and solve these two issues with the proposed Filter-Recovery Network (FRNet). The model structure of FRNet is reasonable, but there are still some unclear parts that need to be addressed:

1. The loss function used to train the proposed model is only based on e.q. 13 without any constraints, how do you guarantee the Filter Net and Recovery Net work properly as their original goals (removing noise voices and extracting missing voices, respectively)?
2. Some audio samples should be released in the supplementary material, especially the separated result at each stage: i.e., a) after Basic Audio-Visual Speech Separator, b) after Filter Net and c) after Recovery Network. This can also somewhat solve my first concern.
3. The inputs of the Filter Net are Vi and Mi, why the mixture X doesn’t need to be fed into the model?
4. For the experimental parts, how do you calculate the scores (SDR and PESQ) if there are multiple output-separated speeches?

The proposed method to separate voices for a variable number of speakers with a single model is not very novel, to my understanding, it is mainly based on a proper setting of the training set.

comparison
Typo:
1.	Page 2, Specifically, Specifically, -> Specifically.
2.	Page 6, k-th speaker -> i-th speaker.

clarity,_quality,_novelty_and_reproducibility: This paper presents the problem and proposed solution quite clearly so that readers may be able to reproduce the results. The proposed model is simple with novelty. 
summary_of_the_review: The prosed method is motivated by two observed problems in the multi-speaker separation. FRNet can also be used as a post-processing module to improve the performance of different audio-visual separation frameworks is a plus. Overall, the paper is well written with good experimental results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is concerned with separating speech signals in a multi-speaker mixture and outputting the speech signals corresponding to each given face. In particular, the authors propose a method called BFRNet consisting of a basic separation module and a Filter-Recovery Network (FRNet). The FRNet filters the noisy speech and recovers the missing parts in the separation results of the basic separator. Experiments were conducted to valid the effectiveness of the proposed method. 
justification_for_why_not_higher_score: All three reviewers ranked the paper as "marginally above the acceptance threshold", and this matches the paper quality and contribution. 
justification_for_why_not_lower_score: This paper addresses an interesting problem and the proposed method is shown to be effective.
Decision: Accept: poster