Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper examines an audio-visual approach for dereverberation, where dereverberation of speech is conditioned on RGB and depth images (either field-of-view or panoramic). It proposes a dataset for this task based on real-world 3D scans of homes, using Librispeech data. Real data is also used The model is based on a U-Net conditioned on embeddings extracted by a "visual acoustics" network doing direct spectrogram prediction. The method is evaluated using PESQ, WER for speech recognition, and EER for speaker verification on synthetic and real data. The audio-visual method is found to perform marginally better than an audio-only version of the model. The audio-only version of the model outperforms several baselines from the literature on synthetic data, with mixed results on real data.
main_review: ### Strengths:

S1) To my knowledge, this is the first audio-visual approach to dereverberation. I think it is interesting to see how much visual information can help.

S2) The proposed dataset seems quite useful. A lot of existing synthetic reverb for such audio tasks is generated using shoebox room simulators, so having more realistic environments seems quite useful. The accompanying data recorded in real environments using loudspeaker playback (such that the clean reference is available) is also quite useful, and I hope the authors plan to release a recipe for this data.

S3) The paper includes a lot of ablations, which I really appreciate. Though the focus of the paper is on audio-visual approaches to the task, I'm really glad the paper evaluates audio-only versions of the model, as well as other components such as the matching loss.

S4) The paper is clear and well-written.

### Weaknesses:

W1) I listened to the demos in the supplementary material, and I feel like I heard a fair amount of artifacts and distortion in some examples: for moderate reverb, the model predictions have a certain "buzziness" to them, and more extreme levels of reverb have a "twanginess" to them. Something that would make this paper stronger would be a subjective human listening evaluation, e.g. MUSHRA, which would measure these kinds of artifacts better than an objective measure like PESQ. Also, alternative loss functions may help reduce artifacts. E.g. L1 loss instead of L2 on spectrogram magnitude, or MSE on power-compressed spectrograms can provide more perceptually-relevant loss functions.

W2) The processing method of using 2.56s frames, then concatenating the non-overlapping parts together seems suboptimal to me. Why not use e.g. a Hann window to stitch the predicted patches with overlap-add, or invert back to the time-domain and do overlap-add and train through this?

W3) Where does the "random speech embedding sampled from the data batch." come from? How is it computed? Is this a speaker ID embedding?

W4) The improvements by adding visual information are marginal compared to the audio-only model. The audio-only model does perform better than the baselines, but degrades PESQ compared to other baselines on real data. To benchmark the audio-only baseline against the literature, it would be interesting to compare to competitive systems in e.g. the REVERB challenge. I'm also not very convinced that visual input really helps that much.

W5) Table 1 would be improved by adding trainable parameter counts. Is the proposed model only doing better because it has more trainable parameters? WPE itself has no trainable parameters, since it's just a model-based EM algorithm.

W6) The lack of noise in the training and eval data limits real-world applicability, but I understand the focus of the paper in only on dereverberation. I'm glad that the authors tested noise robustness with WHAM! data, but 20 dB is a pretty high SNR for noise robustness. It would be interesting to see how performance degrades with decreasing SNR.

### Minor comments

M1) I feel that this statement isn't quite right: "R is a function of...the relative positioning of the speaker and the listener." I would argue that an RIR can be a function of the absolute positioning of the speaker and the listener. E.g. what if a source is close to a wall, versus in the the middle of a room?

M2) For the real data, would it be possible to be more specific about the source-to-microphone distances used in the real recordings, beyond "near-field to mid-field to far-field."?

M3) Also concerning real data: would it be possible to more more specific for this statement? "For each location, we play around 10 utterances.". How many utterances? Gender balanced? Different speakers across rooms?

M4) Nit: U-Net instead of UNet?

M5) "While our model is agnostic to the audio source type,": depends on the training data.

M6) In Tables 1 and 2, I would say "anechoic speech (upper bound)" instead of "clean speech (upper bound)", because to me "clean" suggests there's additive noise.

M7) About the blind RT60/DRR/distance predictions: "After converging, the average errors for them are 3e-3s, 160 dB and 17.92m respectively." Are these absolute or squared errors?
summary_of_the_review: Overall, I vote for marginal acceptance. This is an interesting novel task (audio-visual dereverberation), and the proposed synthetic and real data are quite interesting and useful for the community. But the paper's results show that visual input provides marginal performance gain over the audio-only baseline. The audio-only baseline outperforms some benchmarks in terms of PESQ, WER for speech recognition, and EER for speaker verification on synthetic data, and WER and EER for real data, but I have concerns about the subjective listening quality of the demos, compared to other dereverberation algorithms I have listened to. Thus, I am concerned that this isn't a significant improvement over audio-only prior work, at least in terms of subjective listening quality.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors introduce a novel audio-visual dereverberation approach. They propose a Visually-Informed Dereverberation of Audio (VIDA) model for dereverberation. The authors also create synthetic/simulated datasets and real-world data for experimentation. Finally, they show the impact of the proposed VIDA model on several speech tasks including, recognition, enhancement, and speaker verification. The results are encouraging. The main contribution of this work is the use of visual information as an auxiliary input for dereverberation. 




main_review: While authors suggest visual information is helpful, it isn't reflected in the results. In Table 1, if we compare the "Audio-only dereverb." and "VIDA" results, the gains are very modest. The PESQ 2.32 to 2.37, WER-FT 3.76 to 3.66, and SID EER 2.61 to 2.40. This raises the question about the effectiveness of the proposed approach. I feel the majority of the dereverberation is still learned by the audio model. This should be investigated. 

Overall, the technical contribution is limited to a new dataset and the addition of ResNet for visual information processing. Similar approaches have been used in the audio-visual enhancement with lip information. 
summary_of_the_review: I believe the manuscript in its current form offers a very limited contribution to the research community, and the contribution of visual information in dereverberation is minimal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a deep neural net-based dereverberation algorithm that uses both audio and video modalities. Based on the observation that a visual scene captured by a camera conveys information that is related to room characteristics, the authors propose a visually informed audio dereverberation method that aims to extract clean, anechoic speech from reverberant speech. In doing so, they first construct a large audio-visual dataset synthesized using a 3D simulator for real-world scanned environments and LibriSpeech data. They then train deep neural networks that take as inputs both visual data (RGB and depth images) and audio data (reverberant speech), and output clean speech. When training, two types of losses - one for clean speech spectrogram estimation and the other for reverb-visual matching - are used. Through the experiments with several downstream tasks for speech, they showed that the proposed audio-visual dererberation method outperforms the baseline models, both for synthetic and real-world test data.
main_review: Strengths:
- first multi-modal (audio-visual) approach applied to dereverberation
- paper is well organized and straightforward to follow
- solid experiments with ablation studies and in-depth analyses
- supplementary materials provide useful information not explained/described in the main text

Weaknesses:
- some of the key arguments are not sufficiently validated
- contributions are marginal and limited
- experiments and evaluation are not thorough

The key idea in this work is that visual input captured by a camera conveys information that characterizes room acoustics. With all the experiments and ablation studies, however, I don't believe that the authors provide sufficient information that support the aforementioned key hypothesis. The authors did take into account this matter in two ways: first, they included a reverb-visual matching loss term to penalize a matching between random speech and visual input, but little experiment or analysis was performed on this loss term - varying the weight or using different loss functions, for example. With or without using a reverb-visual matching loss was included in the ablation studies, but the effect was not significant compared with other methods. The performance was better with the audio-only model even in speech quality metric (PESQ). The second way is to use a random image instead of using the image that matches the room impulse response. In such scenario, the images does not carry any information relevant to the room acoustics, and thus the performance should be worse than or comparable to that of the audio-only method. However, the performance is better in general and is even best for some metric. Further experiments and deeper analysis are warranted to investigate this issue.

Another point that I hope to be added is the comparison of the model efficiency. Adding a visual embedding network increases the number of model parameters and computational costs. The proposed method is likely to work in real-time scenarios, and possibly on small devices, the efficiency should also be considered.

Lastly, I'd like to suggest to include R-vectors approaches as another baseline for comparison.
summary_of_the_review: This paper proposes a multi-modal learning framework that is applied to speech dereverberation. The idea is interesting and somewhat novel, but the experimental validation is not thorough enough to support the key idea of reverb-visual matching nor to provide sufficient evidences that visual stream does convey information about room acoustics.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes an audio de-reverberation method which integrates (panoramic, rgb+depth map) visual input with a spectrogram U-Net to estimate a de-reverberated spectrogram and recover the original clean signal.
The proposed method is compared to several baseline (audio-only) de-reverberation methods on speech enhancement, recognition, and speaker verification problems using both synthetic and real data.
Several ablations of the proposed method are compared, along with some quantitative error analysis investigating the effects of distance and environment.
The proposed method appears to consistently improve on speech recognition and speaker verification, and perform about on par with prior work on speech enhancement (as measured by PESQ).


main_review: This paper takes an interesting and novel approach to de-reverberation.
The paper is well written, and each component of the method is well justified.
The qualitative analysis (e.g. figure 4) is a nice touch, and helps shed some light on the behavior of the proposed method (and baselines).
That said, I do have a few questions and suggestions.

My primary criticism is in the reporting of the evaluations (tables 1, 2, 3).
Each setting is summarized by a single average score, and the "best" number is written in bold with no discussion given to the distribution of scores on the test data or the practical meaning of score differences.
This may give readers an inflated sense of performance here: is PESQ=2.38 really any different from PESQ=2.37? 2.33?  (Likewise for WER, EER, etc.)
There is a brief mention of astronomically low p-values (1.56e-60?) which I expect are primarily driven by the size of the test set, and not the size of the effect.
I would strongly encourage the authors to do a more responsible job of reporting their results: include some error bars, or some other indication of the spread of the data!

The proposed method takes an interesting approach to optimizing both magnitude and phase (eqs 1 and 2) for the output spectrogram.
The Griffin-Lim algorithm is then used to refine the phase estimate, with a vague statement of this working better than directly using predicted phase or GL with random initialization.
While I won't ask for more experiments on this, I think the point deserves a bit more explanation than what is given here, especially since there is no ablation study on the phase reconstruction loss.  (Nor is there any discussion of hyper-parameter tuning for equation 4 beyond the settings reported in the appendix.)


I found Figure 5 (qualitative examples) to be pretty difficult to parse, and I say this as a reader with (probably) above-average experience reading spectrograms.
With a bit of squinting I can see what the authors are getting at, but it would be easier to read if the spectrograms were A) larger, and B) plotted on a logarithmic frequency axis so that the low frequencies (where most of the action is) occupy more of the visual real estate.


Finally, it was nice to see the results in table 3, which show quite a strong effect on the performance of the system depending on differences in distance and room geometry.
Not much is said about the geometric differences in the environments though, either for real or simulated data.
In particular, it looks from figure 2 as though there may be configurations of speaker/listener that have no direct path, in which one might say that everything observed is "reverb".
If this is the case, it would be interesting to see how performance (table 1) differs (if at all) in the presence or absence of a direct path.


summary_of_the_review: Overall, I enjoyed this paper!  I think some of the reporting could be done better, but this is a minor complaint.
review_title: null

Meta-review: This paper investigates the dereverberation problem from the audio-visual perspective.  The geometry of the environment is represented by RGB and depth images.  The authors propose a so-called visually-informed dereverberation of audio (VIDA) model and also create a dataset consisting of both synthetic and real data to verify the effectiveness of the model.  Experiments are conducted on speech enhancement, speech recognition and speaker identification tasks.  The authors compare VIDA with audio only dereverberation as well as various established baseline systems in the community.  

The audio-visual way of coping with dereverberation using visual representation of the acoustic environment seems to be interesting. The authors' rebuttal has cleared most of the concerns raised by the reviewers but there are still numerous lingering concerns which affect its acceptance.  First of all,  most of the reviewers consider the novelty not overwhelmingly significant.  Second, the contribution of the visual input seems to be only marginal compared to the audio-only dereverberation. Results on real data are also mixed.  Some of the reported p-values are extremely small, which raises questions whether it is due to the size of the test set.  Third, there are noticeable artifacts in some of the samples in the demo.  Fourth,  there are numerous issues in the paper that are worth further in-depth investigation. For instance, it would be helpful to show in which way exactly the RGB and depth images helps.
Decision: Reject