Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a self-supervised learning framework, termed u-HuBERT, that can utilize both multimodal and unimodal unsupervised speech data including audio-visual, audio and visual to pre-train a single model by utilizing modality dropout. If this pre-trained model is fine-tuned on audio-visual data with modality dropout, it can achieve comparable or better results than the state-of-the-art modality-specific models. In addition, the authors show that the modality dropout used in pre-taining is important for learning modality agnostic features, which makes the model only fine-tuned on audio perform well on audio-visual or visual speech data, achieving zero-shot modality generation for speech recognition and speaker verification.
strengths_and_weaknesses: Originality:

- Previous work on audio-visual ASR are mainly focusing on modality-specific training / fine-tuning. This work, using a shared target token space for all unimodal and multimodal data in a single model, and find-tuning it with labeled data only on a subset of the modalities seen during pre-training but perform well on other modalities in test to achieve zero-shot modality generalization, is a novel idea.

- Even though modality dropout has been used in the literature, the analysis to show that it is important for learning modality agnostic features and achieving huge gains in zero-shot modality generalization scenarios is new.

Quality:

- The authors show that the proposed single model achieves on par or better results than state-of-the-art modality-specific models for speech recognition. It is very promising that the model fine-tuned only on audio data performs well on audio-visual and visual speech data with comparable results to the models fine-tuned on the modality-specific data.

- The experiment using out-of-domain labeled audio data from LibriSpeech for fine-tuning shows convincing results to support the claim that the model's zero-shot modality transfer is not achieved through memorization.

- The proposed model also shows the zero-shot modality generalization ability but only to some extent. The authors suggest this is due to the nature of the speaker verification task, where each modality can only infer a subset of the speaker features. This demonstrates a limitation to the zero-shot transfer.

Clarity:

- The paper is written clearly. The presentation is very well organized.

Significance:

- This work contributes to the community significantly, by showing an approach to pre-train on both multimodal and unimodal speech data to not only improve the state-of-the-art results in audio-visual ASR but also achieve high-quality zero-shot modality generalization. In particular, it brings significant value to the tasks with modalities without much labeled data.
questions: Questions/suggestions:
- The proposed framework assumes the multiple modalities features align in a frame-by-frame fashion. How can it be applied to the case where the assumption does not hold, e.g. between the audio and text modalities?
- Line 194: Is the pre-training modality dropout probability 0.25/0.25 tuned? How sensitive is its value to the results?
- In Table 2, modality dropout on fine-tuning also improves AV-WER for the models fine-tuned on AV data. It would be better to explain why that is the case.
- Line 259: For the claim of u-HuBERT "achieving on par or better than the best modality-specific models", it would be better to mention the caveat as well: it's slightly worse than the best modality-specific visual speech WER (27.2 vs. 25.9) although using much less labeled visual speech training data.

Typos:
- Line 37: it essential -> it is essential
- Line 327: supported u-HuBERT -> supported by u-HuBERT
limitations: Yes, the authors have addressed the limitations in Section 5 and the potential negative societal impact of their work in Appendix C.

For completeness, the authors should also mention another limitation of the system that assumes all the modalities align in a frame-by-frame fashion (as mentioned in my question above), or discuss possible extensions to address this limitation for cases where this assumption does not hold, e.g. between audio and text modalities.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents an approach to handle speech signal in multiple modalities within a single unified model that can be pre-trained and fine-tuned with datasets of mixed speech modalities.  The model includes modality specific feature extraction followed by a shared transformer module that learns modality agnostic representations.  Audio and visual modalities are considered and results are presented on speech recognition as well as speaker verification tasks.  It is also shown that the proposed model exhibits strong performance in zero-shot generalization to modalities for which it was not fine-tuned.



strengths_and_weaknesses: This paper is very close to the recently published AV-HuBERT paper [1], so much so that it is unclear to me if the additional contributions are significant enough for publication.  For instance,

* A key novelty of the paper, as mentioned in lines 54-55 and lines 103-104, is that AV-HuBERT pre-trains with audio-visual multimodal data only whereas the proposed approach pre-trains with multimodal as well as unimodal data.  However, as discussed by She et al. [1], a modality dropout is applied during training to make the model learn how to deal with unimodal inputs, thus effectively pre-training on unimodal data as well.

* Lines 121-123 state that “The main idea of this paper is to apply modality dropout to multimodal speech…”.  However modality dropout was applied in [1] as well, and it is unclear how use of modality dropout in current model differs from that in [1].

* Empirical results are also quite close to those reported by Shi et al. [1].  The best ASR results reported in the current paper are AV/V WER of 1.2/27.2% (these include the TD audio-only corpus during pre-training in addition to 1759 hours of audio-visual data) whereas Shi et al. report AV/V WER of 1.3/26.9% where pre-training is done on 1759 hours of audio-visual data.

[1] B. Shi, W. Hsu, K. Lakhotia, A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” ICLR 2022.

The zero-shot generalization results as well as application to speaker-verification task are novel; we don’t have corresponding results from the AV-HuBERT model.

Minor typos:
Line 37: it essential -> it is essential
Lines 54-62: provide reference to AV-HuBERT
questions: Please see comments above.
limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a self-supevised method in the HuBERT family designed for audio-visual speech, which can be trained on different modalities for pre-training, including single modality audio or visual, and multi-modalities (audio-visual).
u-HuBERT is closely related to AV-HuBERT, but incorporates masked cluster prediction for both unimodal and multimodal data, via masking on modalities, ("modality dropout") which requires the model to predict the same targets using different sets of input modalities.
strengths_and_weaknesses: 

Strengths: 

The proposed model provides the ability to fine tune on any modality and work well on all modalities, with comparable performance to modality-specific models.

Evaluation is thorough.  
Improvements are small in absolute terms on LRS3, but baseline error rates are very small (e.g., 1.4% WER for AV ASR), so a reduction to 1.2% which is a large relative WER reduction of 14%.  Improvements like this are typically very difficult to achieve.  
These are useful results.  

The paper is well written and quite clear. 


Weaknesses:

The paper is not self-contained, and most of the architectural details are contained in other references. 

Novelty relative to AV-HuBERT is not large, but is important. 

The proposed methods are practical, but not based on any deep theory developments.  
questions: I am under the impression that  w2v-BERT produces better representations than HuBERT.  Can you comment on your selection of HuBERT versus w2v-BERT, and whether your proposed additions to AV-HuBERT are applicable to the case of w2v-BERT?  
limitations: Limitations are discussed in section 5
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presented unified hidden unit BERT (u-HuBERT), which leverages unlabeled speech data of many different modalities for pre-training, including both unimodal and multimodal speech. The u-HuBERT model is composed of a set of modality-specific feature extractors and a shared transformer backend. With this architecture design, u-HuBERT can learn modality-agnostic frame-level representations, and can be fine-tuned with either multimodal or unimodal data. Experiment results show the effectiveness of u-HuBERT on two speech processing tasks: speech recognition and speaker verification.

-> After reviewing the author rebuttal, I changed my ratings to 5.
strengths_and_weaknesses: Built upon AV-HuBERT, u-HuBERT can generalize to utilize both multimodal and unimodal data by mapping different inputs into a modality-agnostic feature space using the proposed modality dropout method. Modality dropout also allows zero-shot modality generalization, which means the fine-tuned model can handle modalities in the downstream task that do not cover the ones seen during pre-training. Even though the author showed efficiency of the proposed method in speech recognition and speaker verification tasks, there are still some studies and discussions missing in the paper. For example, the modality dropout rates during pre-training and fine-tuning and what's the effect of different dropout rates in each stage?

Typo
* Table 2 caption "FT-drop-p" should be "FT mod-drop-p".
questions: In section 4.3, when showing the results of fine-tuning on OOD labeled LibriSpeech data in Table 3, what's the performance of baseline methods? The author claimed the zero-shot modality transfer performance is decent on LibriSpeech, however, without comparing with baselines, it's hard to convince the readers.

More descriptions need to be added when comparing with SOTA models in section 4.3 (Table 4). Several questions are:
* What are the baseline model settings?
* What are the pre-training/training/fine-tuning approaches used by baseline models?
* What are the datasets?
limitations: The author had limitation discussion in section 5.
review_title: null

Meta-review: metareview: This paper enjoyed a reasonable discussion between the authors and the reviewers, and the authors have improved the paper in a number of areas, including (1) more experiments clarifying the role of the modality dropout rates on model performance, (2) a baseline for the Librispeech transfer experiment, and (3) more details in the comparisons to other state-of-the-art models.

While the reviewers generally agree that the zero-shot results are novel, the experimental results are strong, and the analysis of the role of modality dropout in learning modality-agnostic representations is a good contribution, they split on their final recommendation on the paper, with three recommending acceptance (at least to some degree) and one sticking to a borderline reject recommendation after discussion with the authors, arguing that the novelty with respect to the AV-HuBERT paper is insufficient.

I am recommending that the paper be accepted, but I think the discussion between the authors and reviewer BiuF includes important points that should be made more strongly in the paper. Specifically, they need to emphasize more strongly the use of targets from multiple modalities (their answer to Q1 in "Response to Reviewer BiUF's Followup Comments"), and they need to explain *in the paper* that "the u-HuBERT from the fourth-to-the-last row to the second-to-the-last row are pre-trained without unimodal data, and are effectively AV-HuBERT" (from their answer to Q2 in "Response to Reviewer BiUF's Followup Comments") to help readers understand the relationship between u-HuBERT and AV-HuBERT. I would also suggest that they add experiments where modality dropout is applied during cluster label generation as an additional constrast.

confidence: Less certain
Decision: Accept