Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes audio-visual HuBert based on the original HuBert model. The general idea is similar to HuBert that predicts clusters of extracted features. The authors further propose the modality dropout and masking by substitution module to further fine-tune the design. Extensive experiments and ablation studies are carried out on the LRS3 dataset, which validate the effectiveness of the proposed method.
main_review: ## Strength:

++ This paper is basically well-written with detailed supplementary.

++ The idea of leveraging Bert for self-supervised training is different from the traditional audio-visual alignment pertaining.

++ Detailed improvements such as the audio-visual stream formulation and Masking by substitution have been proposed.

++ The results on LRSv0.4 outperform the previous SOTA.

## Weakness:

-- This kind of formulation, though novel on the topic of audio-visual speech recognition, has been used on other topics. The whole model is built upon HuBert, which limits the contribution.

-- Why use only the LRS3 dataset for evaluation? The LRS2 dataset should be available. More dataset evaluations could be more comprehensive than one.

-- About the **Masking by substitution**: Why would "the fake segment detection sub-task improves the learned features of the ResNet encoder" when the “filled-in frames are from real videos"? More intuition should be given. It is said that your motivation is to keep "masked region smooth temporally". I understand that it would be smoother than random noise masking, however, segments from the same video cannot guarantee temporal smoothing.

-- I am curious about the "MULTILINGUAL VS. MONOLINGUAL" experiments in Sec. 4.4. The model is kept in a setting with one 1 iteration training and 30h of labeled data. While I understand the limitation in resources, the authors are suggested to show the multilingual results under the full setting. With extra data involved, it might take a longer training time. After all the method is "agnostic to the spoken language".

summary_of_the_review: Overall this paper introduces techniques that the field of audio-visual speech recognition has rarely leveraged into this field, and makes task-specific modifications. The idea is not thoroughly new but the application is good. As I am no expert in this field, it is possible that I have missed something.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a multi-modal (audio-visual) pre-training approach that results in models that can be fine tuned to achieve state of the art performance on both lip-reading (visual only) as well as speech recognition (audio only) on the LRS3 dataset which is the largest public lip-reading benchmark. It extends the idea of masked self-supervised learning to multi-modal scenarios where units that are masked and predicted are automatically discovered via clustering and refined as training progresses.  While the proposed approach is a relatively straight forward extension of previously published HuBERT model it is demonstrated to be very effective.  Paper also presents a number of alternative approaches for leaning embeddings of the visual modality and very interestingly demonstrates the value of multi-modal input to the model even when a single modality is present at test time.

main_review: Pros:

a) Simple but very effective extension of masked self-supervised training to multi-modal tasks

b) State of the art lip-reading and ASR results on the LRS3 benchmark

c) Demonstrates value of providing multi-modal input to the model even when only a single modality is present at test time.  Alternatives that consider only that modality as input (image of lips in this case) are significantly worse.

Cons:

Not a negative, but it will be interesting to see further experimentation with layer-wise supervision approaches that inform visual representation learning via audio or AV networks.  This will have the advantage that the model will only see the visual modality at training time, yet will have the advantage of supervision from audio or AV model at all layers, not just at the output.


summary_of_the_review: Very well written paper, presenting an effective approach for learning embeddings for multi-modal data.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a strategy for self-supervised pre-training for lip reading. To this end, they propose Av-HuBERT which learns embeddings by masking video input and predicting iteratively refined hidden units. The authors propose two sub-strategies for this: visual-only and cross-modal. The pre-trained embeddings are fine-tuned on the lip reading tasks, in the style of wav2vec 2.0 and HuBERT for automatic speech recognition. The authors perform experiments on both limited and full resource settings, on both of which they demonstrate strong performance. 
main_review: The ideas are intuitive and very suitable for the problem. The authors take good advantage of a recent advancement of pre-training in ASR (HuBERT), and applies it to lip reading, aka visual speech recognition. The visual-only HuBERT is a simple adaptation, but the cross-modal iterative refinement is a useful addition. 

The performance is strong on both low and full resource settings, and the results clearly demonstrate the advantages of self-supervised pre-training.

It would be good to also show the effectiveness of the embeddings for other downstream tasks, such as audio-visual speech recognition and speech separation. 

I think Table C1 in the appendix is quite useful for justifying design choices and at least some of it would be beneficial to show in the main paper. It would also be good to show lip reading results after visual-only single-modality pre-training (like audio HuBERT), without cross-modal training.

As an application paper, the paper might be better suited for CVPR or Interspeech, but I still vote for acceptance since the contributions are significant and the results are state-of-the-art.

summary_of_the_review: The authors propose an effective adaptation of HuBERT for lip reading, by adding cross-modal iterative training. The contributions are logical and the results are strong.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new self-supervised learning framework for audio-visual speech. Their framework is trained with BERT-like training that predicts representations in the masked regions using audio and visual inputs. Extensive experiments are performed using two popular audio-visual datasets. Especially, with a small amount of labeled training data (30 hours), their method achieved comparable results with a previous state-of-the-art method which trained with a huge amount of labeled data (31,000 hours). Moreover, when trained with more labeled data (433 hours), it surpasses the previous state-of-the-art method (trained with 31,000 hours) by 3.1% WER.
main_review: Strength:
1.	The authors propose a powerful method to learn speech representations from multi-modal data which can be utilized for both ASR and VSR, in a self-supervised manner.
2.	AV-HuBERT achieves state-of-the-art performance in terms of WER in a sentence-level audio-visual dataset, LRS3. Moreover, even in a low-resource setting (training with 30 hours labeled data + 1,759 hours unlabeled data), their model shows a comparable VSR result with a previous state-of-the-art method that trained with 31,000 hours labeled data.
3.	Their proposed method is well verified in various views with extensive experiments through both manuscript and supplementary. (Performances 1) with different model sizes, 2) with different loss functions (CTC and S2S), 3) with different cluster targets, 4) using multi-lingual data, 5) of both ASR and VSR, and 6) of ablation studies.)

Weakness:
1.	Even the proposed AV-HuBERT shows impressive performances, but it is an expansion of HuBERT to work in multi-modal data.
2.	There are many typos and errors in the manuscript. Please refer to below detailed comments. Moreover, Figure 1 should be improved, the arrows are arranged confusingly.

Questions:
1. The authors use concatenation as the default fusion operator. Which dimension is utilized for concatenation, temporal or channel? If the temporal dimension is used for multi-modal fusion, better to refer to the two papers [R1, R2].
2. For the loss function in equation (4), their final performance seems obtained by setting the alpha as 0. However, it is hard to find because it is placed in the results of ablation studies in the appendix. Since setting the alpha as zero (the same as omitting the second loss term) is better performed, it would be better to describe the effect of alpha in AV-HuBERT after equation (4) or at the experimental setup, briefly.
3. The ASR performance of AV-HuBERT is not presented. It is just described as the AV-HuBERT performs worse than HuBERT in text. Moreover, the last sentence in Sec 4.5 seems not enough to explain why AV-HuBERT is not better than HuBERT on ASR, even if multi-modal data is utilized during training. Have the authors re-examined hyper-parameters (m_a, m_v, p_m, p_a, alpha) for ASR task?
4. 2nd paragraph in Sec A.4. What does the sentence “Both modalities are used at test time for feature extraction” mean? There is no downstream task utilizes both modalities.

Errors:
1. Sec 3.2 line #4. In “The cluster assignment z_(1:T)^a~”, Does z_(1:T)^a mean z_(1:T)^i? 
2. 3rd paragraph on page 4. “When only modality is used”  -> "only one modality".
3. Last paragraph on page 5. “where B consists of all possible...” -> “where B^-1 maps all possible...”. B seems a mapping function thus the word “consist” seems not appropriate.
4. 2nd paragraph on page 6. “we rely on the joint decoder module”. What does the joint decoder mean? Do the authors use a joint CTC/S2S decoder?
5. 2nd paragraph on page 8. ‘iterative pe-training’ -> ‘iterative pre-training’
6. 1st paragraph on page 14. CMUDict (cmu) -> maybe missing reference.

[R1] Chen, Yen-Chun, et al. "Uniter: Universal image-text representation learning." European conference on computer vision. Springer, Cham, 2020.
[R2] Lee, Sangho, et al. "Parameter Efficient Multimodal Transformers for Video Representation Learning." International Conference on Learning Representations. 2020.
summary_of_the_review: Overall, authors approach is promising for reducing the need for a large labeled visual dataset for training VSR models, and the result shown in the paper is significant.
review_title: null

Meta-review: PAPER: This paper introduces an extension of the HuBERT audio-only model for the audio-visual setting, allowing for self-supervised pre-training of multimodal model which also performs well on the unimodal tasks (lip-reading and ASR). The paper applies the idea of modality dropout to their multimodal pre-training setup and introduce the idea of masking with substitution as a way to improve visual representation learning. A strong aspect of the paper is its experimental section, showing strong improvement for lip-reading tasks, bringing a new state-of-the-art performance. The experiments also show improvement for ASR task. 
DISCUSSION: All reviewers seemed to appreciate the experimental results, with new state-of-the-art performance on both unimodal task, when performing multimodal pre-training. The paper does bring some technical novelty, but primarily because of its application to the audio-visual domain. The modality dropout idea was already explored for other audio-visual tasks such as speech-driven face animation (Abdelaziz et al., ICMI 2020), but the idea of “masking by substitution” seems novel and helps learning better visual representations. The authors were able to address many questions and concerns expressed by reviewers. All reviewers took the time to read these responses and acknowledge them.
SUMMARY: This paper brings an interesting extension of the audio-only HuBERT model for the audio-visual setting. The strength of the paper is in its evaluation, with strong performances, establishing many new state-of-the-art results.  All reviewers supported the acceptance of this paper.
Decision: Accept (Poster)