Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation

Authors: Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: robust audio-visual speech recognition, audio-visual corruption, multimodal representation learning

Abstract:
Audio-visual speech recognition (AVSR) incorporates auditory and visual modalities to improve recognition accuracy, particularly in noisy environments where audio-only speech systems are insufficient. While previous research has largely addressed audio disruptions, few studies have dealt with visual corruptions, e.g., lip occlusions or blurred videos, which are also detrimental. To address this real-world challenge, we propose CAV2vec, a novel self-supervised speech representation learning framework particularly designed to handle audio-visual joint corruption. CAV2vec employs a self-distillation approach with a corrupted prediction task, where the student model learns to predict clean targets, generated by the teacher model, with corrupted input frames. Specifically, we suggest a unimodal multi-task learning, which distills cross-modal knowledge and aligns the corrupted modalities, by predicting clean audio targets with corrupted videos, and clean video targets with corrupted audios. This strategy mitigates the dispersion in the representation space caused by corrupted modalities, leading to more reliable and robust audio-visual fusion. Our experiments on robust AVSR benchmarks demonstrate that the corrupted representation learning method significantly enhances recognition accuracy across generalized environments involving various types of corruption. Our code is available at https://github.com/sungnyun/cav2vec.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1730626811103

thread_type:
official_review

<thread_object>
creator: reviewer_vqmw

<content>
summary:
This paper proposes a self-distillation framework for audio-visual speech representation learning. It enhances representation robustness through multimodal and unimodal multi-task learning. Experimental results demonstrate performance improvements over existing methods.

strengths:
+ The target is clear. It aims to handle audio-visual joint corruption in AVSR, which is quite common in real-world.

+ The method is simple and efficient, requiring no additional modules to handle audio-visual corruptions, which makes it practical for implementation.

weaknesses:
- The work is similar to several existing works in the view of methodology, like AV-Hubert, raven, dinoSR, ES^3. One difference is the cross-modal prediction of both A→V and V→A, instead of just V→A, which is mainly due to the introduction of corruptions in both A and V modalities in this work. Although the starting point of tackling corruptions in both modalities is meaningful, this specific learning manner is not so insightful or so original. Compared with those existing works, there are little new things in this work. There are already many works learning audio-visual speech representations with EMA framework at present. I am not clear about the novelty of the work in the view of methodology compared with existing works.

- The validation in this paper is not thorough enough, as the authors only demonstrated improvements on the AVSR task, although they explicitly stated their focus on AVSR. In real-world scenarios, complete corruption of one modality is possible, and missing modalities are common. If the audio-visual representations learned by this method are truly robust, then performance on single-modality tasks (ASR and VSR) would help substantiate this claim.

- The reliance on AV-HuBERT pre-trained weights raises questions about the framework's standalone capabilities. It remains unclear whether comparable performance could be achieved training from scratch, or if robust audio-visual representations are inherently dependent on AV-HuBERT. 

- The manuscript suffers from clarity issues and leaves several questions (detailed below).

questions:
1. Figure 3's notation lacks precision. While the bidirectional arrows suggest symmetric interactions, the prediction task implies directed objectives. 

2. What is the definition of "visual corruption events"? Is it having a single corrupted frame in the whole visual sequence or something else? In Appendix A.2, the description "Object occlusion applied to the speaker's lips occurs once, followed by Gaussian noise or blurring, each with a probability of 0.3", does this mean that occlusion only appears in a single frame within the sequence?

3. The rationale for applying MP (Masked Prediction) exclusively in multimodal settings, while CP (Corruption Prediction) spans both multimodal and unimodal scenarios. It would be better to clarify the reasons.

4. The parameter k in MP's objective (targeting the average of teacher model's top-k blocks output) is not specified, and the motivation behind this design is not explained.

5. The absence of L_AVCP in the overall multi-task loss function requires explanation.

6. What is the motivation for MLM and what is the setting of it? How to acquire the cluster index? Need explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721258873

thread_type:
official_review

<thread_object>
creator: reviewer_etwb

<content>
summary:
This paper presents a self-supervised framework for learning audio-visual speech representations which are robust to noise. This is achieved by including audio and visual corruptions during the self-supervised learning (SSL) phase. As a consequence, the model is more robust to audio and visual noise during evaluation. Results are shown on the LRS2 and LRS3 datasets and the proposed approach outperforms prior SSL approaches on noisy conditions.

strengths:
The paper is well written.

The paper investigates the use of audio and visual corruptions during the SSL phase which has not been thoroughly investigated in previous works.

weaknesses:
The paper has a lot of similarities with AV-data2vec, AV-Hubert, Raven and Braven, especially regarding the masked prediction and cross-modal prediction tasks. It would be very useful if some discussion is added in the related work section explaining the differences with each of these approaches.

Since the proposed model is pre-trained with corrupted data it's reasonable that it performs better than the other SSL methods which have not been trained with such data. The authors should make clear what the contribution is. If it's just the addition of corruptions during pre-training and then it should be clearly stated that the other approaches might also benefit if they are pre-trained with such corruptions.

The proposed model is initialised with AV-HuBERT weights. This is done for efficiency but it’s still different from other methods which are randomly initialised. Can the authors comment on the impact of this initialisation? Are there any results where the model is trained from scratch?

The number of parameters in Table 1 is not 100% accurate. Raven and Braven have 673M during pre-training and during inference, however the other approach have 325M only during the pre-training phase. During inference the number of parameters is double, since models are separately fine-tuned for each modality.

L. 351-353 state that publicly available pre-trained models  are used and then fine-tune them based on their hyperparameter settings. This leads to significantly worse results for RAVEn / BRAVEn than the ones published. For example, on clean LRS3 audio, results of 2.3 / 1.8 % WER for RAVEn / BRAVEn are presented whereas the published results are 1.4 / 1.1 % WER. Can the authors comment on this discrepancy? And maybe explain why this happens?

L.18: Does the model predict “corrupted frames” as stated? I think it predicts clean frames.

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730839408469

thread_type:
official_review

<thread_object>
creator: reviewer_iode

<content>
summary:
The paper introduces a technique called CAV2vec which is a model trained with a so-called corrupted prediction task: the student's encoded output on corrupted input is trained to be similar to the teacher's output on clean (uncorrupted) input in an audio-visual speech recognition task consisting of audio (speech) and video (lip tracks) inputs. They specifically using cross-modal targeting gives additional benfit. The demonstrate success against a variety of strong models, yielding equivalent state-of-the-art for AVSR on uncorrupted input (vs AV data2vec) and better results than state-of-the-art on artificially corrupted data.

strengths:
Paper shows good results against a variety of relevant, strong baselines. In table 1 they show that on both visual and audio corruption CAV2vec shows the best robustness. In some ways, this could be considered an extension of data2vec with the corrupted prediction task yielding better results. In this light, it is notable that the task does not degrade on the Clean condition (e.g. 1.5% is still the best number in Table 1). However on the artificially corrupted noisy conditions, Cav2vec is peforming more than 10% relative on average than the next best systems. The paper also provides ablations to tease out what part of the corruption is helping most, e.g. ACP + VCP.

weaknesses:
The clearest wins are on the artificially created corrupted noise sets---this is often the case when the experiments simulate harder test conditions and corresponding training conditions to address the harder test. Other systems, do not have the benefit of training to match the altered test conditions.

The various acronyms AVCP, mACP, mCP, ACP, VCP are somewhat hard to follow. While Figure 5 helps, perhaps a table indicating exactly what they mean with columns indicating differing dimensions could help (it may not...).

questions:
Are there harder, non-simualted noisier datasets that could be used to evaluated the benefit of CAV2vec over say data2vec?

Is the data2vec system trained on, is it possible to train on, the same distributions and corruptions in training as the CAV2vec system?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731357854223

thread_type:
official_review

<thread_object>
creator: reviewer_adiv

<content>
summary:
The paper introduces CAV2vec, a self-supervised audio-visual speech representation learning framework focused on robustness to joint audio-visual corruption in noisy environments.
CAV2vec uses a self-distillation framework with corrupted prediction tasks. It leverages a teacher-student model where the student learns to predict corrupted frames while clean targets are generated by the teacher model. They proposed an unimodal multi-task learning strategy that mitigates the dispersion in the representation space caused by corrupted modalities, thereby enhancing reliable and robust audio-visual fusion.

strengths:
Clear Problem Definition: The paper clearly identifies limitations in existing methods and proposes a well-reasoned solution.
Novelty: The approach of using unimodal multi-task learning for cross-modal corrupted prediction is innovative and addresses a real-world challenge in AVSR.
Technical Depth: The framework uses a combination of advanced techniques, including self-distillation, multimodal Transformer encoders, and self-supervised learning without architectural changes.
Empirical Validation: The experiments on multiple benchmarks (LRS2, LRS3) and various corruption scenarios demonstrate improved robustness.
Clarity: The paper is well-structured, and technical sections are detailed with visual aids, providing clarity on the approach and results.

weaknesses:
The evaluation on the LRS2 benchmark could be more comprehensive. The authors should consider including additional experimental results, similar to those presented for the LRS3 benchmark. This would better illustrate the model's robustness under real-world conditions.

questions:
There is an error in the sentences on lines 421 to 422. Specifically, the description of CAV2vec’s performance on unseen types of corruption (i.e., hand occlusion and pixelated face) is incorrect. According to Table 1, the correct values should be 5.2 and 5.1, respectively.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732459010737

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
Dear all reviewers,

Thank you for the time and effort you have dedicated to reviewing our paper and providing such valuable feedback. Your insights have been instrumental in enhancing the quality of our paper. In response to your comments, we have uploaded a detailed response for each review and revised the manuscript accordingly, particularly by supplementing it with new experimental results during the rebuttal phase.

We would like to gently remind the reviewers that the discussion period will close in two days. We are committed to addressing any further clarifications or questions you may still have. Thank you once again for your invaluable contributions to improving our work.

Best regards,    
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper received three positive reviews (two 8s and a 6) and a negative review (5). The paper introduces a novel method for AVSR with joint AV corruption incorporating self-distillation. The reviews mainly requested additional clarifications, experiments and discussions that would potentially strengthen the paper and the author addressed most of them. While the negative reviewer requested clarifications on the novelty of the work and additional experiments, which the authors provided in their rebuttal, the review did not responded to the author rebuttal. The AC believes that the author response largely resolved the issues raised by the reviewer and therefore recommends the acceptance of the paper following the majority of the reviewer's ratings.