Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A contrastive audio-visual masked autoencoder model is introduced in the paper. The training of self-supervised model is done with both contrastive and masked-autoencoder losses.

The model architecture is transformer and only unmasked inputs are kept as input for the encoder. There is a separate encoder for each modality as well as a joint encoder. The joint encoder can be run for one or both modalities (with its corresponding layer-norms). In the case when the model is fine-tuned for one of the modalities, the joint decoder would only be run with only that modality input.
strength_and_weaknesses: Strengths:

1. The model seems to be a novel audio-visual self-supervisedly trained autoencoding network that uses both masked autoencoder and contrastive losses. Previous work I believe only used contrastive/coincidence losses for audio-visual learning. Masked autoencoding was only used for each modality separately.
2. The joint encoder is run separately for each modality for the contrastive loss, but it is run jointly for the masked-prediction loss which makes it possible to use for each modality separately too.

Weaknesses:
1. The video is only represented with 10 frames from 10 seconds (1 Hz) which seems low. So, the model probably relies on images rather than the motion.

clarity,_quality,_novelty_and_reproducibility: The paper is clear enough. The idea of using masked-autoencoding for audio and video is novel. Combining both losses is interesting. The results woud be reproducable if the model is made available.
summary_of_the_review: The paper is an interesting read which generalized masked-autoencoders to multiple modalities of audio and video. In addition a contrastive loss is also used. 

Relevant ablations are done by comparing with a vanilla VAE which does not have a contrastive loss. The models are used in classification and retrieval problems and the benefits for CAV-MAE model is shown.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines  masked auto-encoder based learning and contrastive learning for self-supervised learning in audio-visual settings. The method referred to as Contrastive Audio-Visual Masked Auto-Encoder learning is used to pre-train the models on audiovisual Audioset dataset and then fine-tuned on Audioset and VGGSound. Performance improvements in the downstream tasks of event classification are shown. 
strength_and_weaknesses: Strength
1. The proposed approach is simple and logical. It tried to combine strengths of masked data modeling and contrastive learning and applied it to audio-visual settings where they have not been explored. 


2. The paper is very neatly written and easy to read and follow. 


3. The results on the datasets used show good performance improvements. 



Weakness


1. A more exhaustive set of audio-visual tasks would have been better. 


2. Some other weaknesses are pointed out in the review below. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written. The novelty might appear limited as it combines two well known methods in straightforward manner, I believe this is not a major concern here and the approach is neat. 
summary_of_the_review: The paper combines masked auto-encoder based learning with contrastive learning. It develops and investigates the method for audio-visual representation learning. 


1. I believe initializing the weights from MAE models leaves a few things unclear. Performance of the model by training from scratch should be reported. Its unclear how critical such initialization is. Especially, in the current context with MAE based initialization - it becomes a two stage training process. First train each modality with MAE and then combine them using the current approach. 


2. The downstream tasks are limited and other audio-visual tasks should be included. CUrrently the paper just focuses on event classification on two datasets. These two datasets do not provide a comprehensive understanding of audio-visual representation learning. Diverse set of tasks like audio-visual action classification, Epic Kitchen benchmarks, audio-visual segmentation/localization etc. would be able to provide a clearer picture of how good the SSL approach is. In fact, Audioset is not the best dataset for audio-visual learning of sounds. A large portion of visual data in Audioset is just redundant (black frames, audio overlayed over random non-relevant videos and so on). Moreover, Audioset is used for pre-training as well. So finetuning results especially on the full Audioset is not an ideal set of experiments. 



3. Since the tasks are primarily audio tasks, references to some missing audio and speech SSL works are missing (like Wav2Vec and Hubert). [R1] in particular used wav2vec like learning for non-speech audio representation learning and has results on Audioset and several other audio tasks. 



4. In Section 4.3, authors comment on computational efficiency of the proposed method. I am not sure those arguments are convincing. The models are exactly the same and the computational expense of these models will be the same. 


R1: Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks in IEEE ICASSP 2022.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-modal Masked Auto-Encoder for audio-visual tasks. More specifically, the proposed method combines contrastive learning and masked data modeling for better joint and coordinated audio-visual representation. The model based on self-supervised training achieves a new SOTA on VGGSound and great performance on AudioSet.
strength_and_weaknesses: ### Strengths

- The proposed method leverages multi-modal information well with two hot self-supervised ways, contrastive learning, and mask auto-encoder. The method achieves great performance on audio-visual benchmarks.
- The designed method combining contrastive learning and multi-modal MAE is promising for multi-modal tasks.
- Sufficient ablation results are provided.
- This paper is well-written and easy to follow.

### Weaknesses

- This work is not novel enough. It is well-known that contrastive learning and MAE are powerful self-supervised methods, and applying them to a new domain based on relative works is simple yet effective. This paper extends the combination of the two approaches to the multi-modal learning literature, but the architecture is trivial. Besides, this work focuses on the audio-visual task. However, no domain knowledge is leveraged in this work, and the pipeline can be the same as those focusing on text-image multi-modal learning.
- Though lots of experimental results are shown in the paper, the work only focuses on audio-visual event classification and audio-visual retrieval. More relative experiments should be conducted.
- In section 2.3, the authors list some key designs of CAV-MAE. However, these points are all based on the common sense of contrastive learning, MAE, or multi-modal learning. Overall, the framework's design lacks novelty, and the application of the self-supervised methods is so plain. Therefore, the framework's design can be deemed as simple incremental work.
clarity,_quality,_novelty_and_reproducibility: **Novelty:** As described above (weaknesses), I believe limited novelty is the biggest problem of this paper.

**Clarity/Quality:** The presentation is clear, and the work is easy to be followed.

**Reproducibility**: If the code will be released, I believe this work is easy to reproduce.
summary_of_the_review: This paper combines two powerful self-supervised methods, MAE and contrastive learning, and extends the combination to the audio-visual domain.  The overall pipeline is easy to understand, and the paper is clearly written. However, this paper contributes little to the relative areas, including self-supervised learning and multi-modal learning. And this method is not novelty enough.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors extend the Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities and combine it with contrastive learning for self-supervised audio-visual learning. Although the two self-supervised learning (SSL) strategies are not new, the proposed Contrastive Audio-Visual Masked Auto-Encoder nicely unifies them in one single framework. Extensive experimental results demonstrate that the two SSL methods are mutually beneficial and the pre-trained models can boost audio-visual classification and cross-modal retrieval performance. 
strength_and_weaknesses: Pros:

+ The proposed Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) can nicely unify MAE and contrastive learning for self-supervised audio-visual learning. In CAV-MAE, unimodal and joint representations are utilized for contrastive and MAE learning.

+ Extensive experimental comparison and ablation studies on AudioSet and VGGSound can validate the effectiveness of the proposed CAV-MAE.

+ I enjoy reading this paper. It is quite easy to follow. The analysis paragraphs can well validate and explain the proposed method. 

Cons:

- Not all visual objects make sounds and not all sound sources are visible. The audio-visual mismatch may introduce a lot of false positive audio-visual pairs for audio-visual self-supervised learning. Whether the proposed method can mitigate the data issue? How does the issue affect MAE and contrastive learning respectively? I suggest the authors add discussions on this. 

- I was impressed by the reconstructed spectrograms when 75% of the input is masked. Why does audio MAE work? For visual scenes, objects are grouped in certain regions. But spectrograms have quite isolated TF patterns. But, it seems that MAE works for both modalities. Really hope to hear more clarifications and discussions. 

 




clarity,_quality,_novelty_and_reproducibility: Technically, both MAE and contrastive learning are not new. But the proposed CAV-MAE can nicely unify them in one single framework. The thorough experiments can well validate the proposed approach. 

Thorough Implementation details are provided. 
summary_of_the_review: The proposed  CAV-MAE is technically sound and extensive experiments can validate the superiority of the proposed method. I only have a few concerns about the proposed method and I will be happy to upgrade my rating if the authors can address the concerns during the discussion phase. 

***Post-rebuttal***

The new experiments and analysis can well address my concerns. I have increased my rating to 8 from 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends MAE from single modality to audio and visual modalities by combining contrastive learning and masked prediction pretext tasks. Experiments show the improvements on VGGSound dataset and better retrieval score. 

strength_and_weaknesses: Strength:
With the advent of MAE and audioMAE, it is natural to extend such framework to multimodal settings.
The motivation for such idea is strong, and the 3 discussion points are valid given the landscape of the current SOTA.  

Weakness:
First, the major logic is to show performance improvement brought by CAV-MAE, and empirically verifies it indeed leads to learning better joint representation. However, CAV-MAE underperforming VS. MBT on multimodal AudioSet is a counter example. Maybe as <Li, et al. 2022 AudioTagging>pointed out, the data quality issue of AudioSet would cause a variation, I would suggest the authors look into the granular errors in AudioSet and report the exact train/test numbers to ensure an apple-to-apple fair comparison.

Normalization of different branches are tuned differently, and this looks to me like an engineering effort. I understand normalization is crucial for transformers tuning, but just wondering if there could be a better way rather than setting a fixed number since this severely limits the generalizability of the model. 

I get the point of reporting retrieval score on subsets of AudioSet and VGGSound, however this is not a standard dataset for retrieval benchmark, I would suggest the authors reporting on MSR-VTT in a zero shot way. 

In appendix and throughout the text, I realized the authors adopted the vanilla MAE masking, which is the random visual-intuitive masking, but as <Huang et al. 2022 AudioMAE> pointed out, this is not necessarily the most suitable way of masking for spectrogram, I would encourage the authors to experiment with various masking strategies. 

clarity,_quality,_novelty_and_reproducibility: This work is clearly presented, and the results are based on well-known, reproducible baselines including AudioSet and VGG Sound.

summary_of_the_review: In short, the paper presents a valid idea, but the experiments could be augmented and extended to further strengthen the arguments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contributions of the paper are twofold 1) to extend the recently-proposed, well-known masked auto-encoder (MAE) for audio-visual multi-modalities and 2) to combine contrastive learning with masked prediction objective to learn a joint and coordinated representation.  Extensive experiments validate the effectiveness of the proposed method named contrastive audio-visual masked auto-encoder (CAV-MAE), providing new SOTA performance on selected benchmark datasets. Various ablation studies have also performed to help understand the characteristics of the proposed method.  The paper is well-written and easy to follow. 

The paper received unanimous acceptance from 5 confident reviewers with scores 86686. The authors were very responsive in answering the reviewers' comments and in conducting extensive extra experiments as requested. As a result, 3 out of 5 reviewers raised their scores during the rebuttal process. 

I am recommending acceptance.

justification_for_why_not_higher_score: This is a very interesting yet not ground-breaking work. 
justification_for_why_not_lower_score: The proposed method extends the well-known masked auto-encoder method for processing audio-visual data and combines contrastive loss with masked data modelling. The experimental results are strong.
Decision: Accept: notable-top-25%