Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of contrastive multimodal pre-training (assuming triplet training data <video, text, audio>). It starts with uncovering the misalignment phenomenon existing among temporally-associated video clips, subtitles, and audio tracks, which make them falsely considered positive training pairs. It then quantitatively measures the gradient conflicts resulted from this misalignment. Ideally, given a modality-agnostic backbone, gradient updates come from different loss terms on the same positive triplet should reinforce each other, reflected by non-negative cosine similarity. However, this is not the case for existing methods. To this end, the paper proposes two techniques, namely Gradient Realignment (forcing gradients updates from different losses to have non-negative cosine similarity) and Gradient-based Curriculum Learning (filter out samples with low gradient similarity), to alleviate the gradient conflicting issue.
strengths_and_weaknesses: The proposed idea is sensible and interesting. Overall, the experimental evaluation is comprehensive and supports the effectiveness of the proposed method.

The motivation of the idea, however, could be more upfront. As the paper suggests in the rather later part of the paper (line 255), the proposed method is only studied in the modality-agnostic single-backbone setting. However, without this knowledge, understanding the motivation of the paper (gradient conflicts) is cumbersome. Please consider stating the problem setting clearly as early in the paper as possible (on or before line 61).

Moreover, is the proposed method only applicable for modality-agnostic architecture with triplet training samples? How about the modality-specific setting, where the gradients $g_{va}$ and $g_{vt}$ conflict on the video encoder only, but not audio encoder and text encoder (as it is the pivot)? Or in the bi-modal modality-agnostic setting (e.g., image and text [r1]), should the gradients backprop from the image feature and text feature be harmonized? It would be great if the authors could share insights/analyses on whether the proposed method could tackle these scenarios.

The final question is related to the previous one. As $g_{va}$ consists of gradients from both the audio part and the video part, $g_{vt}$ consists of gradients from both the video part and the text part, should this "micro" harmonization between all four of them be considered in addition to the "macro" harmonization showcased in the paper? Any insights are appreciated.

[r1] You et al., MS-CLIP: Towards Modality-Shared Contrastive Language-Image Pre-training. ECCV 2022.
questions: Typos:

i) Line 51, the figure reference is unknown.

ii) The arrow in Colume 6 Row 3 of Tab. 1 is missing.
limitations: Appear to be sufficient.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes two regularization strategies to harmonize the multimodal pre-training. The motivation is that cross-modality alignment (CMA) is problematic when the pre-training data is noisy and misaligned. The contribution of this work is: (1) introducing the cosine similarity between video-audio and video-text gradients as a surrogate indicator on how noisy the alignment of a sample triplet (video-audio-text) is; (2) introducing cross-modality gradient realignment to re-project the gradient direction if they have negative cosine similarity; and (3) using cosine similarity to prioritize training with well-aligned samples. The effectiveness of the proposed strategies are comprehensively evaluated.

strengths_and_weaknesses: ---

Originality: misalignment is a well-known problem in multimodal pre-training data, but is also tricky to solve. The two regularization strategies proposed in this work are novel in the multimodal pre-training community.

---

Quality: 

Strengths: using cosine similarity of video-audio and video-text gradients as the indicator of misalignment is reasonable. The effectiveness of the proposed regularizations are evaluated on various downstream tasks. The work is complete.

Weakness: (1) cosine similarity can only be applied to the scenario with at least triplet. When the pre-training data only contains video-text pairs or only contains video-audio pairs, this surrogate indicator does not work. (2) According to Table 1, the majority of the improvement in video action classification tasks comes from curriculum learning (CL). Combining with gradient realignment (GR) even hurts the performance of CL-only. 

---

Clarity: this paper is well written and well organized. It is easy to follow. Since this work is built upon VATT and provides enough training details, it is easy for readers to reproduce the result.

---

Significance: misalignment is an important problem in contrastive-based representation learning. Although it is novel to use cosine similarity of two gradients to measure the misalignment, the performance improvement is marginal.

---
questions: 1. Line 51: “Figure ??”. This should be a typo.
2. Table 1: when the pre-training datasets are HT100M and AudioSet, the performance of VATT on Text-Video retrieval is quite low. For example, this paper reports R@10=29 of VATT on YouCook2. However, the original VATT paper reports R@10=40.6 (the higher the better) for YouCook2. 

limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper challenges the cross-modality alignment (CMA) assumption in multi-modal pre-training. The authors found that CMA can be weak and noisy because of the often poor cross-modality semantic alignment. They proposed two techniques, i.e., gradient realignment and gradient-based curriculum learning. Experiments on conducted with VATT, on some of the large-scale pre-training datasets. 


strengths_and_weaknesses: Strengths:
-	The authors studied a novel and important problem. CMA is traditionally assumed by default in multi-modal learning (and kinda why most prior works was restricted to instructional video data). This paper takes an important step toward the wild. 

-	The observation of poor CMA is convincing (Figure 1 illustration, and Figure 3 quantification by gradients). This problem of misalignment has not been studied in the existing works on self-supervised multimodal learning, and the insight to analyze the problem from the perspective of the gradient is new to my best knowledge.

-	The two proposed techniques, albeit simple, seem effective and low overhead to implement

-	The authors spent lots of effort on experiments. The benchmarking in Tables 1 and 2 is comprehensive. 

-	The authors are able to scale up VATT pre-training to the more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-art performance in a modality-agnostic setting. This is impressive and validates the benefits of their proposed techniques. 

Weaknesses: 
-	The method has moderate technical novelty since it adapts several known wisdoms from multi-task learning. 

-	This paper in general lacks an ablation study. I can totally understand the (re-)training overhead is huge for such models, but at least some rationale or comparison on choosing their methods’ ad-hoc hyperparameters (there are many) could be provided? 

-	How might the proposed technique scale beyond VATT? E.g., to last V-L models such as DALL-E and CLIP? Do they also suffer from similar problems, and can your methods be helpful? (I’m not asking authors to re-train CLIP or alike, just idea sketching).

-	The writing quality of this paper is barely readable and needs proofreading.



questions: Please see the weakness part 
limitations: No, the current version claimed it has no negative social impacts.
review_title: null

Meta-review: metareview: Authors present a mechanism to improve triple cross-modality alignment between video-text / video-audio with a shared encoding backbone. The basic idea is to measure the cosine similarity between the gradients coming from video-text and video-audio. When the gradients conflict, the authors postulate that this is caused by mis-matched data. In order to address, the gradients can either be re-aligned, or this information can be used for curriculum learning to filter noisy samples.

Authors study on 6 datasets, modifying VATT with their approach, and demonstrate consistent improvements in performance.


Pros:
- [R] Well written (disagreement) and easy to follow.
- [R] Interesting method with clear impact.
- [R] Experimental evaluation is comprehensive
- [R] Method can scale up to noisier data (YouTube8M) to demonstrate even further gains in performance.

Cons:
- [AC] Authors assume that when gradients conflict, this correlates with noisy data. But no study was performed to confirm this. The method may improve performance on end tasks, but the motivation as to why it improves performance is only based on assumptions, and no data. Authors could greatly improve the paper by doing a study on a random sample of conflicting gradients to confirm that the data samples in those situations are misaligned more than when the gradients are aligned.
- [R] Only applies to modality agnostic single backbone setting. Authors should make this more clear in the paper. Authors have addressed this concern.
- [R] Writing could be improved. 

Unanimous reviewer ratings on accept. In light of the utility of the approach, the measured improvements in performance, and reviewer comments, AC recommends accept. AC, however, recommends to authors to reframe their writing to emphasize that they *assume* gradient misalignment correlates to noisy samples, or perform an experiment to confirm that this is the case.

AC Rating: Accept
confidence: Certain
Decision: Accept