Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization
Authors: Junru Wu
Yi Liang
feng han
Hassan Akbari
Zhangyang Wang
Cong Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cross-Modality Alignment
Multimodal Pre-Training
Modality-agnostic

Abstract: Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive learning methods often enforce the feature consistency from cross-modality inputs, such as video/audio or video/text pairs. Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned. For example, even in the (often adopted) instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos collected from unconstrained internet sources. We conjecture that might cause conflicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger and more heterogeneous data. This paper first verifies our conjecture by observing that, even in the latest VATT pre-training using only narrated videos, there exist strong gradient conflicts between different CMA losses within the same sample triplet (video, audio, text), indicating them as the noisy source of supervision. We then propose to harmonize such gradients during pre-training, via two techniques: (i) cross-modality gradient realignment: modifying different CMA loss gradients for one sample triplet, so that their gradient directions are in more agreement; and (ii) gradient-based curriculum learning: leveraging the gradient conflict information on an indicator of sample noisiness, to develop a curriculum learning strategy to prioritize training with less noisy sample triplets. Applying those gradient harmonization techniques to pre-training VATT on the HowTo100M dataset, we consistently improve its performance on different downstream tasks. Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset to further improve the state-of-the-arts.

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization
Junru Wu" Texas A&M University
Yi Liang Google Research yi wallian@@goggle.om com
Feng Han Google Research bladehan@google. com
sandboxmaster@tam edu
Hassan Akbari
Zhangyang Wang University of Texas at Austin atlaswang@utexas. edu
Cong Yu Celonis Inc. / CeloAI cong. yuscelunceensseeeee com
Google Research
hassanak@google.com com
Abstract
Self-supervieed pre-training recently demonstrates success on large-scale multi- modal data, and state-of-the-art contrastive learning methods often enforce the fea- ture consistency from cross- moddlittyl inputs, such as video/audio or video/text pairs. Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned. For example, even in the (often adopted) instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos collected from unconstrained internet sources. We conjecture that might cause conflicts and biases among modal- ities, and may hence prohibit CMA from scaling to training with larger and more heterogeneous data. This paper first verifies our conjecture by observing that, even in the latest VATT pre-training using only narrated videos, there exist strong gradient conflicts between different CMA losses within the same sample triplet (video, audio, text), indicating them as the noisy source of supervision. We then propose harmonize such gradients during pre-training. via two techniques: (i) cross-modality gradient realignment: modifying different CMA loss gradients for one sample triplet, so that their gradient directions are in more agreement; and (ii) pradient-baeed curriculum learning: leveraging the gradient conflict information on an indicator of sample noisiness, to develop a curriculum learning strategy to prioritize training with less noisy sample triplets. Applying those gradient harmonization techniques to pre-training VATT on the How To100M dataset, we consistently improve its performance different downstream tasks. Moreover, we are able to scale VATT pre- pre-training to more complicated non non- narrative Youtube8M dataset to further improve the otate-of-the-arts.
Introduction
Self-supervin pre-training scales up deep learning leverage massive unlabeled data. Beyond the maturity of pre-training over single-modality data such as language (2 or images [3] the recent success multimodal pre-training [4] (5) demonstrates the versatility to synergize the rich multimodal information and to benefit a variety of downstream tasks. Many methods of this category [6-8, 4)
especially the latest contrastive pre-training methods (5 DHT,, consider the most organic supervision as the cross-modality alignmentt (CMA), e.g., in the same video sequence, the video frames, audio
Work done at Google Research
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

"On football pitch"
"Whether it's in kitchen"
"World Cup Semi Final"
"Raising fund UNICEF"
Aligned
Misaligned
SHANE FILANNNL
FILAN
Figure I: Examples of mross-modality alignment and misalignment existing in the complicated and non-narrative Youtube8M dataset [1] Here we show video sequence whose theme is football match, while in some of its clips: (a) the text mentions content unrelated to the football, e-g.. "kitchen", "kitchen,,"raisingg "raising fund"; and (b) the visual content could also be unrelated to the football, e.g., TV cutscenes, the scene the broadcast studio, etc.
stream, and text scripts are temporally aligned and hence should naturally have correspondence. Such assumed cross-modal correspondence could be exploited as "mutual supervision" for each modality to learn consistent semantic features with others. For example, the latest Aadeo-Audio-Tedn Text Transformer (VATT) [TT] binges on this multimodal CMA priori to train a transformer based architecture (even single backbone by sharing weights among three modalities). using two contrastive losses to align video-text and video-audio pairs, respectively. Their results set many new records in downstream tasks with neither supervised pre-training nor predefined inductive biases, showing strong "universal" generalizability despite the modality and domain gaps. In this paper, we focus studying gradients modality-agnostic Linge-backborr setting of VATT .
Unfortunately, the CMA assumption rests on a very shaky foundation, especially for uncurated videos collected in the wild. The CMA be weak even for the instructional videos such as HowTo
[12 that are commonly adopted as the current multimodal pre-training data source. For instance, speaker can refer to something before or after actually demonstrating visually; she or he could also skip verbally explaining something that is visually happening, because that may be trivial or be already clear enough in the visual context [13] Such discrepancy would be amplified if considering many other outliers in video (e.g. background objects) or language (e.g. off-topic small talks). The semantic misalignment will be even more severe, and in unpredictable forms, when we go beyond instructional videos and explore training with narrativve free-from free-from internet videos. Figure shows a few examples of such cross-modal misalignment the non-narrative Youtube8M dataset [I], which would put CMA-guided pre-training in jeopardy. This important problem is yet under-studied the self- supervieed multimodal pre-training literature. That said, there has been a few recent efforts such Multiple-Insannce Instance Learning (MIL) NCE [13] and noisy pair pruning in [4]
We conjecture that the vanilla CMA might provide only weak, noisy and even misleading supervision for multimodal pre-training; hence it might constitute a hidden bottleneck when scaled up with more realistic data (beyond instructional videos) whose modalities are poorly aligned. We verify our conjecture by inspecting two large-scale multimodal datasets, HowTo100M [15] and Youtube8M M. Specifically. when we adopt pairwise CMA losses (e.g. video-audio, video-text) within the cross-modal sample triplets (video, audio, text), majority of the gradients strongly conflict with each other terms of their directions. We further identify (in Figure 33 that such strong conflict in gradients are correlated to noisiness of samples, indicating the noisy and misaligned supervision could be an important cause gradient conflicts. Moreover, the cross- cross-modal conflict leads to not only unstable training convergence, but also modality-spccife overfitting; c.g. semantically strong representations for one modality while collapsed representations for another [16], yielding weak overall performance in mross-modality tasks such as VideoVideo Retrieval as shown in Table II
Build upon the above conjectures and observations, we propose harmonize those gradients so that they become mutually compatible in learning the unified representations. Specifically, we introduce two techniques: (i) cross-modal gradient realignment, where we modify different pairwise CMA loss gradients to align their directions for the same sample triplet, using gradient surgery [17 developed for multi-task learning; and (ii) gradient-bar curriculum learning, where we leverage the gradient conffict information to indicate sample noisiness (c.g. triplet whose CMA gradients are more

---

agreement is considered less "noisy" and would be prioritized for cross moosl pre-training) and curriculum learning strategy is developed accordingly. Both techniques are found boost VATT performance on downstream tasks, not only on instructional videos, but even more when the more complicated non-narrative data is involved.
Our contributions can be summarized as the following:
We suggest that the commonly assumed CMA might be an important yet overlooked perfor- mance hurdle for scaling up multimodal pre-training. and we observe severe misalignment even from the state-of-the-art (SOTA) model [[]] on the relatively aligned data [2]
We propose to consistently improve the pre-training pipeline the only baseline in this direction, VATT (modality-agnostic setting), resulting in better downstream performance.
With the help of our proposed techniques, we consistently improve the pre-training of VATT, resulting in better downstream performance, e.g. up to 58% gain in the rank metric of video- text retrieval task. Moreover, we are able scale VATT pre-training to more complicated non non-narrative Youtube8M dataset, which further yields new state-of of-the-art performance in a modality-agnosstcc setting.
2
Related Work
2.1
Self-Supervised Multimodal Pre-training
The most organic self-supervision can arguably be found in the multimodal data that abundantly available the digital world: their temporal alignment naturally lead to cross-modality mutual supervision, hence meaningful features can be learned without requiring human annotation. Mul- timodal pre-training can thus leverage richer semantic cues than single-modality counterparts, and outperform in various downstream tasks such as image classification [T), video action recognition [3] audio event classification [5] [O], and video-text retrieval [1 Most of those methods formulate pre-text tasks, including cross- cooss-mdalltyy correspondence 6 10], modsltyy
clustering [18. cross-modality longer context prediction [19]. or a combination of multiple pre-text
tasks 20
Contrastive learning 3 21 has recently emerged as one dominant approach in self-supervised learning. In multimodal mre-training, the temporal co-occurrence naturally supplies the positive sample to contrast with. [5] adopts a multimodal versatile network to embed each modality into the same vector space, that trained from end end with multimodal pairwise contrastive losses (video-audio, and video text). CrossCLR (9] further takes the intra-modality similarities into account. [10] generalizes the instance discrimination idea design cross-modal discrimination task, i.e., predicting which audio matches a video. Most recently, VATT [I]] combines the strength of convolution-fre Transformer and multimodal contrastive learning. Specifically, it first validate the idea of using modality-agnostic single-backbone transformer, to work on video, audio and text modalities, it follows the exact BERT 22 and ViT 23 architecture, except injecting modality- specific tokenization layers and linear projections.
hinge on the "free" mross-modalities correspondence mined through the multimodal contrastive loss, i.e., the CMA assumption. As a result, they are often trained with well curated or narrated video datasets, such as AudioSet 24 YouCook2 [15), and HowTo100M [12. It remains questionable whether such CMA-guided contrastive learning can generalize well to larger-scale multimodal data in the wild. In fact, even in the How 10M dataset, the authors of (12) estimate that around 50% of clip- dii-nnarattion pairs are not well aligned. The validity of CMA has arisen concerns. Most related to us is [33] which explicitly models the misalignment noise using the MIL-NCE loss, that is inherited by VATT [1]
Despite their success, existing multimodal contrastive learning methods 5
2.2 Multi-task versus Multimodal Learning
While earlier self- seff-supervised works used different modality speciffc encoders, the latest multimodal pre-training sees a tendency to explore a versatile and "universal" model shared across modalities [5 It is easy to notice the resemblance between multimodal learning (especially with a unified backbone) and multi-task learning (25 26). The later often assumes multiple tasks to share transfer- able knowledge and can help each other learn. Yet in practice, different tasks can be heterogeneous in
nature too, often leading to heavily misaligned gradient scales (27) or directions [77. Such conflicting gradients from multiple losses would lead to several undesirable effects, including (i) strong bias
3

---

Audio Tokerizer
Audio Head
Video-Audio CMA Loss
000000
Unifled Transformer Encoder
Video Tokenizar
Video Head
000000
9
Video- Text CMA Loss
"He's professional whether kitchen on the football pitch"
Text Takenbrer
Text Head
Forward Path
Gradient Flow g
Gradient Flow 9vt
Figure 2: One main model used in this paper: the VATT model 11 with a modality-agnostie, single- backbone Transformer by sharing weights among the three modalities. VATT is trained with two contrastive losses between video-audio and video-text pairs, respectively. The positive cross-modality pairs contrastive with are based their temporal co-oceurrence (e.g., the CMA assumption). We primarily study the conflicts between gradients yea and gut, which indicate the often poor alignments, and discuss how harmonize their conflicts.
towards learning some tasks with largest gradient scales; and (2) slow and unstable convergence. Similar problems often faced by multimodal pre-training
As a result, several works have been developed to mitigate the gradient dilemma in multi-task Iearning. Methods such gradient normalization (27] and adaptive loss weight [28 29 address the gradient scale/learning speed imbalance issues. Gradient surgery [7 tries to project gradients with conflicting directions onto the normal plane of each other, to prevent each gradient's 's interfering component from being applied to the network. Later, Gradient vaccine (30 further generalizes this idea extending the gradient projection to arbitrary angle, and uses the exponential moving average of cosine similarity to measure the aligmment between different tasks. When it comes to the multimodal learning, [31] investigates optimally blending modality-specifft gradients by linearly re-scaling them. Meanwhile, to our best knowledge, there has been no prior study on modality-speciftt gradient directions.
Methodology
Pre-requisites: Without loss of generality, we focus on the SOTA multimodal contrastive learning model, VATT [11), backbone setting due to its compelling performance-effccinng trade-off, and due to the fact that is the most intuitive setting modify gradients (shared between paired modalities) w.r. different end-point objectives.
default subject of study; and we follow its modality-agnostie single-
In our modality-agnosti single-backbone setting of VATT, the goal of pre-training is to find the parameter of the model Se to learn meaningful features for target modalities (video, audio, text), usually measured by set of pairwise similarity metrics. VATT uses two pairwise contrastive losses for video-audio and video-text respectively to solve cross-moot moosl alignment:
min CMA,a + CMA,((0)
where CMA and CMA,t penalize the cross-modal alignment for video-audio and video-text pairs, using the Noise-Contrestive Estimation objective 33. respectively.
Figure 2 illlssrates the VATT pipeline. First, video-audio-text triplets are sampled from random temporal locations, and then video-text and video-audio pairs are formed accordingly. Positive pairs are formed by selecting the two modalities at the same temporal locations from the same video clip: while the negative pairs are obtained by randomly sampling any video, audio, or text from other video clips. We follow the convention in [5 [1 use the vanilla NCE loss for video-audio pairs, and to
use the MIL-NCE loss proposed in 13 to align video-text pairs. Hence, CMAps and CMA
defined as follows:
exp(2,T 2a/T)
CMApa (0) log
(2)
exp(2x] 2//T) + EveN exp(22 2'a//)
EnEPA(E) exp(2. exp(x.2217)
CM) = -logg
(3)
ExePh(E) exp(2] 2//7) EveN exp(2' 2'eT)

---

where N is the pool of negative pairs, Pr(zi) denotes the k-neighbouring narrations surrounding the original text 2, The gradients of two pairwise losses CMA and CMA,ts denoted as Ina and Jut will be together applied (i.e. simple adding) to updating the model fo-
3.1 Observation of Conflicting Gradients
While qualitatively recognizing the cross-modal misalignment is easy, it is a highly challenging endeavor to quantify the misalignment and to automatically find misaligned triplets at scale, due to the vaguely defined criterion as well as the absence of labels [33 14). Therefore, the foremost need is to identify a surmgate indicator how noisy the alignment of a sample triplet is, without making unrealistic assumptions nor incurring much computational overhead.
Our empirical choice of surrogate indicator is the directional alignability (i.c., vector angle) of gradients generated by different pairwise cross-modal losses, i.c., the cosine similarity between Jee and Iur in the particular example of VATT, after flattening them into two vectors. The choice is firstly rationalized by the coherent gradient hypothesis [3], that healthy model training should witness the overall gradient stronger in certain directions along which the samples' gradients reinforce each other. Secondly. comparing the gradient directions between multiple pairwise losses could be considered as an ensemble or "cross-check": intuitively, if both video-audio and video-text pairwise consistency lead to the same update direction, then there is good chance that those modalities are well aligned and the update direction is reliable; otherwise, at least one pair (video-audin or video-text) might suffer from misalignment and provides noisy cross-modality guidance.
Overall, we conjecture that: aligned video-text-audio triplets should have higher cosine similarity for Goo. and gut, and vice versa.
To further validate the conjecture, we conduct a sanity check, where we started from a pre-trained VATT network and further optimize it for 200 iterations with batch size of 64, Youtube&M dataset. We then randomly sample ridco-text-audin triplets out of the samples with top 5% and bottom 5% most aligned gradient, measured by cos(gea> Pea)- In Figure 3. we visualize 10 frames within 32-frame video clip and its corresponding text narration (including 8 neighbouring text narrations similar
semantically aligned words in the corresponding text narration (highlighted in green) thus enjoy better cross cross-modality alignment, while the bottom 5% group triplets has fewer semantically aligned words, therefore are much more noisy in their alignments. We include more visualization samples in the Appendix.
13) We visually observed that the top 5% group triplets has
cos 9..)
Top 5%
0.0858
you
Sanko
one the three mldor...ddde with crack rewile.. salea brazil and countriks worldwide... walers market shoad ncude and luden
0.0583
feam america cociting meen lierrall the proudest moment of... -cpp in square aguinvt tiger world this camning giy find out we spent some time with him while practicing. hed hoped he Rittahed 29th grahare alread scored a decent.... the presidents cup how oftan de you practice like you are you out... culled that playoll board fournement dide's quis
the pressure the energe emotion
knste
things
looking forward...
who
Bottom 5%
-0.0547
lightning could pull off rumbo freahh it they can hurr up heme airght Jos jonse... hhet very close eunto closm... all ralens his fing Jonns and roger. looks lie brand command gorma firish
in place shel..hes poing they gottu faish figure out Sunko gotna get third... comments are gornna <ppii strogping
-0.0533
this advice loon has than sootte coffee... help when you need horm the wiadom of... bofore banofi then the break remerbirr that il mosta will love besh brownss... .keenan wills cindl and donnis in look the beet beloved....
theme timee contiict harmony would be. be...fndd
ank
bleanings jayoue
especiatly
<umko
their love another
Figure 3: Qualitative example of our proposed measure based agreement of gradients on Youtube8M dataset, semantically aligned words are highlighted in green, cos(gea, Bot) reflect the agreement of between Ivo and 9ot, by measuring their cosine similarity.
5

---

With the assumption that misalignment in gradients correlates to the noisiness of the sample's cross-modality alignment, we use the former as the computable surrogate indicator of the later, and examine the cosine similarity between gradient Yva and fot (over 500k iterations) of VATT on the HowTo100M dataset [2]. Figure 4a| we plot the distribution of cosine similarities, cos( Bot), across 500k iterations of pre-training. We observe that cos(gea: 9et) at any iteration resembles normal distribution, and about half of the gradients 9va and Geo have misaligned directions (negative cosine similarities). The plot echos the empirical finding in [15] that around 50% of video-text pairs are not well-aligned on the How T0100M dataset. In the Appendix, we have included similar plots for the YouTube8M dataset M. Comparing those plots verify that non-na nor-narrative datasets have much worse misalignment than narrative ones, hence challenging the CMA assumption even more.
vit
(a) Multimodal pre-training baseline
(b) w/ Cross Moosssstyyltttyy Gradient Realignment
(c) w/ Gradient-baid based Curriculum Learning
Figure 4: Visualization cosine similarity between gradient Jea and get across 500K Iterations on the HowTo 100M dataset. (a) shows the baseline setting in multimodal pre-training: (b) with only cross-modality gradient realignment; (c) with only gradient-base curriculum learning. Youtube8M follows very similar observations, see Appendix.
The existence of conflicting gradients not only makes the convergence harder, but also makes the learned representation heavily overfit some modalities. In case, we observe the training to be biased towards video/audio modalities, which favors the downstream performance of video/audio classification task, yet sets back the performance of video-to-test retrieval task (verified in Table T).
3.2
Harmonizing Gradients during Pre-training
We present two strategies to mitigate the conflicting gradients: a "hard" way to project gradient directions, and a "soft" way to progressively select sample triplets in curriculum.
3.2.1
Cross-Modality Gradient Realignment
Since Iva and Ivt are misaligned in terms of their directions, one nat- ural idea is to re-align the cross- modality gradients, by enforcing or re- projoeeet their directions be aligned. An intuitive choice is an ef- fective remedy developed for multi- task learning named Gradient Surgery 177. Here, instead of applying it to gi and gj where i, are the two hetero- geneous tasks, we apply to Gva and Jot, which refer the gradients of the two pairwise losses CMA,p and CMA, w.r.t the weights in the model, following our rationale in Section 3.1
Algorithm 1 Cross-Modality Gradient Realignment
Require: Model parameter 0, minibatchs B minibatchs
Bes
for (bear bet) E (Brar Bet) do
Iea + VoCMA (0), gut + VoCMA,, (0)
Jea fatten((@aa) ). Jut +- flatten(goe)
Joa + Ivas Jot Jut
Jea gut < then
Iea + 9va Auni
projection Jee Iea
l
Out + Ove
projection Jea. + Jut
Ilaval
end
Jca reshape(@ra). goe reshape((ws)
AO < gut + Joa
sum up gradients update parameter
update 8 with A9
If the gradients Iva and Jve have nega- tive cosine similarity, we alter them by projecting each onto the normal plane similarity, we retain the original gra-
end for
of the other. Otherwise, if Gea and Yot have zero or positive dients. We refer to this particularly adapted form of gradient surgery as
Gradient

---

Realignment. Its update procedure is outlined in Algorithm|] [ and the overhead very cheap, mainly just computing inner products of two flattened gradients. We also tried another similar algorithm named Gradient Vaccine [30], and the results are almost identical to using Gradient Surgery [17. We hence only report one for the sake of conciseness.
3.2.2 Gradient-based Curriculum Learning
We next propose a gradient- curriculum learning strategy that gradually identifies and re- moves more misaligned samples based the similarity indicator as the training goes For- mally, given a video-audio pair bea and video-text pair dyt. we only update the parameter cos( 9ut) > y and drop this sample triplet otherwise. This ensures tize training with well-aligned samples while suppressing the noisy supervision from potentially misaligned samples. The drop rate here is controlled by a cut-off threshold 7. In general, 7 is a negative value monotonically increasing during training, and there is an underlying rationale. Initially, the network is under-trained and the similarity between its learned features is not reliable to indicate strong CMA; hence we are more con- servative in excluding samples. As the network becomes more sufficiently trained, the features become more se- mantically aligned, allowing us to re- move more misaligned samples the later stage of training. The full pro- cedure of Gradient-based Curriculum Learning is outlined in Algorithm 2 We empirically choice To = -0.3 at the beginning and linearly increase to 7500K at the end, we include
gradually priori-
Algorithm 2 Gradient-based Curriculum Learning
Require: Model parameter 0, minibatchs Br minibatchs Ber, initial To for (beas bur) E (Bear Bar) do update 7y
b curriculumly update 7
Iva VoCMApa (0), gut t of
Ioa + flatten (9ea). gut +- flatten(get)
gea 9ut > 7 then
Yva < reshapee (Iva)s Oot x- reshape(gm)) sum up gradients
AO + Iva + 9ut
update 0 with AO
> update parameter
end if
end for
an ablation on the adaptive choice of y in the Table 2
Similar classes of ideas, i.e, "gradually focusing a small training subset", have been explored by a prominent class of algorithms which rely sample selection to robustify [34136] or to accelerate 37 training. Most of them use the "small loss trick" wherein a fraction of samples with smallest loss values below a certain threshold are considered as reliable. However, they did not explore gradient conflict indicators, which targets solving our CMA problem here. Also note that our two steps can fully reuse the cosine similarity computed, applying them together incurs extra overhead.
4 Experiments
4.1 Pre-Training Datasets and Settings
In this paper, we focus on studying gradicnts in a modality-agnose single-backbone setting, thus we do not consider multimodal pre-training baseline with modality-specific models, e.g. 38), MMVB), PerceiverIO (99) AVLnet 40) Apart from VATT[]] there exist other concurrent work that modality-agnosti setting 20, 41 42). yet they all have their respective limitations. For example, [n) lacks audio modalities thus cannot constitute video-audio CMA loss, VLM[4]] applies separate attention mask for each modality while Poly VIT 42 use alternate training between modalities and tasks, thus eliminate the possibilities of any gradients conflicts. Therefore, this leave 11 be the most suitable baseline besides it SoTA performance.
Our pre-training adopts three settings for comprehensive comparison: (1) using HowTo100M as training set, following 20 [2]; (2) combining HowTo100M and AudioSet as training set following [5] and VATT Ii: and (3) additionally merging a noisy Youtube8M dataset alongside with How To100M and AudioSet, create much more compound and complicated training set than considered before, to stretch-test our proposal's 's benefits to scale up pre-training in uncurated and poorly aligned data.
For each setting, we i.i.d sample video-audio-text clips from the mixture of all candidate sets in every batch. HlowToo000 Audioset or Youtube8M, if applicable). For all setting, we only use subset of 100M. AudioSet, Youtube&M, Kinetics400 and Youcook2 dataset in compliance with Youtube's 's wipe-out policies.
7

---

How Tow00M0'' [1] is large-scale dataset of narrated videos with an emphasis on instructional videos that have well synchronized modalities. The training set consists of 127 Millions video clips with the corresponding close-captioned text from automatic speech recognition (ASR).
AudioSet' [24] is a large-scale audio-visual dataset originally intended for audio event classification. It consists of 1.78 Million 10-second clips sampled from Youtube Videos that contains variety of audio tracks, such as musical instruments, animals or mechanical sounds.
Youtube8M [ is a large-scale video classification dataset based on Youtube videos. It consists of diverse vocabulary of 3862 visual entities and total number of 8 Million videos. To the best of our knowledge, we are the first to scale contrastive multimodal pre-training beyond instructional videos, onto this noisy Youtube8M videos that more faithfully represent videos the wild. For data pre- processing, we split each video into 58 clips and use ASR close-captions to generate the corresponding text, resulting in 165 Millions video-audio-text clips.
4.2 Pre-Training Settings
Network Architecture: For all of our experiments, we use modality-agnostic Transformer variant in [Τ]. SATT-MA-Mediung Specifically, we use 12-layer Transformer, in which each layer has a foed-forward MLP with hidden size of 4096, and 16 attention heads with a hidden size of 1024.
Pre-training Hyperparameter: We strictly follow the setting in [. pre-training VATT from scratch with Adam optimizer with an initial learning rate of le-4, 10k warmup steps, 500k steps in total, a batch size of 2048 and using a cosine learning rate scheduler to anneal the learning rate from le-4 to 5e-5. Our framework is implemented in Tensorflow 2.8, and train with 256 TPUV3s, it took total of 3 days to train our models.
4.3 Downstream Tasks for Evaluation
We evaluate the performance of our pre-trained representations when tuned towards various down- stream tasks, including variety of video/audio and cross modalities tasks.
Video Action Recognition: Following [5), we evaluate on UCF101 [43] (101 action classes, 13,320 videos) and the HMDB51 4M] (51 classes, 6,766 videos) benchmarks. For both settings, we directly attach a linear classifier on the learned representations while freezing the transformer backbone. We also evaluate Kinetics4004 45 dataset (400 classes, 234,584 video), where fine-tune the transformer backbone.
Following 5. we used [66 (50 classes, 2000 audio clips) as the benchmark for audio modalities. Similarly, we directly attach and tune a linear classifier on top of the frozen transformer backbone.
Audio Event Classification:
Zero-shot Vixeo-Text Retrieval: To evaluate our performance on cross-mddality task, we follow [5] using YouCook2 [15 (3. video-text clips) and MSSRVVTT 4 (1K video-text clips) as our evaluation datasets. We directly use the embedding from the pre-trained model without any fine- tuning, and report the recall at 10 (R @ 10) and Median Rank as described in [5].
4.4 Result Analysis HowTo Now-o10001 1o100M and AudioSet Pre-training
Gradient Re-weighting We set up simple gradient re-weighting baseline similar to 131. where we simply scale up the gradient magnitude of goo by 2.5 times and scale down the gradient magnitude of its counterpart Per by 0.5 times, we denote this baseline as Gradient Re-weighting (VA), we also did it vice versa and denoted it as Gradient Re- Reightig (VT). As shown in Table I we did not observe much performance boost with gradient weightting,, on the contrary, there are significant performance drop in Text- Video Retrieval task on both YouCook2 and MSRVTT datasets.
Cross-Modality Gradient Realignment Compared to Gradient Re-weighting, our Cross- ModaliMy Gradient Realignment mitigates the gradient conflicts by modifying gradients direction. In Table [ we can see significant performance gains in video-text retrieval task (e.g. 58% gain in rank metric on YouCook2, 46% gain in R@ 10 metric on MSR-VTT, while maintaining a comparable performance in video/audio tasks. Since video-text retrieval task heavily hinge on cross-modalin alignment iniiismmpeeee
(CMA), this improvement verify the benefits of re-aligning the cross-modality gradients.
IIt is important to note that Howto100M. Youtube8M, AudioSet, Kinetics400, MSRVTT and YouCook2 videos are sourced from YouTube. Many of the videos have been made explicitly unavailable [48] hence we only train and evaluate over the subset of data that is publicly available at the time.
8

---

Dataset
Tasks
Video Action Cls
Lext-Video Retrieval
Audio Cls
Dataset
UCF101
HMDB51
Kinetics400
YouCook2
MSRVTT
ESC50
Metric
Top11
Top5t
Topp1
Topst
Top1"
Top5t
Rank]
R 101
Rankj,
R 101
['Tppt
TopSt
VATT I
78.53
95.24
57.36
86.07
74.71
92.69
93.50
17.10
73.00
16.70
71.50
91.75
+ RW (VA)
78,24
95.01
58.74
85,39
70.55
90.41
168.90
9.24
100.20
13.56
71,56
93.02
+ RW (VT)
78.03
95.35
58.23
86.80
75.66
92.80
90.35
19.26
62.50
18.02
71.29
91.38
HTT00M
+ GR
78.44
95.37
54.38
83.64
76.72
92.72
47.00
24.94
68.00
19.20
69.00
93.00
+ CL
79.10
96.16
56.41
86.06
77.25
93.38
40.00
26.01
56.00
23.90
72.50
94.25
+ Both
79.24
96.58
58.24
87.37
76.59
93.26
42,00
25.87
54.00
24.50
72.05
94.16
VATT (11]
84,40
63.10
79.23
94.30
34.00
29:00
67.00
23.60
81.20
RW (VA)
83.44
97.28
59.55
88.02
76.56
93.52
238.50
6.80
147.00
12.30
81.75
97.25
+ RW (VT)
84.42
97.49
62.30
86,61
78.59
94.17
76.00
18.72
120.00
15.20
80.75
96.75
HT100M
+ GR
84.77
97.38
62.30
90.38
79.29
94.32
29.00
31.65
70.00
21.40
81.50
97.00
AudioSet
+ CL
86.04
97.75
65.45
88.94
79.89
94.71
33.00
29.17
65,50
20.97
82,00
97.25
+ Both
85.46
97.58
65.52
89.74
79.26
94.48
31.50
30.26
69.50
19.96
82.00
97.00
VATT
88.28
98.73
65.84
91.43
79.39
94.56
29.00
29.66
56.00
26.90
80.75
97.00
+ RW (VA)
86,97
98
61.06
89.66
77.70
93.83
99.00
14.25
75.50
19.70
83,50
97.25
HT100M
+ RW (VT)
88.19
97.96
61.13
90.51
78.43
94.38
27.00
31.07
48.50
27.70
82.25
96.75
AudinSet
GR
87.49
98.10
60.99
88.35
79.73
94.57
32.00
29.56
60.00
27.20
85,00
98.00
+ CL
89.02
98.33
65.77
92.15
79.70
94.80
31.00
31,34
48.50
28.70
83,50
97.75
YTSM
Both
89.70
98.35
64.35
92.08
80.01
94.69
29.00
31.86
45.00
29.10
84.50
98.00
Table I: Results of pre-training on How To 100M, AudioSet and Youtube8M. Best results are high- lighted in blue, second best results are highlighted in light blue. RW(VA); Gradient Re-weighting (Scale up Video-Audio), RW(VT) Gradient we-weighting (Scale up Video-Text), GR: Cross-
Gradient Realignment, CL: Gradient- based Curriculum Learning. (best view in color)
Hyperparameter
Video Action Cls
Video Retrieval
Audio Cls
Dataset
UCF101
HMDB51
YouCook2
MSRVTT
ESC50
Metric
Topl1
Top51
Toplf
Top5t
Rank]
R@ 101
Rankl
R@101
[opp1
Top5t
VATT 11
78.53
95.24
57.36
86.07
93.50
17.10
73.00
16.70
71.50
91.75
(to, Nhook) (- 0.3, 0.0)
79.10
96.16
56,41
86.06
40.00
26.01
56.00
23.90
72.50
94.25
CL
(to Naook) (- 0.2, 0.0)
77.75
94.94
56.87
87.18
53.00
22.21
57.00
20.80
70.75
94.75
(to, [[oox)] (-0.1, 0.0)
44.49
75,42
33.44
68.78
81.00
17.13
68.50
19.10
70.25
93.25
(7o: Naook) 00.0,0.0)
10.05
24.88
7.31
23.76
457.50
5.02
153.50
11.20
58.50
87.50
Table 2: Ablation on the choice of y in Gradient-based Curriculum Learning on Howtol00M, We linearly decay during training, Yoo "Sookk denote at the beginning and at the end of the training, respectively. Besl results are highlighted in blue. CL: Gradient-based Curriculum Learning.
Gradient-based Curriculum Learning Compared to Cross-Modality Gradient Realignment, Gradient-based Curriculum Learning inherents its performance gains in video-text retrieval task, yet enjoy an additional performance boost in audio classification (e.g. 1.40% gain ESC50 dataset in Table I and video classification (e.g. 1.94% gains UCF101 dataset in Table D). We also perform an ablation on the adaptive choice of y on Howto 100M dataset in Table We can see that curriculumly evolving 7 (i.e. 7500k) (-0.3, 0.0)) yields the best overall results, while maintaining a fixed (i.e. (70, 7500x) ) = (0.0, 0.0)) yields the worst results, this verify the effectiveness of curriculumly removing mis-aligned samples during training.
Cross- Cross-Modality Gradient Realignment + Gradient-based Curriculum Learning We further verify the effecttivness with the combination of both techniques. Specifically, given video-audio pair and video-text pair, (a) if cos(gva: ( S 7 we drop this sample triplet; (b) if < cos(gna- 9ve) < O, we apply gradient projection in Algorithm (c) if cos(gva; 9ur) 2 O, we retain the original gradients. We found they are complimentary to each other and the combination of both lead to better performance in the majority of downstream tasks.
4.5 Scaling Pre-training to YouTubeSM
As shown in Table I adding YouTubeSM dataset brings the video classification and video-text retrieval performance to new level (e.g. 88.28% Topl on UCF101, 56.0 rank on MSRVTT), yet it set back the performance in audio classification task, we conjecture this is largely due to the noisy misaligned samples introduced by Youtube8M dataset.
However, by leveraging our proposed techniques, we further yields a new state-of-the-art performance in the modallty-agnostte setting, without any modification in architecture, striking 89, 70% Topl on
9

---

UCF101, 92.15% Top5 on HMDB51, 80.01% % Top1 on Kinetics400 and 85.00% Topl on ESC50 dataset, demonstrating the effectiveness of our method when scaling to noisy Youtub8M dataset.
4.6
Feature Visualization
To better illustrate why gradient harmonization techniques works, in Figure 5 we follow the vi- sualization used in VATT[I], which take YouCook2 Video-Text Retrieval dataset, showing the output space visualization of video and text, respectively. We can see the initial video and text feature representation of VATT baseline are mixed together (which aligned with the findings in [I],, however, after using gradient realignment, curriculum learning or applying both, the video and text representations become disentangled, this indicates our gradient harmonization techniques adds to cross-modality diversity, which makes video/text more discriminative from each other, making the model more discriminative and better in generalization.
Dutp
Space
Output Space
Outout Space
Ourput Space
Mdcco
Addco
Video
Video
Tett
Text
Text
25NE2
24
66
1-5ME-1
I-SNE-1
6-NE-1
6-NE-1
(a) VATT Baseline
(b) w/ Gradient Realignment(c) w/ Curriculum Learning
(d) w/ Both Techniques
Figure 5: t-SNE visualization of the VATT output space on YouCook2 dataset.
4.7
Extending Gradient Realignment to Modality-Specific VATT
So far, our techniques were built upon the modality-agnostie setting of VATT with shared backbone. We also try to apply the same technique to the modality-specific setting, given there is still a shared video head where we can harmonization the gradients. In Table we observe that applying Gradient Realignment (GR) to modality-specific setting has some gain over uni-modality tasks (e.g. Audio Classfication), yet set back on cross- moos-tt tasks (e.g. hypothesize this is due the lack of modaliyy signals: since there is no longer shared backbone across modalities, the only shared video head would provide bias signala towards the video modality, for cross-modality alignment. We leave the improvement of modality-specific VATT for future work.
Retrieval), We
Methods
Video Action Cls
Text- Video Retrieval
Audio Cls
Dataset
UCF101
HMDB51
YouCook2
MSRYTT
ESC50
Metric
Toplt
TopSt
Toplf
Top5t
|Rank]
R@101
Rank]
R 10
Toplt
Top51
Modality-Speciin VATT [1]
82.36
97.35
58.51
89.32
19.00
38.84
27.00
32.38
74.72
93.75
GR
82.06
97.35
61.45
90.45
22.00
36.82
40.00
27.32
76.50
93.00
Table 3: Extended experiments on applying Gradient Realignment to the modality-specific VATT.
5 Conclusion
In this paper, take deep dive into the common CMA assumption used in multimodal pre- training. Using the gradient directions between pairwise losses as the surrogate indicator, we observe ubiquitous gradient conflicts even in relatively well-aligned narrative datasets. We then propose two plug-and-play techniques to harmonize the gradients during training. They are demonstrated to substantially enhance VATT pre-training across a number of downstream tasks, and further scale it up to pre-training with even more complicated and poorly aligned data. Our findings underline the (often overlooked) importance to carefully revisit the basic assumption and training dynamics of multimodal learning, besides advancing the model architectures. However, our crods-mo gradient harmonization technique still has room to generalize better to the modality-specifft setting. Additionally, we did not fully explore other regularization techniques in multi-tasking learning literature, such as 29). The potential negative societal impacts include the malicious use of multimodal models, such as unintended use of its downstream visual/audio recognition capability.
Acknowledgments and Disclosure of Funding
We give special thanks to Tianqi Liu and Jialu Liu for the constructive feedback and discussions. Wang is supported in part by US Army Research Office Young Investigator Award W91 INF2010240.
10

---

References
[1] Sami Abu-EI-Haija, Nisarg Kothari, Joonseok Lcc, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark, arXiv preprint 1609.086600 2016.
[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zet,lemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 190.v 2019.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[4] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[5] lean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovie, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. NeurIPS, 2(6):7, 2020.
[6] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, pages 609-617, 2017.
[7] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435-451, 2018.
[8] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. arXiv preprint arXiv: 1807.00230, 201 8.
[9] Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crosselr: Cross-modal contrastive learning for multi-modal video representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1450-1459, 2021.
[10] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12475-12486, 2021.
[11] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104. 2021.
[12] Antoine Miech, Dimitri Zhukov, Dean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol 00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
[13] Antoine Miech, lean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2020.
pages
[14] Jue Wang, Haofan Wang, Jincan Deng, Weijia Wu, and Debing Zhang. Efficientclip: Efficient cross-modal pre-training by ensemble confident learning and language modeling. arXiv preprint arXiv:2109.04699 2021.
[15] Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional vidcos. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[16] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang. and Longbo Huang. Modality competition: What makes joint training of multi-modal network fail in deep preprint arXiv 2022.
arXiv
11

---

[17] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint (XXi::0001.06782, 2020.
[18] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv:1911. 126677 2019.
[19] Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang. Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, et al. Broaden your views for supe--sue video learning. arXiv preprint 2033:2203..6555 2021.
[20] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06355, 2020.
[21] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.044 04297, 2020.
[22] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint a01iv:2010.1 1929, 2020.
[24] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and labeled dataset for audio events. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEEE, 2017.
[25] Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint 2017.
[26] Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. arXiv preprint arXiv:2109.0466 2021.
[27] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803. PMLR, 2018.
[28] Ozan Sener and Vladlen Koltun. Multi-task learning as multi- optimization. arXiv preprint arXiv: 1810.04650, 2018
[29] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018
[30] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874, 2020.
[31] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695-12705, 2020.
[32] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[33] Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization. arXiv preprint a7Xiv:2002.10657, 2020.
12

---

[34]
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama. Co-teaching: robust training of deep neural networks with extremely noisy labels. In Proceedings of the 32nd International Conference on Neural Information P Processing Systems, pages 8536-8546, 8546, 2018.
[35] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2304-2313. PMLR, 2018.
[36] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: From clean label de- tection to noisy label self-correction. In International Conference on Learning Representations, 2021.
[37] Angela H Jiang. Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary Lipton, al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.0077 2019.
[38] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hi- erarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200, 2020.
[39] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint 11..i::107.1499:
2021.
[40] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, et al. Avlnet: Learning audio visual language representations from instructional videos. arXiv preprint arXiv:2006.000 0999,, 2020.
[41] Hu Xu, Gargi Ghosh, Po-Yo Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feicht- enhofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic video-language model pre-training for video understanding. arXiv preprint arXiv:2105.099: 099966 2021.
[42] Valerii Likhosherstov, Anurag Amab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers images, videos and audio. arXiv preprint 1rXiv:2111. 12993, 2021.
[43] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv:1212.0402, 2012.
[44] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pages 2556-2563. IEEE, 2011.
[45] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1705.07000000 069500 2017.
[46] Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015-1018, 2015.
[47] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9288-5296. 2016.
[48] Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. short note on the kinetics-700.. 700-2020 human action dataset. arXiv preprint 2010. 10864. 2020.
13


Reviews:
[{"review_id": "fS7XDYMlCtx", "reviewer": "Reviewer_CREH", "summary": "This paper proposes two regularization strategies to harmonize the multimodal pre-training. The motivation is that cross-modality alignment (CMA) is problematic when the pre-training data is noisy and misaligned. The contribution of this work is: (1) introducing the cosine similarity between video-audio and video-text gradients as a surrogate indicator on how noisy the alignment of a sample triplet (video-audio-text) is; (2) introducing cross-modality gradient realignment to re-project the gradient direction if they have negative cosine similarity; and (3) using cosine similarity to prioritize training with well-aligned samples. The effectiveness of the proposed strategies are comprehensively evaluated.\n", "questions": "1. Line 51: “Figure ??”. This should be a typo.\n2. Table 1: when the pre-training datasets are HT100M and AudioSet, the performance of VATT on Text-Video retrieval is quite low. For example, this paper reports R@10=29 of VATT on YouCook2. However, the original VATT paper reports R@10=40.6 (the higher the better) for YouCook2. \n", "limitations": "Not applicable.\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "---\n\nOriginality: misalignment is a well-known problem in multimodal pre-training data, but is also tricky to solve. The two regularization strategies proposed in this work are novel in the multimodal pre-training community.\n\n---\n\nQuality: \n\nStrengths: using cosine similarity of video-audio and video-text gradients as the indicator of misalignment is reasonable. The effectiveness of the proposed regularizations are evaluated on various downstream tasks. The work is complete.\n\nWeakness: (1) cosine similarity can only be applied to the scenario with at least triplet. When the pre-training data only contains video-text pairs or only contains video-audio pairs, this surrogate indicator does not work. (2) According to Table 1, the majority of the improvement in video action classification tasks comes from curriculum learning (CL). Combining with gradient realignment (GR) even hurts the performance of CL-only. \n\n---\n\nClarity: this paper is well written and well organized. It is easy to follow. Since this work is built upon VATT and provides enough training details, it is easy for readers to reproduce the result.\n\n---\n\nSignificance: misalignment is an important problem in contrastive-based representation learning. Although it is novel to use cosine similarity of two gradients to measure the misalignment, the performance improvement is marginal.\n\n---", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8DWahYTDYO", "reviewer": "Reviewer_LhYL", "summary": "This paper challenges the cross-modality alignment (CMA) assumption in multi-modal pre-training. The authors found that CMA can be weak and noisy because of the often poor cross-modality semantic alignment. They proposed two techniques, i.e., gradient realignment and gradient-based curriculum learning. Experiments on conducted with VATT, on some of the large-scale pre-training datasets. \n\n", "questions": "Please see the weakness part ", "limitations": "No, the current version claimed it has no negative social impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tThe authors studied a novel and important problem. CMA is traditionally assumed by default in multi-modal learning (and kinda why most prior works was restricted to instructional video data). This paper takes an important step toward the wild. \n\n-\tThe observation of poor CMA is convincing (Figure 1 illustration, and Figure 3 quantification by gradients). This problem of misalignment has not been studied in the existing works on self-supervised multimodal learning, and the insight to analyze the problem from the perspective of the gradient is new to my best knowledge.\n\n-\tThe two proposed techniques, albeit simple, seem effective and low overhead to implement\n\n-\tThe authors spent lots of effort on experiments. The benchmarking in Tables 1 and 2 is comprehensive. \n\n-\tThe authors are able to scale up VATT pre-training to the more complicated non-narrative Youtube8M dataset, which further yields new state-of-the-art performance in a modality-agnostic setting. This is impressive and validates the benefits of their proposed techniques. \n\nWeaknesses: \n-\tThe method has moderate technical novelty since it adapts several known wisdoms from multi-task learning. \n\n-\tThis paper in general lacks an ablation study. I can totally understand the (re-)training overhead is huge for such models, but at least some rationale or comparison on choosing their methods’ ad-hoc hyperparameters (there are many) could be provided? \n\n-\tHow might the proposed technique scale beyond VATT? E.g., to last V-L models such as DALL-E and CLIP? Do they also suffer from similar problems, and can your methods be helpful? (I’m not asking authors to re-train CLIP or alike, just idea sketching).\n\n-\tThe writing quality of this paper is barely readable and needs proofreading.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1gUMYTJYRv8", "reviewer": "Reviewer_VTo5", "summary": "This paper studies the problem of contrastive multimodal pre-training (assuming triplet training data <video, text, audio>). It starts with uncovering the misalignment phenomenon existing among temporally-associated video clips, subtitles, and audio tracks, which make them falsely considered positive training pairs. It then quantitatively measures the gradient conflicts resulted from this misalignment. Ideally, given a modality-agnostic backbone, gradient updates come from different loss terms on the same positive triplet should reinforce each other, reflected by non-negative cosine similarity. However, this is not the case for existing methods. To this end, the paper proposes two techniques, namely Gradient Realignment (forcing gradients updates from different losses to have non-negative cosine similarity) and Gradient-based Curriculum Learning (filter out samples with low gradient similarity), to alleviate the gradient conflicting issue.", "questions": "Typos:\n\ni) Line 51, the figure reference is unknown.\n\nii) The arrow in Colume 6 Row 3 of Tab. 1 is missing.", "limitations": "Appear to be sufficient.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed idea is sensible and interesting. Overall, the experimental evaluation is comprehensive and supports the effectiveness of the proposed method.\n\nThe motivation of the idea, however, could be more upfront. As the paper suggests in the rather later part of the paper (line 255), the proposed method is only studied in the modality-agnostic single-backbone setting. However, without this knowledge, understanding the motivation of the paper (gradient conflicts) is cumbersome. Please consider stating the problem setting clearly as early in the paper as possible (on or before line 61).\n\nMoreover, is the proposed method only applicable for modality-agnostic architecture with triplet training samples? How about the modality-specific setting, where the gradients $g_{va}$ and $g_{vt}$ conflict on the video encoder only, but not audio encoder and text encoder (as it is the pivot)? Or in the bi-modal modality-agnostic setting (e.g., image and text [r1]), should the gradients backprop from the image feature and text feature be harmonized? It would be great if the authors could share insights/analyses on whether the proposed method could tackle these scenarios.\n\nThe final question is related to the previous one. As $g_{va}$ consists of gradients from both the audio part and the video part, $g_{vt}$ consists of gradients from both the video part and the text part, should this \"micro\" harmonization between all four of them be considered in addition to the \"macro\" harmonization showcased in the paper? Any insights are appreciated.\n\n[r1] You et al., MS-CLIP: Towards Modality-Shared Contrastive Language-Image Pre-training. ECCV 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept