LGDN: Language-Guided Denoising Network for Video-Language Modeling
Authors: Haoyu Lu
Mingyu Ding
Nanyi Fei
Yuqi Huo
Zhiwu Lu
Venue: NeurIPS 2022 Accept (2022)
Keywords: video-language modeling
video-text retrieval
language supervision
cross-modal alignment

Abstract: Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically correlated, and focus on video-language modeling at video level. However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single video-level description; (2) A raw video typically has noisy/meaningless information (e.g., scenery shot, transition or teaser). Although a number of recent works deploy attention mechanism to alleviate this problem, the irrelevant/noisy information still makes it very difficult to address. To overcome such challenge, we thus propose an efficient and effective model, termed Language-Guided Denoising Network (LGDN), for video-language modeling. Different from most existing methods that utilize all extracted video frames, LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2--4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins. We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-language work.

LGDN: Language-Guided Denoising Network for Video-Language Modeling
Haoyu Lu Mingyu Ding Nanyi Fei Yuqi Huo Zhiwu Lu 12.
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods The University of Hong Kong, Pokfulam, Hong Kong JJ Corporation, Beijing, China [1hy1998, wuzhiuu)&ruc..oo .edu.cn.nccc
Abstract
Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically correlated, and focus on video-language modeling at video level. However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single video- level description: (2) A raw video typically has noisy/meaningless information (e.g,, scenery shot, transition or teaser). Although a number of recent works deploy attention mechanism alleviate this problem, the irrelevant/noisy information still makes it very difficult to address. To overcome such challenge, we thus propose an efficient and effective model, termed I. Language-Guidedd Denoising Network (L.GNN), for video-language modeling. Different from most existing methods that utilize all extracted video frames, L.GDN dynamically filters out the misaligned redundant frames under the language supervision and obtains only 2-4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins. We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future rideo-languaee work.
Introduction
Humans are exposed to the world through a variety of sensory organs, such as eyes, ears, and the sense of touch. In the past few years, multi-modal data (e.g., text or video) has grown and accumulated rapidly on the Internet, which brings the increasing demands for video-language understanding. As one of the fundamental topics, video-language modeling still challenging due to the heterogeneity of the video-text data. More notably, the video-text data is typically noisy (e.g.. misaligned or semi-relevant, as shown in Figure 1), leading to intractable lano-lan modeling.
The dominant paradigm [8, 30, 13, 14, 45] for video- language modeling is to first extract language features and dense video features via off-the-shelf language and vision models (e.g., BERT [7], 3D CNN [48]]. J), and then model the cross-modal representation by defining the objective function (e.g.. triplet loss [17]) within a joint semantic space. Although achieving great success, these methods typically densely sample frames from full sequence of raw video to obtain richer representation and thus cost excessive computation. Since the heavy computation makes it challenging to train the whole network end-to-end, they often achieve sub-optimal performance in video-language modeling. Recently, ClipBERT [25] proposes sparse sampling strategy to tackle this drawback. Concretely,
'The corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Salient
Frame
Masliiged Rohndaant Frame
cartoon womun cries bench while woman in blue appears
An example of video-text pair where the raw video contains misaligned and redundant video frames given the text description. Instead of aggregating all video frames for token-level alignment, we obtain only 2-4 salient frames per video by filtering out the misaligned ones. We find that utilizing 2-4 salient frames is much more effective while enjoying faster speed.
Figure I:
ClipBERT first samples video frames sparsely (8-16 frames per video), and then models the cross- modal alignment at frame-level. This sparse sampling paradigm enables end-to-end training, leading to much better performance. Nevertheless, token-level cross-modal interaction, which has achieved great success in image-text modeling [21, 26], is still not well explored for video-language modeling due to the heavy resource computation (even with 8-16 frames per video). Moreover, both the dominant paradigm and ClipBERT's s sparse sampling paradigm assume that video frames and the text description (w.r.t. a video-text pair) are semantically correlated, which is often invalid in practice.
The correlation hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is hard to cover all frames with a single video-level description; (2) A raw video often has noisy or meaningless information (e.g., scenery shot, transition or teaser). For the dominant paradigm which utilizes densely-sampled frames, though often with self-attention mechanism [43], the irrelevant/noisy
information makes hard learn high-quality video-language representation. For the sparse sampling paradigm used in ClipBERT that models the cross-modal alignment at frame-level, the misaligned frame-text pairs are wrongly forced to become closer, which inevitably leads inaccurate cross- modal alignment. Overall, due to this noise issue (see Figure 1), video-languag, modeling is still challenging. Note that humans also encounter such problem in reality, but seem to be born with the ability to resist noise. That is, everyone can quickly scan through the entire video, easily ignore the noisy frames and focus on the salient ones given the text.
Motivated by this human ability, we propose a Language-Guide Guided Denoising Network termed LGDN dynamically filter out irrelevant or redundant information under the language supervision for better video-language modeling. Concretely, we devise Salient Frame Proposal (SFP) mechanism which adopts four strategies estimate frame-level relevance scores under the language supervision and proposes/selects only salient frames (per video) for precisely video-language modeling. Although the frame embeddings and text embeddings can be (roughly) aligned by introducing Momentum Video-Level Contrastive Learning (MVCL) module, it is vital to precisely establish frame-text alignment for proposing salient frames. Therefore. based on multiple instance learning (MIL), we propose a Momentum Frame-Level Multiple Salient-instance learning (MSL) -Contrastive Learning (MFCL) module for video-language modeling at frame-level. Finally, with our SFP mechanism, we propose Language- Guided Salient Frame Matching (LSFM) module for fine ine-grained ali alignment, which adopts a token-aware cross-attention Transformer for cross-modal token-level alignment.
Lunddd
Our main contributions are as follows: (1) We devise a salient frame proposal mechanism that can dynamically filter out irrelevant information under the language supervision, meanwhile maintaining salient information.
(2) We propose an end-to-end framework termed LGDN for video-language modeling with cross-modal interaction three levels: language-guide
salient frame matching at token-level, momentum frame-level MSL-contrastive learning, and momentum video-level contrastive learning. (3) We evaluate our LGDN on five public datasets and find that our LGGNN outperforms the latest competitors by large margins. We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-lane video-language work.
2 Related Work
Video- Language Modeling, Video-language modeling, fundamental research topic that is bene- ficial for search engine and video recommendation, has attracted a lot of attention in recent years with the rapid growth of web videos. Previous works have made great efforts to model richer rep-

---

resentations for video and text modalities and then align the features of the two modalities by the objective function (e.g., triplet loss). One common representative approach [5, 20] is adopt Graph Convolution Network (GCN) to extract richer information for video-text retrieval. Another representative approach [30, 13, 14, 52, 29] to exploit extra experts (e.g., object, motion, speech) for Video-language modeling. Recently, ClipBERT [25] proposes sparse sampling strategy that enables end-to-end training, thus achieving higher performance. Moreover, Frozen in Time [2] also follows sparse sampling paradigm, and proposes an end-to-end trainable model that is designed to take advantage both large-scale image and video captioning datasets. However, as illustrated video typically has noisy/meaningles information, and thus the presence of
in Figure 1, misaligned frames is inevitable during video- video-language modeling. Note that most existing methods assume that the video frames and paired text are semantically correlated, without considering the noise phenomenon. Although a self-attention mechanism has been widely applied, the misaligned frames still harm the modsl alignment. In this work, we thus propose salient frame proposal mechanism to effectively (and directly) address this problem.
Cross-Modal Alignment Objective Functions Most previous methods adopt triplet loss major objective function for video-language modeling. CGMSCD [14] points out that the triplet loss sometimes leads a wrong learning direction and thus devises adaptive margin triplet loss for representation learning. More recent works [41, 12, 19] propose to apply the InfoNCE contrastive loss [47, 38, enhance representation learning. Particularly, BriVI. [18], ALBEF [26] and COTS [32] introduce momentum mechanism [15] to maintain more negative samples for image-text contrastive learning. Following these state-of-the-ar models, we propose momentum video-level contrastive learning for video-text global alignment in this paper. Note that MIL-NCE [35] enhances the InfoNCE loss with multiple-instane
learning (MIL) to cope with the misaligned narration descriptions in HowTo TowTo100M [36]. In this work, we thus propose momentum frame-level MSL- contrastive learning to assist in addressing the misaligned frame problem.
Methodology
Figure 2 gives a brief overview of our LGDN framework for video-language modeling, which is composed of four main components: 1) language and vision representation extractors; 2) momen- tum video-level contrastive learning; 3) momentum frame-level MSL-contrastive learning, and 4) Lunguage-guides salient frame matching. In the following, we will describe each component in detail.
3.1 Feature Representation
Vision Representation. Given an input video V as sequence of frames { where N is the length of the video, we utilize 2-D vision Transformer (e.g.. ViT) as our vision backbone to extract frame-level features E {E1, E2, .... Ex). Each frame E, of video V can be represented as E, = (ects: e1; ...; - E RD. where eals denotes the [CLS] token, k₂ denotes the patch sequence length, and D. denotes the dimension of the patch embeddings. We utilize a fully-connected layer to project the [CLS] token into the frame embedding fr. We then deploy a temporal module T (e.g., a Transformer layer) to aggregate the frame embeddings to obtain the final video embedding:
T(f 1 (V),
(1)
where f denotes the entire vision (video) encoder.
Language Representation. Given an input text L, we utilize BERT-Base as our language backbone to extract text feature L, which can be represented as L = [las;; l;;..; - E R xD. where late is the [CLS] token, k is the token sequence length, and D; is the dimension of the token embeddings. We deploy a fully-connected layer to project the [CLS] token into the text embedding f' (L,),, where f is the language encoder.
3.2 Momentum Video-Level Contrastive Learning (MVCL) Module
Note that our LGDN is designed to filter out the unmatched/reunndat frames for better token-level alignment, without leveraging the temporal information of the videos explicitly. Therefore, we firstly introduce a Momentum Video-Level Contrastive Learning (MVCL) module to address this problem.
3

---

Language Guided Salient Frame Matching Learning
Momcnttu: Video-Levd
Contrastive Learning
Conteseive
Multi-modal Fusion Transformer
Tamporal Aggresation
Momentum Frame-Leva Ass--Cotraetive Learning
INMl-Contrastive Learning
Coninilive
Salient Framn Pruposal Mechanisen
Loss
Posttive
Salient Frame Proposal Meclunism
Momentum Video-Len
Conrrstivve Learning
Remnking
Salient
Vision Encoder
Lungunge Encoder
Eniinating Relecnne Soare
mus jompp onsto lodge of buildigg.
...
Tosal Frames
Normal Frame Fentures
Salient Finmte Footures
Video Festures
Festures
[CLS] Tokon Festures
(a)
(b)
Figure 2:
(a) A schematic illustration of the proposed LGDN framework. (b) Details of each
component of our LGDN. denotes that calculation performed only at the [CLS] token level.
The MVCL module utilizes a temporal module (e.g., Transformer block) to aggregate the frame embeddings to obtain the video embedding. Contrastive learning is then applied for holistic video-text alignment. However, video data takes up large GPU memory and the mini-batch size tends to be small with strict resource, which brings harm to contrastive learning. Inspired by MoCo [15], we introduce the momentum mechanism to maintain massive negative samples in memory bank for contrastive learning. Concretely, We firstly maintain video memory bank M (4/) and text memory bank M = ) to store video/text features, where Nm denotes the memory bank size and / a/ denotes the j-th stored video/text feature vector. Let f (with parameters 0²) and f (with parameters 8²') denote vision encoder and vision momentum encoder, respectively. Similarly, let f' (with parameters 0') and j (with parameters 0') denote language encoder and language momentum encoder. respectively. The parameters of momentum encoders are updated by:
g- = m + - m) 0" 0' = m 0² + +(1 m) 8¹
where m is the momentum coefficient hyper-paramettr
The loss function is thus constructed follow: for each video 1/2 in mini-batch B, define the video-to-text contrastive loss between its paired text Lt and all negative samples in the text memory bank M', resulting in an InfoNCE loss (with 7 being the temperature hyper-parameteer
exp(cos(f; f1)/T)
LVXT
Σ log exp(cos(f), + EgeEM exp(cos(ff) G/)/))
(V.L)))
where P! (1,), and the similarity of two features is measured by the cosine similarity. Similarly, given each text description L; in mini-batch B, we define the text-to-video contrastive loss as:
exp(cos(f). (/)/T)
Lryv
Σ
log
(4)
B
f/)) (x)(cos(((,f")//)))
exp(cos(f),
(tarrr)+Zzeeea Eagem exp(cos(f") G7)/T)
(V,
where P.- = f'VV). Finally, the objective function for MVCL is defined as follows:
LMVEL Lvzr Lrvv.
(5)

---

Table 1: Four strategies for estimating relevance scores.
SimDot
Momentum
CrossMom
Collaborative
ROl) E
RGIi E2
R(j)) -
f2 G + E2
RGJi (C,
ft.)) (C) R
3.3
Salient Frame Proposal (SFP) Mechanism
As shown in Figure 1, video-text data inevitably contains misaligned frame-text pairs. Although an attention mechanism has been applied in Eq. (1), the irrelevant and noisy information would still mislead the cross- mrosl- alignment our model. To alleviate this problem, we thus propose a Salient Frame Proposal (SFP) mechanism for video-language modeling.
The core idea of our SFP mechanism is dynamically filter out misaligned or redundant frames and maintain only few important frames to represent the video well, which are called as salient frames. Formally, for each video-text pair, we first identify the relevance score R(jli) between the text L; and the j-th frame E,g of the video V. Further, we perform language-guides denoising to retain only top-Nsatiien salient frames by filtering out the unmatched/redundatl frames from each video.
Since only video-level annotations are provided, we need to estimate the relevance scores R automat- ically. As shown in Table we introduce four strategies for estimating relevance scores. (1) SimDot prediction relies on the output of two separate encoders (i.e., frame encoder f and language encoder f') to model the relevance score R(jli) by computing the dot product of the frame embedding (L.). However, since video-text data is noisy, only
(E and the text embedding utilizing single-modality encoders may result in incorrect salient frames. (2) Momentum prediction improves SimDot prediction by introducing the supervision of momentum encoders (i.e., momentum frame encoder f and momentum language encoder ft), where the momentum frame embedding f = (E,,), and the momentum text embedding f. - f' (3) CrossMom prediction consid- ers the frame-text alignment that is directly built on the interaction between one modality encoder and another modality's 's momentum encoder. (4) Collaborative prediction combines Momentum prediction and CrossMon prediction for better performance.
Although the frame embeddings and text embeddings can be (roughly) aligned through applying video-text contrastive learning in Sec. 3.2, it is vital to precisely establish frame-text alignment for proposing/selecting salient frames. To this end, we introduce the MFCL. module below.
3.4 Momentum Frame-Level MSL-Contrastive Learning (MFCL) Module
To dynamically filter out the unmatched/redundat frames, we propose to adopt frame-level con- trastive learning to directly measure the relevance scores R between video frames and paired text. However, video data often contains misaligned frame-text pairs. Simply applying standard NCE-based contrastive learning would force the misaligned frame-text pairs to be pulled closer, which inevitably has negative effect on learning high-quality frame-text representation. Inspired by MIL-NCE [35], we thus propose a Momentum Frame-Level Multiple Salient-instance Learning (MSL) Contrastive Learning (MFCL. module to assist in alleviating the noise problem. The core idea is to use the salient frames filtered by the SFP Mechanism in each video to form a set of positive candidate pairs, instead of considering each positive pair independently. In this work, we suppose that MFCL and SFP have mutual interdependence so that they can bring boost to each other during training.
Similar to MVCL, we additionally maintain a frame-level memory bank M = (G 1 to store frame features, where N,, is the memory bank size, N is the number of sampled frames per video, and qh, is a stored frame feature vector. Given each text description L, in mini-batch B, we select salient frames filtered by the SFP Mechanism in the paired video VÚ to form set of positive candidate (frame-text) pairs S; and all frame samples in M to form the negative ones. We then define the text-to-frame contrastive loss
Enger efr exp(cos(f), fT)
1
Σ log (EB Enzer; exp(cos(f), + EM exp(cos (f). G;)/T)'
(6)
|B
where S (
1-W is the positive frame set of the video V4, N the frame sequence length of the video, f = (L,), and t.. (f (E
5

---

Similarly, given each positive frame set S,, we define the frame-to-text contrastive loss as: Σt, EFe exp(cos((E) /T)
1
LEZT
log Etzer; exp(cos(ff) f.J. + En Egjeeme exp(cos af.)/T))
B
(S.
where = f and f ( (E: (text memory bank M = l プー defined in Sec. 3.2). As a result, by combining the text-to-frame and frame-to-text contrastive losses, the objective function for MFCL is given by:
Nm is
CMPCL = LEZT + Lr2E.
3.5 Language-Guidedd Salient Frame Matching (LSFM) Module
After obtaining language-guided salient frames, utilize multi-modal cross-attention fusion Transformer (see Figure 2) to capture token-level semantic alignment between visual patches and words for better performance (see the design details of this Transformer in the supp. material). Further, we take the [CLS] token embedding outputted by the multi-modal fusion Transformer as the joint representation of a frame-text pair (V,, L.), and deploy fully-connected layer to predict the matched probability, which is similar to the sentence pair classification task in BERT's pre-training phase. The matching loss is defined as:
LISPM
log P(ys. Ea,j L.),
where E, denotes j-th frame feature of video V, L, denotes text feature, Daalient is the set of salient frame-text pairs obtained by applying the SFP mechanism to the mini-batch, and yi the ground-truth matching label (0 1) of the frame-text pair (E,,,, L;). During inference, we use mean pooling layer to aggregate all salient frame scores as the video-level prediction score.
Finally, by combining all the proposed modules for rideo-language modeling at three levels, we train our LGDN model via minimizing the total objective function:
LLGNN
LMVCL LMFCL + LLSEM-
(10)
4 Experiments
4.1 Datasets and Settings
Pre-Training Datasets. Due to the restricted computing resources, we follow COTS [32] to pre-train our LGDN on the pure image-text datasets. Our pre-training datasets consists of Conceptual Captions [42], SBU [39], VG [23] and MSCOCO [28], which contains 5.2 million image-text pairs. We additionally apply CC12M [3] (about 2 million URLs now invalid) for better performance, which accumulates 15.2 million image-text pairs in total.
Downstream Datasets. We evaluate our proposed LGDN on four public vidco-text retrieval datasets: MSR- [50]. MSVD [4], DiDeMo [16], and VATEX [46]. To further demonstrate the general public video-question answering dataset: MSRVTT-QA [49]. We present the details of these downstream datasets as well as the evaluation metrics for downstream tasks in the supp. material.
applicability of our LGDN, we also carry out experiments
Implementation Details. Following previous work [25], we sample N 16 frames per video:
each video is equally split into 16 segments and one frame is randomly sampled from each segment. We empirically set the initial learning rate to le-5 and adopt AdamW [31] with a weight decay of 0.02 for 5 epochs. In the warm-up stage (first epoch), the model is trained to optimize Eq. (10) without applying SFP mechanism. We also set the other
uniformly as: salient 0.99,
frame numbers Nsatient 2, bini-batch size [ 24, momentum Hyper-parameee temperature т = 0.07, and queue size Nm = 9, 600. We adopt pre-trained BERT-Base as language encoder and ViT-Base [9] vision encoder. More details are given in the supp. material.
Evaluation Metrics. We adopt two widely- used metrics in cross cross-modal retrieval: Recall at K (R@K, K= 1, 5, 10), and Median Rank (MdR) / Mean Rank (MnR). R@K K means the percentage of correct matching in the nearest points, and MdR MnR measures the median mean rank of target items in the retrieved ranking list. We also report two additional metrics named 'R @ Sum' and @@@ean in our ablation study, which sums/averages all recall metrics for overall evaluation. Following ClipBERT [25], we also report accuracy (Acc) in video-question answering task.

---

Specilip
MusilggMan
Spoottp
Nettons
Mattteee
del
104
33
23
51
Mastient
Nastient
(6)
(b)
(4)
Figure 3: (a-b) Effect of value change of N satient- The speedup (green line) is computed w.r.t. the slowest case sationt 16 (w/o SFP). We find that utilizing 2 salient frames is much more
effective while enjoying faster speed. (c) Effect of value change of N on the MSR-VTT Ik-A test set.
Table 2: Ablation study for our full LGDN model. Retrieval results are reported on the MSR-VTT Ik-A test set. Local: only leken-level alignment during inference. Global: only global alignment during inference. Ensemble: ensemble of global and local (token-level) alignment.
Text-to- Video Retrieval
Video-to-Texe Retrieval R@S R@ 10
Method
Inference
R@SUM
R@1
R@5
R@ 10
MdR
R@I
MdR
CISPM (w/o SFP)
Local
31.4
59.8
70.3
4.0
34.9
61.9
72.9
3.0
331.2
LisP (w/o SFP) + +MMELL (w/o MSL.)
Local
32.2
58.6
70.0
3.0
34.7
61.7
73.3
3.0
330.5
CISPM (wf SFPP -MFFLL
Local
33.2
60.2
71.0
3.0
34.7
73.6
3.0
335,2
CISIM (w/o SFP) -MMUCL LMVEL
Local
33.0
60.4
71.2
3.0
35.6
62.2
73.7
3.0
336.1
CISPM + +SRCLL LAEVLL
Local
35.3
65.0
75.3
3.0
36.3
65.0
76.0
3.0
352,9
CISPM LMRCL LAIELL
Global
32,5
60.4
71.7
3,0
32.1
61.8
72.2
3.0
330,7
LISPM + CAREE. + Lueee.
Ensemble
38.9
65.7
76.5
2.0
37.9
65.4
76.0
2.0
360.4
4.2 Ablation Study
In this subsection, we conduct comprehensive ablation study to investigate the contributions of different components of our full model. If not specifically indicated, we set N 16 for global alignment and Naalient = for token-level alignment as the default setting.
Effect of Value Change of Naatient and N. A common perspective for hideo/video-langueee understanding that more frames per video bring better performance. We thus conduct experiments on the frame number used for token-level alignment in Figure 3(a-b). We sample N = 16 frames from each video and evaluate different variants that use satient E {1. 2, 3, 4, 166 frames. Note that when Nsatient 16, sampling by our SFP degrades to w/o SFP. It can be observed that utilizing only Nsatient (2,3, 4) salient frames filtered by our SFP significantly outperforms utilizing all 16 extracted frames meanwhile enjoying the faster speed (see the green lines). This suggests that our SFP mechanism not only selects correct salient frames but also alleviates the noise problem. To investigate the influence of value change of N on our LGDN, we evenly sample N E {2,3, 4, 8} frames per video and freeze N salient = {1, 2} salient frames. The results in Figure 3(c) indicate that more extracted frames per video are beneficial to the token-level alignment in our LGDN model, it provides larger candidate set for selecting salient frames. Mcanwhile, when N becomes larger (> 4), the performance tends to converge, further demonstrating the redundancy in the videos.
Contributions Fach Components. We further demonstrate the contributions of the three objective functions as well the salient frame proposal (SFP) mechanism used in our full LGDN model in Table 2. We start with the objective function LISFM (w/o SFP), which means only applying matching loss in token-level alignment without using the SFP mechanism. It can be observed that: (1) LMFCI. (and LMvcr.) combined with LISFM (w/o SFP) can bring improvements, suggesting that global alignment is beneficial to token-level alignment (during the training stage). (2) Simply applying the frame-level alignment may cause negative effect while combing with our MSL design brings better results. This demonstrates that design of LMFCL does help alleviate the noise problem. (3) When the SFP mechanism is added (see LLSFM (w/o SFP) + + + LMVL vs. LLSFM + CMFCL + LMVCL), the performance is significantly improved. which clearly shows the effectiveness of our proposed SFP mechanism. (4) For the same trained full LGDN model, combining the global and token-level alignment during inference can bring further improvements. Note that full LGDN still achieves the state-of-tte tfate-of-the-ar the-art on MSR-Y VTT even without considering global alignment during inference.
7

---

Table 3: Comparison to the state-of-the-arts for video-text retrieval on MSR-VTV Extra Expert: methods utilized expert features (e.g., object, motion and OCR features). # PT Pairs: the number of pre-training pairs. denotes that our LGDN is additionally pre-trained with CC12M [3].
Extra Expert
Text-to- Video Retrieval
Video-to Text Retrieval
Method
#PT Pairs
R@!
R@5
R 10
MdR
R@!
R@5
R@ 10
MdR
Full Split:
HGR [5]
9.2
26.2
36.5
24.0
15.0
36,7
48.8
11.0
CE [30]
10.0
29.0
41.2
16.0
15.6
40.9
55.2
CMGSD [14]
100M
32.0
17.2
43.6
T2VLAD [45]
100M
12.7
34.8
47.1
12.0
20.7
48.9
62.1
6.0
LGDN (ours)
5.2M
22.9
46.0
56.8
7.0
41.8
65.2
74.6
2,0
LGDN (ours)
15.2M
27.5
51.7
61.9
5.0
50.2
73.9
82.3
1.0
7k-1k Split:
HERO [27]
100M
16.8
43.4
57.7
U33VL.333
100M
CliBBRT [25]
5.6M
22.0
46.8
59.9
TACo [52]
100M
24.8
52.1
64.5
5.0
LGDN (ours)
5.2M
34.3
62.5
72.2
3.0
34.7
60.8
70.4
3.0
LGDN (ours)
15.2M
39.8
65.2
77.0
2.0
39.2
66.4
76.1
1k-A Split:
[13]
100M
26.6
57.1
69.6
27.0
57.5
69.7
Support Set [40]
/
100M
30.1
58.5
69.3
3.0
28.5
58.6
71.6
3.0
TACo [52]
100M
28.4
57.8
71.2
4,0
Frozen in Time [2]
5.5M
31.0
59.5
70.5
3.0
LGDN (ours)
5.2M
38.9
65.7
76.5
37.9
65.4
76.0
LGDN (ours)
15.2M
71.6
80.6
Table 4: Results on the VATEX set.
Table 5: Results on the MSVD set.
Method
R@
R@5
R@ 10
MdR
Method
R@
R@5
R@ 10
MdR
VSE [22]
28.0
64.3
76.9
VSE++ [10]
15.4
39.6
53.0
9.0
VSE++ [10]
33.7
70.1
81.0
2.0
Multi, Cues [37]
20.3
47.8
61.1
6.0
Dual [37]
31.1
67.4
78.9
3.0
CE [30]
19.8
49.0
63.8
6.0
HGR [S]
35.1
73.5
83.5
Support Set [40]
28.4
60.0
72.9
4.0
Support Set [40]
45.9
82.4
90.4
Frozen in Time [2]
33.7
64.7
76.3
3.0
LGDN (ours)
57.1
87.5
93.6
LGDN (ours)
39.7
70.2
79.8
2.0
L.DN (ours)
61.0
90.2
95.1
LGDN (ours)
43.2
73.3
82.4
2.0
Table 6: Results the DiDeMo test set. notes using temporal labels of captions.
de-
Table 7: Results MSRVTT-QA. denotes utilizing large-scale VideoQA datasets.
Method
R@:
R@5
R@10
MdR
Method
#PT Pairs
Acc
S2VT [44]
11.9
33.6
13.0
Heterogeneous Memory [1i]
33.0
FSE [53]
13.9
36.0
11.0
IICRN [24]
35.6
CE [30]
16.1
41.1
8.3
SSML [1]
100M
35.1
ClipBERT [25]"
20.4
48.0
60.8
ClipBERT [25]
5.6M
37.4
Frozen time [2]
34.6
65.0
74.7
Just Ask [S1]
69.0M
41.5
LGDN (ours)
44.1
71.9
82.3
2.0
LGDN (ours)
5.2M
42.4
L.DN (ours)
47.8
76.2
83.3
2.0
L.GDNT (ours)
15.2M
43.1
4.3
Comparison to the State-of-the-Aris
We first report the text-video retrieval results on MSR-VTT with three data partitions in Table 3. can be observed that: (1) our LGDN outperforms all previous works by large margins. Particularly, as compared with the most recent model Frozen in Time [2], our LGDN achieves an improvement of 7.9% (38.9% 31.0%) for Text-to- Video R @@1 on the MSR-VTT Ik-A test set. (2) Our LGDN also outperforms methods utilizing extra modalities (c.g., motion and audio) or those pre-trained on extremely-large video data (c.g., HowTol00M (3) When leveraging much larger pre-training (image-text) dataset, our LGDN (marked with ") achieves significant improvements.
To demonstrate the robustness of our model, we also evaluate it on VATEX, MSVD, and Didemo Tables 4-6, respectively. Due to limited space, only text-to-video retrieval is considered here. For VATEX (Table 4), our LGDN significantly outperforms the tate-ooftthearr method Support Set which is trained on an order of magnitude more data. Our LGDN still performs the best on MSVD (Table 5) and Didemo (Table 6). Particularly, in the Didemo dataset, each description is

---

Table 8: Further evaluation results by directly applying our SFP mechanism to Clip4CLIP [34] (re-implemented) for video-text retrieval on the MSR-VTT 1k-A test set. results of Clip4CLIP and Clip4CLIP+SFP.
denotes the ensemble
Text-to- Video Retrieval
Vidco-to- Retrieval
Method
R@1 1
R
R 10
MdR
MnR
R@1
R@5
R@ 10
MdR
MnR
Clip4CLIP [34]
44.9
71.8
81.7
2.0
14.2
46.2
73.9
84.3
2.0
10.8
SFP
45.3
73.0
83.4
2.0
13.4
47.6
75.5
85.3
SFP*
47.2
73.4
83.9
2.0
13.0
48.1
76.7
86.1
9.3
Table 9: Results by applying SFP to different sampling techniques on the MSR-VTT 1KA test set.
w/o SFP (R (R@SUM)
wISFP (R@SUM)
Sampling
4 frames
8 frames
16 frames
4 frames
8 frames
frames
Random Sampling
164.7 166.5
169.5
172.8
168.0
174.3
179.4
Dense Uniform
171.3
174.3
173.1
179.4
180.6
Sparse Sampling
168.0
171.9
173.7
179.1
180.3
181.1
Table 10: Ablation study for relevance score estimator. Retrieval results are reported on the MSR- VTT Ik-A test set. Random / SFP: N satient frames are sampled randomly/by our SFP mechanism. #
Frames: Naalient / for local / global alignment.
Text-to- Video Retrieval
Video-to-Tee R@s
Retrieval R@
Method
Strategy
Frames
R@SUM
R@1
R@S
R@ 10
MdR
R@!
10
MdR
All
Random Random
16/16
35.7
63.8
74.2
3.0
35.4
63.8
74.9
3.0
347.8
Random
2/16
34.1
62.1
73.4
3.0
34.3
61.6
74.0
3.0
339.5
SimDot
SFP
2/16
37.4
65.0
76.4
3.0
37.2
65.1
75.4
2.0
356.5
Momentum
SFP
2/16
38.1
65.8
76.4
2.0
37.9
65.4
75.9
2.0
359.5
CrossMom
SIP
2/16
38.4
65.4
76.5
2.0
37.9
65.3
76.2
2.0
359.7
Collaborative
SIP
2/16
38.9
65.7
76.5
2.0
37.9
65.4
76.0
2.0
360.4
annotated with localization information, in other words, annotations may only be aligned with the localized moments, thus causing the noise problem as many methods utilize all frames as the input. Recent works exploit temporal labels of captions to alleviate the noise problem and achieve higher performance. However, even without considering this, our LGDN still largely outperforms the most recent method Frozen [2], further demonstrating the effectiveness of our LGDN.
To show the general applicability of our LGDN, we evaluate our LGDN the VideoQA task in Table 7. Even without utilizing large-scale video datasets devoted to the VideoQA task, our LGDN outperforms all competitors, validating the effectiveness of our LGDN in VideoQA. In addition, to reveal the critical importance of solving the noise issue for video- video-language modeling, we directly apply the SFP mechanism the latest model CLIP4Clip [34] in Table We find that applying the SFP mechanism brings boost to Clip4CLIP. The ensemble mechanism further improves the results, indicating that the proposed SFP mechanism is complementary to the baseline.
4.4 Additional Results
Applying SFP to Different Frame Sampling Techniques. Note that our SFP mechanism must be combined with frame sampling technique since we adopt two-stage sampling strategy in this paper. Thus, we apply our SFP mechanism to three frame sampling techniques: Sparse Sampling, Random Sampling, and Dense Uniform (equally interval sampling). The obtained results on the MSR-VTI 1kA test set provided in Table 9. It can be observed that our SFP significantly boosts different sampling strategics, further demonstrating the general applicability of our SFP mechanism.
Expansion of Relevance Score Estimator. In Sec. 3.3, we have proposed four relevance score estimators for the LSFM module. To find out which is the best, we present the ablation study results for different relevance score estimators in Table 10. We can see large gap between SFP and random sampling (w/o SFP), directly demonstrating the effectiveness of the proposed SFP mechanism. Meanwhile, both Momentum and CrossMom outperform SimDot, suggesting that introducing momentum encoder is beneficial to relevance score estimation. Collaborative that combines Momentum and CrossMom generally leads to further improvements.

---

Table 11: The model capacity of different recent methods on the MSR- VTT IkA test set.
Methods
Visual Encoder
Lingual Encoder
Fusion Layer
R@SUM
TACo [52]
155M
110M
14M
279M
157.4
Support Set [40]
136M
220M
356M
157.9
Frozen in Time [2]
114M
66M
180M
161.0
LGDN (global)
93M
55M
148M
164.6
LGDN (ours)
93M
55M
68M
215M
181.1
Rullsonce Scorre
Query7544: goes racing down the road.
0320
1221
1UR 0003
Frame
Framae 2
Frame
Frame
Frame 5
Relevence Secrca
Query7550: soccer team walking out the field.
$.350
JPNDNED
NED
10TH MINUTE
Frame
Frame
Frame
Frame
Frame 5
Visualization of our LGDN the MSR-VTT test set. We uniformly sample frames from each video and the red denotes salient frames selected by our SFP mechanism. We find that our SFP mechanism indeed filters the unmatched/redundant frames under the language supervision.
Figure 4:
Model Capacity. We also provide the detailed comparison to other methods in terms of model capacity and @@SUM (on the MSR-VTT 1kA test set) in Table 11. can be clearly seen that: (i) When fusion layers are not used (i.e., only global alignment is adopted), our LGDN (global) outperforms the state-of-the-att method Frozen in Time [2]. but with much less model parameters. (ii) Our full LGDN performs much better than all the competitors, but its parameter number (215M) still comparable to that of Frozen in Time (180M) and even significantly smaller than those of the other competitors. These observations suggest that the performance gains obtained by our LGDN is due to utilizing more model parameters.
4.5 Visualization Results
We provide visualization of our LGDN in Figure 4. We uniformly sample 5 frames from each video and provide relevance scores of 5 frames the left, and the red ones denote salient frames selected by the SFP mechanism. It can be seen that: (1) Although the holistic video is semantically related to the paired text, there still exist noisy frames (e.g., the transition in Frame I and Frame 3 of Query7500) and unrelated frames (e.g., in Frame 4 and Frame 5 of Query7544, man is rolling while the paired text is car goes racing down the road'). (2) The relevance scores obtained from the SFP mechanism correctly measure the consistency between each frame and the paired text, which indeed helps our LGDN precisely filter out noisy information for better video-land lideuaanggagee modeling.
5
Conclusion
In this work, we propose novel Language- Guided Denoising Network (LGDN) for video-language modeling, which can dynamically filter out the unmatched or redundant frames under the language supervision and thus maintain only 2-4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that our LGDN outperforms state-of-the- -ftte-ar-----ss by large margins. In the future, we will consider aggregating temporal information on salient frames and apply our approach to more challenging vadeo-language tasks (e.g., video grounding).
Acknowledgments and Disclosure of Funding
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BIJWZYJHO) 2019 1000000900099) and the Research Seed Funds of School of Interdisciplinary Studies, Renmin University of China.
10

---

References
[1] Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex M. Bronstein. Noisc estimation using density estimation for self-supervised multimodal learning. In AAAI, pages 6644-6652, 2021.
[2] Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1728-1738, 2021.
[3] Soravit Changpinyo, Piyush Sharma, Nan Ding. and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training recognize long-tail tail visual concepts. In CVPR, pages
2021.
[4] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, pages 190-200, 2011.
[5] Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, pages 10635-10644, 2020.
[6] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geolfrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, pages 1597-1607, 2020.
[7] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4 4171-4186, 2019.
[8] Jianfeng Dong, Xirong Li, and Cees GM Snoek. Predicting visual features from text for image and video caption retrieval. IEEE Transactions on Multimedia, 20( 1)):33377-33888 2018.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[10] Fartash Faghri, David J. Fleet, Jamic Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC, page 12, 2018.
[11] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heterogencous memory enhanced multimodal attention model for video question answering. In CVPR, pages 1999-2007, 2019.
[12] Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang. Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang. Hao Sun, and Ji-Rong Wen. Towards artificial general intelligence via multimodal foundation model. Nature Communications, 13:3094, 2022.
[13] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for vidco retrieval. In ECCV, pages 214-229, 2020.
[14] Feng He, Qi Wang. Zhifan Feng, Wenbin Jiang. Yajuan Lu, Yong Zhu, and Xiao Tan. Improving video retrieval by adaptive margin. In SIGIR, pages 1359-1368, 2021.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, pages 9726 9735, 2020.
[16] Anne Lisa Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pages 5804-5813 0813,581,2222 2017.
[17] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. arXiv preprint 1703v 07737777777.777 2017.
[18] Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang. Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. WenLan: Bridging vision and language by large-scale multi-moda modal pre-training, arXiv preprint 2103::220030056660 2021.
[19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoc V Lc, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In JCML, pages 4904-4916, 2021.
[20] Weike Jin. Zhou Zhao, Pengcheng Zhang, Jieming Zhu, Xiugiang He. and Yueting Zhuang, Hierarchical cross-modal graph consistency learning for video-text retrieval. In SIGIR, pages 1114-1124, 2021.
[21] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and and-language transformer without convolution region supervision. In ICML, pages 5583-5588 5894, 2021.

---

[22] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual- visual-semantic embeddings with multimodal neural language models. arXiv preprini arXiv:1411.25333 2014.
[23] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 1JCV, 123(1):32-73, 2017
[24] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. CVPR, pages 9972-9981, 2020.
[25] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: ClipBERT for video-and- aale--and-language learning via sparse sampling, CVPR, pages 7331-7341, 2021.
[26] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NewrIPS, pages 9694-9705, 9705, 2021.
[27] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for video+ language omni-represenntation pre-training. EMNLP, pages 2046-2065, 2020.
[28] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740-755, 2014.
[29] Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, and Zhongyuan Wang. HiT: Hierarchical transformer with momentum contrast for video-text retrieval. In ICCV, pages 11915-11925, 2021.
[30] Yang Liu, Samuel Albanic, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retricval using representations from collaborative experts. In BMVC, page 279, 2019.
[31] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[32] Haoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao, Zhiwu Lu, and Ji-Rong Wen. COTS: Collaborative two- stream vision-language pre-training model for cross-modal retricval. In CVPR, pages 15692-15701, 2022.
[33] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang. Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou UniVI: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint 2rXiv:2002.06:: 06355, 2020.
[34] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2 (.X.v:2204.000 204.08040000 2021.
[35] Antoine Micch, Jean- Ban-itatiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, pages 9879-990 17799-989,.20 2020.
[36] Antoine Miech, Dimitri Zhukov, Jean- -ran-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning text-video emhedding by watching hundred million narrated video clips. In ICCV, pages 2630-2640, 2019.
[37] Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy- Chowdhhuu Learning joint embedding with multimodal cues for cross-modal video-text retrieval, In ICMR, pages 19-27, 2018.
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1xXiv:1807.03777 2018.
[39] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using million captioned photographs. In NeurlPS, pages 1143-1151, 2011.
[40] Mandela Patrick, Yao Huang, Yuki Markus Asano, Florian Metze, Alexander G. Hauptmann, João F. Henriques, and Andrea Vedaldi. Support-ser set bottlenecks for video-text representation learning. In ICLR, 2021.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In /CML, pages 8748 8763, 2021.
[42] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmmed, image alt-text dataset for automatic image captioning. In ACL, pages 2556-2565, 2018.
12

---

[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, pages 5998-6008, 2017.
[44] Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, and Kate Saenko. Translating videos to natural language using deep recurrent neural networks. NAACL-HLT, pages
2015.
[45] Xiaohan Wang, Linchao Zhu, and Yang. T2VLAD: global-local sequence alignment for text-vidco retrieval. In CVPR, pages 5079-5088, 2021.
[46] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VaTeX: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, pages 4580-4590,, 2019.
[47] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pages 3733-3742, 2018.
[48] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning:
trade-off's in video classification. In ECCV, pages 318-335, 2018.
[49] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACMMM, pages 1645-1653. 2017.
[50] Jun Xu, Tao Mei. Ting Yao, and Yong Rui. MSR-VIT VTT: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.
[51] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning answer questions from millions of narrated videos. In ICCV, pages 1676,1677, 2021.
[52] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. TACo: Token-aware cascade contrastive learning for video-text alignment. In ICCV, pages 11562-11572, 2021.
[53] Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In ECCV, pages 385-401, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specified all the training details in Sec. 4.1 and Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We included the details training time and resources used in Appendix.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the creators of all used datasets in Sec. 4.1.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "pCgvuv_uFgJ", "reviewer": "Reviewer_kV1h", "summary": "This paper tackles one important issue in video-language modeling, that is,  previous works made assumptions that video frames and the text descriptions for the video are semantically correlated. However, this assumption is not valid for the real world <video,text> pairs, since first,  the video-level text descriptions may not cover all information in the video, and second, a raw video often contains noisy or meaningless information that does not appear in the text descriptions. The previous correlation assumption did not take account of these noises, and the self-attention mechanism still leaves mis-aligned frames which harm cross-modal alignment. \n\nThis paper proposed an E2E language-guided denoising network, LGDN, for video-language modeling. LGDN includes (1) a salient frame proposal (SFP) mechanism to dynamically filter out irrelevant or redundant frames and sparsely sample salient frames to improve video-language modeling,   (2) cross-modal interactions at three levels, i.e., salient frame matching at the token-level guided by language, momentum frame-level MSL-contrastive learning (MFCL), and momentum video-level contrastive learning (MVCL). Experimental results showed that LGDN outperforms the current SOTA. Ablation studies further demonstrated the importance of tackling the noise issue.\n", "questions": "The paper is clear written. However, the authors mentioned the pre-training is set up as described in the paper due to restricted computation resources. It would be useful to explain the experimental setup alternative if more computation resources are available, that is, how to scale up the pre-training setup.", "limitations": "The paper missed the important section of Limitations (and potential negative societal impacts). It is highly desired that the authors add discussions on limitations and potential negative societal impacts of this work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper addressed an important limitation in previous works of video-language modeling, that is, the impractical assumption that video frames and the text descriptions for the video are semantically correlated. There is no sufficient tackling of noises in raw videos. The existing employment of self-attention mechanism cannot fully address this problem. These irrelevant or redundant information still harms cross-modal alignment.  Hence, this work is valuable to the research community.\n\n2. The SFP mechanism and the three-level cross-modal interactions are sound, well motivated, and have good novelty.  SFP is achieved through exploring MVCL and MFCL. After obtaining the language-guided salient frames, LGDN also has a language-guided salient frame matching (LSFM) module to conduct token-level semantic alignment between visual patches and words for improved performance. The final loss of LGDN is the combined loss of MVCL, MFCL, and LSFM.\n\n3. The Related Work section is clearly written. It summarizes significant works in the past and their limitations, and clearly explained the choice of momentum frame-level MSL-contrastive learning to help address the mis-aligned frames.\n\n4. The paper also conduct the interesting investigations of the four strategies for estimating relevance scores.\n\n5. The evaluations are comprehensive, on four public text-video retrieval datasets and one VQA dataset. The evaluation settings support a fair comparison to previous models.  Performance gains from LGDN over existing approaches, including SOTA, are quite strong.  Experimental results showed that LGDN outperforms previous methods, including the current SOTA,  by a large margin on text<->video retrieval on MSR-VTT. LGDN also outperforms methods exploiting extra modalities or those pre-trained on very large video data.  LGDN also shows promising model capacity, as its performance is significantly improved when trained on much larger pre-training data. The extensive evaluations on other text-video retrieval datasets and VQA also demonstrated that LGDN outperforms existing approaches including the current SOTA.  \n\n6. Ablation studies verified the contributions from the proposed SFP, MVCL, MFCL, and LSFM. Last but not least, the visualization results are interesting and helpful for directly illustrating the importance of addressing the noisy frame issue and showing that the SFP mechanism helps LGDN to improve video-language modeling.\n\n7. Although the code is not released, the main body and Appendix provide enough details to help reproducibility. The Appendix also includes more detailed experimental results on analyzing effect of SFP mechanism, different relevance score estimations, and memory bank sizes, and useful additional visualization results.\n\nWeaknesses:\n\n1. The paper missed the important section of Limitations (and potential negative societal impacts). It is highly desired that the authors add discussions on limitations and potential negative societal impacts of this work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c6WO4PtpRYv", "reviewer": "Reviewer_HH2K", "summary": "This paper introduces LGDN, which utilizes a self-taught mechanism to select salient frames to train the fusion layers. The vision and language encoder still use all frames and texts to train, and their goal is to learn representations while computing relevance scores for the frame selection. The whole mechanism by contrastive learning helps to filter out noisy information in the last few layers, boosting performance while reducing computation costs. They also demonstrate better empirical results over the baselines.", "questions": "1. ClipBERT samples frames before feeding them into the model while your approach does the sampling only before fusion layers, so I wonder about the memory efficiency and speed of LGDN compared to ClipBERT, Frozen in Time, and other methods.\n2. I understand that the SFP might address the misalignment issue, but how it can help to ease the redundancy in the video. If you compute the relevance score between text and frames, it's likely that similar frames have similar scores, and I am not sure what can prevent LGDN to select the other frames that are similar to the salient frame. I feel like some diversity loss might be necessary to address this problem.\n3. What's the model size of other baselines? LGDN is composed of vision, language, and fusion Transformers while Frozen in Time doesn't have a fusion Transformer, making me concerned if the performance gain somewhat comes from the bigger models. For better comparison, you can try to use lesser layers in vision and language encoders.\n4. What are global and local alignments used in the inference phase? I didn't find an explanation in the paper. Why does using global inference generates such low performance?\n5. The salience sampling breaks the temporal dependency of the video (compared to uniform sampling), which is an important characteristic to solve most video tasks. Do you think this sampling will prevent LGDN from applying to other video tasks that highly depend on temporal information?", "limitations": "The paper's writing can be improved. The method has many components, and I was confused about which part addresses what problem at the beginning. But in general, I still understand the idea and contribution of the paper. My main concern is about the fairness of the experiments and some questions about the technical approach (refer to Questions). I will raise the score if those problems are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Pros:\n\n* The paper proposes an effective method to select frames from noisy video, leading to an improvement in both fine-tuned accuracy and computation efficiency.\n\n* To address the noisy problem in video is novel to me, it might become a new approach to constructing multi-modal models. There is no perfect and exact matching between any kinds of pair data so that the impact can be extended to other domains.\n\n2. Cons:\n\n* The empirical results are not fully convincing to me. I think it would be better if the authors can report the model capacity of each method (see Questions 3).\n\n* The central claim of this paper is that noisy video might harm the training, but the model still highly relies on learning on noisy data as the vision and language encoder still use full video to learn how to match the relevance between language and frames.\n\n* It seems to me that the redundancy issue has not been addressed by LGDN (see Question 2), and the sampling might introduce another problem (see Question 3).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CfmlGTnpaDU", "reviewer": "Reviewer_UdJY", "summary": "The paper proposes a Language-Guided Denoising Network (LGDN) for video-language modeling, which can deal with the noisy information in video frames. LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2–4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that LGDN outperforms the state-of-the-arts by large margins.", "questions": "1. Table 2 shows the improvement of $\\mathcal L_{MVCL}$ is marginal. Is it necessary to the LGDN model? Can MIL-NCE be used to compute the $\\mathcal L_{MVCL}$?\n2. The authors claim that the proposed LGDN outperforms the state-of-the-arts by large margins. However, as shown in Table 8, LGDN performs worse than Clip4CLIP. This point needs further discussion. \n3. Since the pre-training with CC12M brings significant improvements, it would be nice to compare some SOTA methods under the same pre-training settings.\n", "limitations": "The authors did not discuss the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe paper is well organized and easy to read. The noise issue in video-text retrieval task studied by this work is practical, and the proposed salient frame proposal mechanism seems simple and effective. Extensive experiments are performed on video-text datasets and the experimental results are promising.\n\nWeaknesses:\nThe paper needs more analysis and ablation studies to support its network architecture and proposals. For example, in-depth comparison between salient frame proposal (SFP) mechanism and sparse sampling or other frame sampling techniques. Also, some experimental results (e.g., Effect of SFP mechanism) in supplementary material can be included in the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept