Revitalize Region Feature for Democratizing Video-language Pre-training of Retrieval
Authors: Guanyu Cai
Yixiao Ge
Binjie Zhang
Jinpeng Wang
Rui Yan
Xudong Lin
Ying Shan
Lianghua He
Xiaohu Qie
Jianping Wu
Mike Zheng Shou
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language retrieval. Despite the impressive results, VLP research becomes extremely expensive with the need for massive data and a long training time, preventing further explorations. In this work, we revitalize region features of sparsely sampled video clips to significantly reduce both spatial and temporal visual redundancy towards democratizing VLP research at the same time achieving state-of-the-art results. Specifically, to fully explore the potential of region features, we introduce a novel bidirectional region-word alignment regularization that properly optimizes the fine-grained relations between regions and certain words in sentences, eliminating the domain/modality disconnections between pre-extracted region features and text. Extensive results of downstream video-language retrieval tasks on four datasets demonstrate the superiority of our method on both effectiveness and efficiency, e.g., our method achieves competing results with 80% fewer data and 85% less pre-training time compared to the most efficient VLP method so far.

Under review as a conference paper at ICLR 2023
REVITALIZE REGION FEATURE FOR DEMOCRATIZING VIDEO-LANGUAGE PRE-TRAINING OF RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Recent dominant methods for video-language pre-training (VLP) learn transfer- able representations from the raw pixels in an end-to-end manner to achieve ad- vanced performance on downstream video-language retrieval. Despite the impres- sive results, VLP research becomes extremely expensive with the need for mas- sive data and a long training time, preventing further explorations. this work. we revitalize region features of sparsely sampled video clips to significantly re- duce both spatial and temporal visual redundancy towards democratizing VLP research at the same time achieving state-of-the-art results. Specifically, to fully explore the potential of region features, we introduce novel bidirectional region- word alignment regularization that properly optimizes the fine-grained relations between regions and certain words in sentences, eliminating the domain/modality disconnections between pre-extracted region features and text. Extensive results downstream video-language retrieval tasks on four datasets demonstrate the superiority of our method both effectiveness and efficiency, e-g., our method achieves competing results with 80% fewer data and 85% less pre-training time compared to the most efficient VLP method so far (Lei al. 202T)
INTRODUCTION
Video-language pre- pre-training (VLP) (Lei et al. 2021 Li et al. learns video and language representations in self supervised manner has become the most popular practice to cope video-language retrieval (Lee et al. 2018: Liu et al. 2019a). Recently, end-to-end methods (Bain et al.) 2021 [222]/elleer Zellers et al.) 2021) that learn Video representations from raw pixels have dominated due their strong performance
2020a Miech et al. (2020) that jointly
downstream tasks. Despite significant progress, these methods are quite data-hungry due to large number of model parameters and uncurated raw inputs. The pre-training stage turns out to be inefficient and expensive with massive pre-training data and long pre-training time, making it difficult for researchers to pursue research VLP.
Previous work (Lei et al. 2021) attempts lower the barrier for VLP via removing visual redun- dancy. They point out that video clips with sparsely sampled frames are sufficient enough to capture key semantics for pre-training, since adjacent frames often contain similar scenes. The effort en- ables more efficient VLP with competitive downstream performances. Besides the temporal visual redundancy, we argue that, in contrast to the text with highly abstract semantics, each frame of the video clips also has heavy spatial redundancy.
Towards this end, we further propose to remove the redundant spatial information in sparsely sam- pled video clips via the claim that frame is actually worth around 30 objects (based experiments Section (4.4). Specifically, we revitalize offline region features that were all the rage in image- language tasks (Liu et al., (019a) to encourage efficient VL.. Region features are generally pre- extracted by a pre-learned object detector (Anderson et al., 2018). Rather than the dense and con- tinuous visual signal of the raw pixels, the region features
sparsely distributed with the compact information of salient visual contents, which are the most useful for video-text understanding, The sparse sampling significantly reduce the complexity of attention mechanism, which enables our model to have larger capacity with less FLOPs. We further advocate "less is more" for one more step towards democratizing VLP research.
As is known, methods using off-the-shelf features (Lee et al. 2018) have been phased out in visual- language tasks due the inferior downstream performances. Previous work (Lei al. 2021) at-

---

Under review as a conference paper at ICLR 2023
Paoplejare having Eai pouble.
UniVL
Frozen
MSRVIT
with is Cactpea He rodie
He is
ClipBERT
UniVL
school.
Frozen
ClipBERT
Ours
going to help
aroorre no leem to ndg bicycl
@@1
40
3,000 4,000 GPU Hours
(a) Region- Region-Word& Alignment.
(b) MSRVTT Retrieval.
(c) Pre-training Time.
Figure 1: Region- word alignment and results. (a) Region- woodd alignment to reason the detailed correspondence between salient regions and words. (b) and (c) demonstrate that our method signifi- cantly improves text-to-video retrieval while reducing pre-training time to a large extent.
tributes the unsatisfactory pre- pre-training performance of pre-extracted features their disconnections with the current domain and language modality. We would like to clarify that such disconnections be properly eliminated by imposing fine-grained cross-modality alignment regularization.
Specifically, besides the common late fusion regularization on the global visual-text representations (Bain et al.] (021), we introduce novel bidirectional region-wort wood alignment regularization under the observation that objects extracted from video frames are naturally associated with certain words in the corresponding sentences. For instance, as demonstrated in Fig. I the keywords "people", "car" and "bicycle" share high-level semantics with cropped regions (highlighted with bounding boxes), respectively. To model and promote such a detailed cross- cooss-mo relationship, we build bidirectional connections between extracted regions and words. In the Regi inngionnWorr manner, we estimate the region-to-sentene similarity resorting to the similarities between each region and all the words in sentence. The average eegion-to-sentence similarity over all the regions of video clip is treated as the iido-to-ssentencs similarity, which further maximized for positive pairs. Similarly, the Word woord--eeiioo manner is conducted to measure and optimize the sentence-to-vides similarity according to the similarities between each word and the corresponding regions. We surprisingly find that the proposed fine- tne-grained region- woodd alignment constraints also be seamlessly integrated into end-to-end VLP methods (Bain et al. 2021). achieving promising performance gains.
In summary, contributions three-fold: (1) We revitalize region features towards democratiz- ing VLP via removing both temporal and spatial visual redundancy. Specifically, our efficient VLP model can maintain state-of-the-al performance on multiple downstream tasks with 80% fewer data and 85% less pre-training time than ClipBERT, which is the most efficient end-to-end VLP method so far. (2) We clarify that the inferior performance of off-the-ste
features in previous attempts (Li eet al. 2020a Zhu & Yang 2020; (Snn et al.) 2019 Yu et al.] [0018] Gabeur et al.) (0200) lies in the sub-optimal learning regularization. We tackle the challenge with newly proposed bidirectional region- word constraint, which optimizes fine-grained visual-text relations and properly eliminates the domain/modality disconnections of the region features. (3) Our method shows competitive re- sults four downstream video-langun language retrieval tasks. We surprisingly observe that the introduced region-word alignment regularization can also effectively boost the end-to-end method (Bain et al. 2021) with noticeable improvements.
RELATED WORK
Video- Languag Pre-training, Early VLP methods (Li et al. 2020a Zhu & Yang 2020 Sun et al. 2019 Yu et 2018 Gabeur al., 2020) introduce pretrained models on other tasks pre-extract video representations. Some of them (Li et al. 2020a Zhu & Yang 2020 Sun et al. 2019) utilize action recognition backbones (Feichtenhofer
al.) 2019 Hara et al. 2018) 8) pre-extract video representations. These backbones are designed with 2D (He et 2016) and 3D (Hara et al.) 2018) CNNs to capture spatial and temporal information in videos. Others Yuu al. 2018 Liu et al,, 2019b Gabeur et al., 20200 Wang et al. (021b) fuse multiple "Experts" that are trained on different modalities, such as audio classification (Hershey et al., 2017). OCR (Gabeur et al. 2020), image classification (Huang et al.) 2017) and so to fully exploit cross-modal high-level semantics in videos. Recently, and-to-end models (Miech et al. 2020 Lei et al.] 2021 Bain et al.
2021 Zellers
et al. 2021 Fu et al. 2021) are proposed. Some (Miech al. 2020
Lei et al. 202
Zellers al.)
2021 utilize CNNs to extract video features, others (Bain et al.
2021
Fu et al. 2021)
replace CNNs
with ViT (Dosovitskiy al. 2021) build pure Transformer- based
VL. model.
2

---

Under review as a conference paper at ICLR 2023
Visual Sampling
N/
-150 vediors
Grid Feature (ClipBERT)
Video
Video
(Frozen)
Dense Sampling
-30 vediors
oepin
mooppr MND
Region Feature! (Ours)
Sparse Sumpling
Figure 2: Comparison conventional VLP visual sampling and our method. We introduce region
feature to the VLP pipeline achieve sparse sampling and accelerate the pre-training period.
Region Features in Image-Language Pre-training, Region features extracted by an object detec- tor (Anderson et al. (2118) are adopted by image-language pre-training (ILP) methods (Chen et al.) 2020b Li et al. 2020b). They represent an image to set of region features that focus salient regions that are much more natural basis for attention (Egly et al. 1994 Scholl 200T]. Region features enable the interaction between image and text at the object level. Recent ITP methods (Li
(ct al. 2020b) Chen et al. 2020b Yao et al. 2022) attempt to form fine-grained alignment between image and text based region features.
Fine-Grained Alignment. Fine-grained alignment between visual and textual contents has been explored in image-language retrieval for several years. Some studies (Lee et al., 2018 Liu al., 2019a; ei al. 2017] Chen et al. 2020aa Wang et 2019) propose different attention mechanisms match visual regions and captions. Recent ITP methods (Li al.) 2020b Chen et al., 2020b) also explore the aligmment between regions and words. Considering the low efficiency for video processing, previous VLP methods (Luo al., 2020) [Sun al. [019 Gabeur al., 2020) adopt simple contrastive loss or binary classification to align video and language. Recently, TACo (Yang) al. 202T) has attempted to build up fine-gra ine-grained alignment. However, still suffers from so large computation cost that simplifies its original design. Benefiting from our efficient pipeline, we propose a novel region-word word alignment method to better connect the video and language.
METHOD
A video and its corresponding caption are taken as input. The video is initially embedded to region features by an off-the-shelf detector (Anderson al., 2018) before being fed into a video encoder. The corresponding caption is encoded to language features by language encoder. During the pre- training, the video and caption features are optimized towards the objective of global-local align- ment. After pre-training, the model can be transferred to video-language retrieval with fine-tuning.
3.1 MODEL ARCHITECTURE.
Input. Given paired video V and sentence T' as raw inputs. Similar to previous VLP methods (Bain al. 2021 Lei al. 2021 3. T' is tokenized into word tokens (w> [f where denotes the number of tokens in T and We denotes the [CLS] token. These word tokens are inputted to language encoder. Video frames {f1, f2, .... fM} are sampled from V. where M denotes the number of frames. Given frames (fi, f2. fM } of a video, objects and salient regions in each frame are detected by Faster RCNN (Anderson et al., 2018)) pre-trained on Visual Genome (Krishna et al., (017).. Pooled Rol features of these regions are concatenated together as features {o,, Rd R
===1 of the whole video, where denotes the number of regions. Video encoder takes these Rol features as the inputs.
In this work, efficient pre-training. Tuning the detector requires extra computational and memory overhead, and it is also infeasible acquire bounding box annotations in video-language datasets.
pre-extracted region features rather than fine-tuning the detector end-to-end for
Temporal Modeling, We add [CLS] token On to represent the whole video so that the input be- comes fo, \N We combine temporal position embeddings and location embeddings to model temporal clues in videos. To encode location information for each region feature, we use a linear layer FC to project 7-dimensional location vectors (Ln [21, ; 1/, T2, 92, W7, h, w h) N x (normal-
3

---

Under review as a conference paper at ICLR 2023
ized top/left/bottom//ight coordinates, width, height and area) to (i R In=o:
k FC.().)
The change of location reveals movement of an object. To indicate frames at different moment, we introduce learned temporal position embeddings clues are revised as o, = on + 4 + Pmm. where m denotes o, is extracted from m-th frame.
R RMxd Thus, region features with temporal
Compared with video transformer (e.g.. TimeSformer (Bertasius al. 2021)), [), our method captures temporal clues more efficiently by removing time attention and reaches comparable performance. Comparison of complexity and performance described Section 3.4 and Appendix (A.4.5
Video Encoder. Video Encoder encodes {or, Pn with a Transformer-basee network Ev:
fra L = Ev (fo,
(2)
where Tn denotes the output region feature. We introduce VIT (Dosovitskiy et al. 2021 Ev. Video Encoder makes region features interplay with each other, and outputs refined region features frn Jyy
where is encoded from [CLS] token and treated the global video feature.
Language Encoder. Similar to previous VLP methods (Lei et al. 2022 Bain et al. (202)) Language Encoder is based on BERT (Devlin et al. 2018). Word tokens encoded to { E R' N=00
{4) Evv (20)
where ta denotes the output language feature and we adopt DistillBERT (Sanh al., 2019 as E {ti [=) are then aligned with region features by our proposed pre-training objective.
3.2 REDUCE VISUAL. REDUNDANCY.
To democratize video-language pre-training, we attempt remove both temporal and spatial redun- dancy. We involve two training strategies to ensure the pre-training time is acceptable.
Temporal Visual Redundancy Frame Sparse Sampling. To save pre-training cost and ensure the performance on downstream tasks, we adopt frame sapling strategy similar to ClipBERT (Lei) jet al., 202T). During the pre-training period, only a single frame is sampled randomly for each video. When finetuning on downstream tasks, a dense sampling strategy is adopted to capture more visual information from vidcos. We observe that such extremely sparse sampling still achieves competing results on downstream tasks in our framework. Experiments in Section 4.4 show how frame sampling strategy affects pre-training and finetuning.
Spatial Visual Redundancy Object Region Extraction. Considering the number of objects and salient regions detected by a detector uncertain, to balance the efficiency and performance, we explore strategies to determine the number of objects per frame and how choose objects. To determine the number objects, we conduct an ablation study and find that 30 objects per frame is the most suitable setting. Because it achieve performance close more objects and less objects cause a obvious performance drop (Section 4.4).
As shown in Fig 2 compared with methods using dense sampling strategy (e.g., Frozen (Bain al., 2021) with patch embeddings, ClipBERT (Lei et al.) 202T) with grid features), our method utilize sparse feature vectors to represent frame. Due self-attention in Transformer is (NN)² time complexity, our method achieve much better efficiency.
3.3 PRE-TRAINING OBJECTIVE.
To jointly learn video-language representations from large-scale datasets (Bain al. 2021), we in- troduce global-local video-language alignment to connect video and language from coarse to fine. The global-local alignment consists of two pre-training objectives: and ii) region-word alignment. For video-sentence alignment, we employ contrastive learning with [CLS] tokens to align video and language globally. In the region-wond alignment, regions and words are aligned to reach fine-graince alignment via newly proposed bidirectional regularization.
video-sentence alignment,
Video-Sentence Alignment. Two losses are optimized to increase the video-to-language and language-to-video similarity:
B
exp(r cls (2) Σ exp /0)
gloobal
log
(4)
B
4

---

Under review as a conference paper at ICLR 2023
gglobal 12v
Σ log
expi exp Tch//
(S)
exp(Cel /o)
where rals and t'ss are normalized embeddings of i-th video's ' [CLS] token and j-th caption's [CLS] token batch of size B, respectively. a denotes the temperature coefficient.
Region-Wood Word Alignment. The local alignment also attempts to maximize video-to language and sunuuage-to-iiee similarity, but in a -ne-graieed way. We take video-to-langee tllaguto- as an example to illustrate the details of region-word alignment. For i-th video with region features {r, and j-th
sentence with word features {4) n-th region feature firstly attends to each word feature to pick up the most relevant words according to attention weights:
exp((rn,ti) ti))
and Ek= (xpp(re (2))
(6)
where (:.-) denotes the cosine similarity, (n,,1 denotes the similarity between the 12-th region feature and 1-th word feature. Intuitively, semantics in videos are usually associated with nouns, verbs and
adjectives Yang et al. (2021; Liu et al. (0019a). Other types of words, e.g., prepositions, are not
"concrete" enough in videos Yang et al. 2222I. Thus, to make the region only attend to the most relevant words and ignore other noisy words, we refine an as follows:
1 Σ anaa] an.i
d = Iaand - L
where [x:] = if x: >> 0 else 0. d',, is the refined similarity between the n-th region feature and I-th word feature. By regarding the average of attention weights as threshold, we set attention weights of irrelevant words (below threshold) O. By conducting this operation, we ensure the model only focuses on the relevant words with respect to a video.
Given the refined attention weights, an attended sentence feature with respect to n-th region feature is calculated as: a₂ = Σi= t4, where an denotes the attended sentence feature. We compute such attended sentence feature for each region feature. By summing up the similarity between all region features and their corresponding attended sentence features, we get the similarity of i-th video and j-th caption:
1 Si N
(En, and
(8)
The video-to-language loss is calculated with contrastive loss:
Σ log exp(St.:/o)
B
Clocal
Σ exp(Sr./o)
where denotes the temperature coefficient. The language-to-video loss Clocal is similar to Clocal More details are described Appendix [..
Overall Pre-training Objective. Combining the above losses, the overall pre-training objective is:
L= Gloobll + - +C - The proposed alignment can fully exploit previous knowledge learned by region features and build up a more precised alignment compared with existing align-
ments Yang al. 2021 Yao et al. 2022) (We examine the effectiveness and analyze the reasons in
Section
3.4 COMPLEXITY ANALYSIS.
VLP methods (Li et al. 2020a Zhu Yang 2020) are criticized for excessive demand on memory and computation. However, we claim that our method is more efficient than recent end-to-end methods (Bain et al. 2021 Lei et al. 202T) overall. Although we need pay extra time to extract region features, it negligible compared with the pre-training time that our method saves, especially we only need to extract them once but can reuse them for plenty of times. When it comes downstream tasks, the detector only needs to extract features for queries, as we can assume gallery features are pre-extracted and stored in practical. Such extra time is also acceptable considering the performance improvements. We give complexity analysis as follow.
Previous
Pre-training, We compare the pre- tre-training time and R@1 on MSRVTT retrieval of our method with different settings and other methods. Because different methods use different number of GPUs,
5

---

Under review as a conference paper at ICLR 2023
Forward Backward
Method
Data
GPU Hrs
R@1
VIOLET
395.1
758
Frozen
5.8M
4800
31.0
UniVL
132M
2496
21.2
HERO
7.6M
8064
20.5
Frozen
87.6
217.7
VIOLET
185.8M
2240
34.5
ClipBERT
5.6M
768
22.0
Ours (4H/1.0)
5.8M
1600)
36.3
Ours (1F/1.0)
5.8M
800
36.0
Ours
75.1
Ours (IF/0.5)
2.9M
416
36.4
Ours (1F/0.2)
1.2M
104
34.6
T1m00m5)
Table I: Comparing the pre-training efficiency with existing video-language pre-training meth- ods. 4F means that frames per video are sam- pled for pre-training. 0.2 means that only 20% pre-training data are used.
Figure 3: Comparing the running time of a training loop with existing video-language pre- training methods. Batchsize is set to 16 and mixed precision disabled for all methods.
introduce GPU Hours, i.e., number of GPUs x pre-training time, to measure the computational cost. In Table [ our method is significantly more efficient than other methods. Our method achieves the best performance, even when we only use 20% pre-training data and extremely sample only frame per video. Compared with the most efficient VLP method so far, i.c., ClipBERT (Lei et al. 2021), our method save 85 % pre-training time. The reason why the model pretrained with 50% data performs better than those pretrained with more data is that finetuning results may not be strictly correlated with the data scale. Zero-shot results reflect the impact of data scale more precisely. In Fig. 6 zero-shot results degrade slightly as the size of data and the number of frames decreases.
Besides the whole pre- tre-trainig. we also explore the rumning time of training loop. We compare our method with Frozen (Bain et al., 202T) and VIOLET (Fu et al. 2021) to show the efficiency of our method. As shown in Fig. 3] our method requires much less time than Frozen (Bain et al.) 2021) and VIOLET (Fu al. 2021 The forward time of our method only 43.8% Frozen's 's and 9.7% of VIOLET's. The backward time of our method is only 34.5% of Frozen's 's and 9.9% of VIOLET's.
test inference time of each model components on text-to-video retrieval. We build up galleries: i) test split of LSMDC that contains 1000 samples; ii) 12000 samples collected from WebVid's train split. In practical, samples in gallery are pre-extracted, thus record the running time of detection, feature extraction of query and ranking.
Downstream Retrieval.
As shown in Table (T~VV denotes Text-to- Text-to-Video), detection costs around 100ms for video. Feature extraction of texts and videos takes very little time. As the gallery expands, ranking time becomes the major consumption. Considering the gallery capacity is usually larger than 12000 in practical, the extra time for detection is acceptable.
12000 Samples
1000 Samples
Frozen
Ours
Parts
Parts
V-T
FLOPs
Param
FLOPs
Param
Language
5.6ms
4.8ms
Language
9.7B
66M
9.7B
66M
Video
17. 61.m
16.4ms 103. 2ms
Video
590B
121.4M
41.8B
86.8M
Detector
103. 2ms
Detector
455.2B
60.6M
Rank
37s.2ms
345 8ms
31.8ms
30.0ms
All
599.7B
187.4M
506.7B
213.4M
Table 2: Inference time of each component.
Table 3: Inference FLOPs of each component.
In Table We compare our method with Frozen on inference FLOPs and parameters of each com- ponent. Frozen introduces TimeSformer the video encoder. The dense visual sampling and time attention in T'ieeformer increase the inference FLOPs rapidly. Thus, our method even has lower inference FLOPs with more model parameters.
EXPERIMENTS
4.1 DATASETS
Following the recent works (Bain al. 2021 Fu al. 2021 involve video- language dataset and an image-language dataset to pre-train our model. i) Web Vid2.5M (Web Vid); and fi) Google Conceptual Captions (CC3M).
Pre-training.

---

Under review as a conference paper at ICLR 2023
TextoVVidoo
Vidco
Method
Method
R@1
R@5
R@ 10
R@1
R@5
R@ 10
Base
22.5
59.5
Frozen
31.0
59.5
70.5
Ours RWac++WAA)
36.0
61.0
71.8
Frozen+RWA
35.1
60.7
70,9
Zero-shot
Zero-shot
Base
16.1
34.9
44.2
Frozen
18.7
39.5
51.6
RWWA
22.5
48.1
59.5
Frozen+RWA
21.7
42.2
52.2
(a) Results on MSRVTT retrieval with Basc.
(b) Results on MSRVTT retrieval with Frozen.
Table 4: Effectiveness of our region- word aligmment. "Base" denotes the baseline that adopts the
same region features as input while only using video-sentence global alignment as the objective.
Text=xVide
Text- Video
Method
Method
R@1
R@S
R@ 10
R@1
R@5
R@ 10
Base+TACC IACO
29.8
60.1
70.7
Ours w/o Refinement
34.5
58.3
70.2
Base+FILIP
27.8
55.0
65.1
Select Token
32.8
55.5
69.2
Ours RWA)
36.0
61.0
71.8
Ours
36.0
61.0
71.8
Table 5: Comparison other alignments. "Base" is the model whose pre-train objec- tive only contains global alignment.
Table 6: Ablation study of refinement. Select To- ken denotes that input captions only contain nouns and verbs.
Downstream Tasks. We evaluate our method on text-to-video retrieval, across 4 downstream tasks. We report results on i) MSRVTT; ii) DiDeMo; iii) LSMDC and iv) MSVD. We adopt the common @@K K (K=1, 5, 10) metric and Median Rank (MedR) measure the performance of retrieval.
Details of datasets and implementation are described in Appendix A.2 and A.3
4.2 ANALYSIS OF REGION-WORD ALIGNMENT
In this section, we verify the effectiveness of region-word alignment (RWA) by compare with a baseline model and other alignment methods. Furthermore, by analyzing experimental results, we examine that region- baseed models require fine-grained aligmment to exploit prior learned by region features, and an appropriate mechanism essential for precise alignment.
Effectiveness. We conduct an ablation study of RWA on Base (i.e., the pre-training objective is modified into L - C. ba C and Frozen. Results on MSRVTT retrieval are shown in Table
It's obvious that RWA improve the performance of Base and Frozen on all metrics for finetuning and zero-shot settings. The results demonstrate that RWA can better align language and videos and thus enhance the performance of VLP. Another advantage of RWA is that can adapt to end-to-end methods (e.g., Frozen) seamlessly shown in Table 4b
Comparison with other alignment methods. We compare RWA with other fine fine-grained alignment methods, e.g., TACo (Yang et al.) (202I) and FILIP (Yao al. 2022), MSRVTT retrieval. Resutls are shown in Table Though they somewhat eliminate the disconnections between region features and language modality with improved performance
baseline, they are still much inferior to since they only associate the image/region with a single word. However, region can correspond to multiple words, and these cases are covered by RWA. The above sub-optimal alignment is not critical for their end-to-end frameworks due to the incremental gains, but is fatal to our framework due to the incomplete elimination of disconnections, leaving a significant gap from SOTA results.
Ablation on Refinement. We refine attention weights in Eq 7 to choose the most relevant words and avoid noisy words hurting the alignment between words and regions. To determine the effect of this operation, we compare it with another operation, i.e., Select Token, which manually selects nouns and verbs as the input caption. Because nouns and verbs usually contain more semantics than other types of words (Yang al. 2021). As shown in Table 6 our proposed refinement improve results the model without refinement. Select Token even hurts the performance baseline. It means that filtering out noisy words indeed help align words with regions, however, appropriate implement important.
Discussion. As is known, region features extracted by detector have already learned the concepts of different "objects". Utilizing such prior to better align video and language is the intuition of region-word alignment. In Table 4 our region-based model performs worse than end-to-end
7

---

Under review as a conference paper at ICLR 2023
Text
Video
Text
Video
Method
Method
@
R@5
R
10
MedR
R@1
R@5
R@ 10
MedR
JSFusion
9.1
21.2
34.1
36.0
MMT
26.6
57.1
69.6
4.0
MEE
9.3
25.1
33.4
27.0
ActBERT
16.3
42.8
56.9
10.0
CE
11.2
26.9
34.8
25.3
SupportSet
30.1
58.5
69.3
3.0
MMT
12.9
29.2
38.8
19.3
AVLNet
27.1
55.6
66.6
4.0
AVLNet
17.0
38.0
48.6
11.0
TACo
29.6
59.7
4,0
Dig
15.8
34.1
43.6
14.3
ClipBERT
22.0
46.8
59.9
6.0
Frozen
15.0
34.1
39.8
20.0
Frozen
31.0
59.5
70,5
3.0
VTMCE
14.9
33.2
19.0
Ours
36.0
61.0
71.8
3,0
MDMMT
18.8
38.5
47.9
12.3
Zero-shot
Ours
25.2
45.5
54.5
7.0
SupportSet
12.7
27.5
36.2
24.0
Zero-shot
Frozen
18.7
39.5
51.6
10.0
Ours
14.3
25.8
32
49.5
Ours
24.0
44.0
52.6
8.0
(a) LSMDC retrieval
(h) MSRVTT retrieval
Video
Text Video
Method
Method
@@1
R@5
R@
MedR
R@D
R@5
R@ 10
MedR
S2VT
11.9
33.6
13.0
VSE
12.3
30.1
42.3
14.0
FSE
13.9
36.0
11.0
VSE++
15.4
39.6
53.0
9.0
CE
22.6
51.1
5.0
MCues
20.3
47.8
61.1
6.0
ClipBERT
20.4
CE
19.8
49.0
6.0
Frozen
31.0
59.8
72.4
3.0
SupportSet
28.4
60.0
72.9
4.0
Ours
41.4
67.6
77.6
2.0
Frozen
33.7
64.7
76.3
3.0
Zero-shot
Ours
50.9
78.9
87.0
1.0
Frozen
21.1
46.0
Zero-shot
Ours
29.6
53.0
65.1
4.0
Ours
41.6
69.8
80.5
2.0
(c) DiDeMo retrieval
(d) MSVD retrieval
Table 7: Comparisons with statc-of-the-ar ahr results on text-to-video retrieval.
model (i.e., Frozen) when only global alignment is conducted. However, region- based model out- performs end-to-end model when adding fine tneeggained alignment. The results verify that fine- tne-grained alignment is exactly the appropriate way to exploit the implicit semantics region features.
However, as shown in Table existing fine-grained alignment improve the Base model with limited gains. It is attributed the way these methods build up fine-grained alignment, where they only choose one word for region. In real-world scenario, region could be associated with multiple words. In our methods, we align region with multiple words and vice versa. This modification enables our method build up more precise alignment between video and language.
4.3 COMPARISONS WITH STATE-OF-TIIE-ART
We compare our method with cate-of-the-an art methods on text-to-video retrieval tasks. All results in this section is based the model that samples 1 frame per video during pre-training and samples 8 frames per video during finetuning.
Table summarizes the results four benchmarks. Across all tasks, method achieves sig- nificant improvement compared with previous methods both finetuning and zero-shot settings. 2021) by 6.4 on R 1. Similarly, on MSRVTT, our method surpasses Frozen (Bain et al. (021) by 5.0 and 5.3 on R for finetuning and zero-shot settings. For DiDeMo and MSVD, our method brings more than 10 improvement compared with existing best methods. Another advantage of our method is that our zero-shot per- formance significantly surpasses other methods. This means that our method builds up a good align- ment between video and language during pre-training and generalizes well on different datasets.
On LSMDC, our method outperforms MEE (Dzabraev
4.4 ABLATION STUDY
Data Scale. We explore the impact of different pre-training data scales. We compare models that are pre-trained with different scales of data, i.e., 20%, 50%, 60%, 80% and 100%, on MSRVTT retrieval. As shown in Fig. as the data scale decreases to 50%, R@ 1 of different scales are very close. Only if the scale drops 20%, R@ decreases 34.6, which is still better than other state-of- the-art methods' results, e.g., Frozen, SupportSet and TACo. The model without RWA has a similar trend to the with RWA across different scales.
8

---

Under review as a conference paper at ICLR 2023
R@1
1
R@1
40
36
36.3
35.2
36.4
34.6
40
33.6
34.6
36
36.3
36.2
22.5
22.8
22.3
21.5
20.6
30
21.6
22.6
24
24.2
20
10
Ours
10
Finetune
Ours w/o RWA
Zero-shot
0
0
100%
80%
60%
50%
20%
10
20
30
40
Scale
Objects
Figure 4: Effect of different data scales.
Figure 5: Different number of objects.
R@1
R@1
40
36
35.4
36.3
35.4
40
29.9
34.2
35.1
36
36
30
24
4251
24.6
26.2
30
26.4
20
20
10
Finetune
10
Finetune
Zero-shot
2
8
16
Frames
Frames
Figure 6: Number of frames (pre-training).
Figure 7: Number of frames (finetune).
Number of Regions. To determine how many regions represent video frame, we vary the number of regions and test our method's performance on MSRVTT retrieval. As the results shown
Fig. 5 R @ is improved as the number of regions increases from 10 to 30. When the number continues to increase, the gain on R@ is not obvious. Considering more regions increase time complexity rapidly, we generally represent a frame with 30 regions.
Sampling Strategy. To quantitatively evaluate the impact of sparsely sampling strategy during pre-traiming, we compare four models that sample 1, 2, 4, 8 frames per video respectively. When conducting downstream tasks, 8 frames are sampled. The results on MSRVTT retrieval are shown in Fig.66) As the number of frames increases, the zero-shot results are improved, whereas the finetuning results not closely related to the number of frames. The results show that dense sampling during pre-training is not necessary. This means that we can sparsely sampling frames during pre-training to save pre-training time.
Similarly, we evaluate the impact of sampling strategy during finetuning. We use the same pre- training model and then sample 2, 4, 6. 8, 16 frames per video respectively during finetuning. The results on MSRVTT retrieval are shown in Fig. 7 Generally, as the sampling frames increase, the retrieval performance increases. When the number of frames is 16, compared with sampling 8 frames, no gain is achieved. Thus, 8 frames is enough for our model.
Results in Figs. and prove that sparse sampling during pre-training and dense sampling during finetuning is a reasonable solution for text-to-video retrieval.
5 CONCLUSION
In this work, we introduce the most efficient rideo-language pre-training method for video-language retrieval to date, which revitalizes region features with bidirectional region-word alignment reg- ularization. Region features that focus salient areas remove spatial visual redundancy, which enables our VI.P method to be extremely time-saving. The region-word alignment constraint builds up fine-grainee connection between video and language. Experimental results on downstream tasks and ablation studies prove the efficiency and effectiveness our method.
Limitation. In current implementation, we need a pre-trained detector to extract region features in advance. This step brings extra time consumption, which is not significant compared with saved time. However, we still acknowledge it is limitation. A feasible extension is to knowledge distillation to enable the model learn region features an end-to-end manner. Despite the limitation, method still provides an efficient paradigm to democratize VL. for researchers.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nayyer Aafaq. Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. Spatio-temport dynamics and semantic attribute enriched visual encoding for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12487-12496, 2019.
Ameen Ali, Idan Schwartz, Tamir Hazan, and Lior Wolf. Video and text matching with conditioned embeddings. In CVPR, pp. 1565-1574, 2022.
Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. Noise estimation using density estimation for self-supervised multimodal learning. pp. 6644-6631 2021.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-u up and top-down attention for image captioning and visual question answer- ing. In CVPR, pp. 6077-6086, 2018.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. ICCV, pp. 1728-1738, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05005 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-
scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, pp. 10638-10647, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. ECCV, pp. 104-120, 2020b.
Jacob Devlin, Ming-Wec Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1801v01810.044055, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. Mdmmt: Mul- tidomain multimodal transformer for video retrieval. In CVPR. pp. 3354-3363, 2021.
Robert Egly, Jon Driver, and Robert D Rafal. Shifting visual attention between objects and locations: evidence from normal and parietal lesion subjects. pp. 161, 1994.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint 17X.v1707.006122 2017.
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Hetero- geneous memory enhanced multimodal attention model for video question answering. In CVPR, pp. 1999-2007, 2019.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. ICCV, pp. 6202-6211 6211, 2019.
Tsu-Jui Fu, Linjie Li, Kevin Lin Zhe Gan, William Yang Wang, Lijuan Wang. and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. In arXiv:2/11. 1268, 2021.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. ECCV, pp. 214-229, 2020.
Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co- co-memory networks for video question answering. In CVPR, pp. 6576-6585, 2018.
10

---

Under review as a conference paper at ICLR 2023
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, pp. 6555, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. ICSAPP, pp. 131-135, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, pp. 4700 2017.
Yunseok Jang, Yale Song, Youngjac Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR, pp. 2758-2766, 2017.
Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. Divide and conquer: Question- guided spatio-temporal contextual attention for video question answering. In AAAI, pp. 11101- 11108, 2020.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv: 14Xiv::4/1.2533 2014.
Bruno Korbar, Fabio Petroni, Rohit Girdhar, and Lorenzo Torresani. Video understanding as ma- chine translation. arXiv preprint (xXiv:2006.0703:. 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, pp. 32-73, 2017.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In CVPR, pp.
2020.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pp. 201-216, 2018
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, pp. 7331-7341, 2021.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training. In EMNLP. 2020a.
Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiaogang Wang. Identity-aware textual-visual matching with latent co-attention. In CVPR, pp. 1890-1899, 2017.
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond mns: Positional self-attention with co-attention for video question answer- ing. In AAAL pp.
2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pp. 121-137, 2020b.
Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzuu Zhang, Bin Wang, and Yongdong Zhang. Focus your attention: A bidirectional focal attention network for image-text matching. In ACMMM, pp. 3-11, 2019a
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. BMVC, 2019b.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal under- standing and generation. arXiv preprint 2002v:2002.06333. 2020.

---

Under review as a conference paper at ICLR 2023
Antoine Miech, Ivan Laptev, and Josef Sivie. Learning text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.02516, 2018.
Antoine Miech, lean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zis- serman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, pp. 9879-9889, 2020.
Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. Learning joint embedding with multimodal cues for cross- cross-modal video-text retrieval. In ICMR, pp. 19-27, 2018.
Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10870-10879, 2020.
Mandela Patrick, Po- Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao F. Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2021.
Andrew Rouditchenko, Angie Boggust, David Harwath, Samuel Thomas, Hilde Kuehne, Brian Chen, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, et al. Cascaded mul- tilingual audio-visual learning from videos. Interspeech, pp. 3006-3010, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100 2019.
Brian J Scholl. Objects and attention: The state of the art. pp. 1-46, 2001.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: joint model for video and language representation learning. In ICCV, pp. 7464-7473, 2019.
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and
Kate Saenko. Translating videos to natural language using deep recurrent neural networks. arXiv preprint 141iv:1412.444 2014.
Jianyu Wang, Bingkun Bao, and Changsheng Xu. Dualvgr: A dual-visual graph reasoning unit for video question answering. 7MM, 2021a.
Wenzhe Wang, Mengdan Zhang, Runnan Chen, Guanyu Cai, Penghao Zhou, Pai Peng, Xiaowei Guo, Jian Wu, and Xing Sun. Dig into multi-modal cues for video retrieval with hierarchical alignment. In IJCAI, 2021b.
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. Camp: Cross-modal adaptive message passing for text-image retrieval. In ICCV, pp. 7764-5774 2019.
Hu Xu, Gargi Ghosh, PooYao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In ICCV, pp. 11562211552. 2021.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In ICLR, 2022.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, pp. 471-487, 2018.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park. Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In NIPS, 2021.
12

---

Under review as a conference paper at ICLR 2023
Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In ECCV, pp. 374-390 $74-390, 2018.
Junchao Zhang and Yuxin Peng. Object-aware aggregation with bidirectional temporal graph for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8327-8336, 2019.
Qi Zheng, Chaoyue Wang. and Dacheng Tao. Syntax- Syntax-aware action targeting for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 13096-13105, 2020.
Linchao Zhu and Yi Yang. Acthert: Learning global-local video-text representations. In CVPR, pp- 8746-878 8755, 2020.
13

---

Under review as a conference paper at ICLR 2023
woman getting ready get in
CLS
Region WWodd Alignment
CLS
Attended
Attend to words
Sentence Feature
woman is
Byy.
standing by
Language Encoder
his window
Video Encoder
Object Detector
woman and man went out the car
Casine
Similarity
Similarity Sty,r
They got lost
driving their
cE
Figure 8: Overview of the region-word alignment. Given region features of video and word tokens of caption, we encode them via the video and language encoder respectively. Region-Wort Word Alignment builds up a fine-grained connection between each region feature and word feature.
A APPENDIX
In this appendix, first provide complete description of language-to-videe loss. Then, we give more details of downstream tasks and implementation details. To show the generation of our frame- work, we give more additional experimental results, including video-to-text retrieval, video question answering and visualization.
A.1 MORE DETAILS OF REGION- WORD ALIGNMENT.
In Fig.
we show the pipeline of region-word word alignment. region feature firstly attends to word features to get attention weights of each word, and then fuse these word features to an attended sentence feature according to the weights. Each region corresponds to a specific attended sentence feature. Then, compute the similarity between paired region features and sentence features. By summing up all these similarities, we get get the similarity between a video and sentence.
A.1.1 LANGUAGE-TO- VIDEO Loss.
We also illustrate the details of language-to-video loss. The language-to-video loss Claca caleu- lated in the similar way as Clocal Given the similarity at, between 1-th word and n-th region, the attended video vector with respect to 1-th word is calculated:
Σ ar.a] a1,
(10)
where 3 denotes the attended video feature. The similarity of j-th caption and i-th video is:
S
1
Σ Bi)
(11)
I
The language-to-video loss L is then given follows:
local
B
log
exp(Sas/o)
(12)
B
Σ exp(S,./o)
14

---

Under review as a conference paper at ICLR 2023
A.2 DETAILS OF DATASETS.
Pre-training, i) Web Vid2.5M (WebVid) consists of 2.5M video-language pairs collected from web. Web contains manually generated captions that are well-formed sentences and well-aligned with videos. ii) Google Conceptual Captions (CC3M) consists of 3.3M image-language pairs. Images and raw textual descriptions are harvested from the web following similar process to WebVid.
Text-to- Videoo Retrieval. i) MSRVTT consists of 10K videos with 200K captions. We follow the previous works (Bain et al. 2021 Liu et al., 2019b) to train on 9K train and validation videos and test on the IK test set. ii) DiDeMo consists of TOK videos with 40K captions. We concatenate all sentences of video to single query previous works (Bain et al.) 2021 Lei et al., (02I). and we do not use the ground-truth localization annotations of this dataset. üi) LSMDC contains 128K videos. We follow the setting in (Bain al.) 2021 test our method on the 1K test set. iv) MSVD contains 1,970 videos. We split MSVD into 1,200, 100 and 670 videos as the train, validation and test set.
Video Question Answering. i) MSRVTT Multiple Choice is a question answering task that videos questions and captions are answer candidates. Each video has 5 captions, and only caption is the positive one. ii) MSRVTT-QA is based on MSRVTT dataset. 243K open-ended questions and 1500 answers are annotated. tii) MSVD-QA is based on MSVD dataset. It contains total number is an open-ended QA tasks based on TGIF (Jan al. 2017) dataset. TGIF (Jang et al. 2017) contains 165K QA pairs on 72K animated GIFs.
of 1,970 videos and 50,505 question-answer pairs. iv)
Video Captioning. i) MSRVTT Caption is video captioning task that requires the model to de- scribe the visual content of a given video in natural language. The dataset consists of 10K open- domain video clips. Each video clip has 20 ground-truth captions. We use the standard captioning split, which has 6.5K training videos and 2.9K testing videos.
A.3 IMPLEMENTATION DETAILS.
We implement our method with Pytorch and train all models Tesla V100 GPUs with a batch size of 128 per GPU, We use the Adam optimizer with different learning rate schedule on pre-training and downstream finetuning. For pre-training, rate of 1 x 10-5 The learning rate decays to 1/10 of the previous one at 30 and 40 epochs. For fine-tuning, all experiments on downstream tasks are trained in 10 epochs. We densely sample 8 frames for each video. The initial learning rate is 1 x 10-5 and decays to 1/10 at 2, 4 and 8 epochs. Detailed settings of each downstream task
train our model in 50 epochs using an initial learning
below.
Text-to- Video Retrieval. We sum up the video-sentence and region-word similarities as the final similarity between video and text. The final similarity used rank all video-language pairs.
The compared methods includes JSFusion (Yu et al. 2018), MEE (Miech et al. 2018). CE (Liu al. 2019b), MMT (Gabeur al. 2020), AVLNet (Rouditchenko et al. (021). ), Dig (Wang al.
2021b), Frozen (Bain et al., 2021). VTMCE (Ali et al.) (0022). MDMMT (Dzabraev et al. 2021)
ActBERT (Zhu & Yang 2020). SupportSet (Patrick et al. (021). TACo (Yang et al. 2021). Clip-
BERT (Lei et al.) (021). ). S2VT (Venugopalan et al.) 2014). FSE (Zhang et al. 2018). VSE (Kiros)
et al.
2014). VSE++ (Faghri et al. 2017), and MCues (Mithun et al.12018))
Multiple Choice. For MSRVTT Multiple Choice, we directly use the model finetuned for MSRVTT retrieval to evaluate the performance. Specifically, we regard videos as questions and regard captions as answers. Each video contains 5 captions, and only one matches the video. By using the model pre- trained for MSRVTT retrieval, we perform retrieval on each video and its corresponding 5 captions. The caption with the highest similarity is the predicted answer.
The compared methods includes JSFusion (Yu et al. 2018), ACtBERT (Zhu & Yang 2020), Clip- BERT (Lei et al. 2021), VideoCLIP (Xu et al. 2021), MERLOT (Zellers et al. 2021). VIOLET (Ful
et al. 2021).
Open-Ender Question Answering. For MSRVIT-QA, MSVD-QA and TGIF-FrameQA that open-ended question answering tasks, we adopt the framework in BUTD (Anderson et al.) 2018). which uses multi-label classifier based region features and text features of questions to perform
15

---

Under review as a conference paper at ICLR 2023
I.Ibb
Linear
Classification
Product
Linear
Linear
Softmax
Sum
Linear
Language Encoder
Video Encoder
Question
Video
Figure 9: Overview of the proposed method applied open-ended QA. We utilize BUTD Ander- son et al. 2018) head to fuse video and language features to perform open-ended QA tasks.
question answering task. As shown in Fig. features of [CLS] from language encoder and region features from video encoder are fused by the BUTD head and conduct classification to choose a proper answer.
The compared methods include Co-Mem Gao et al. SSML (Amrani et al.) 202T), HCRN (Le et al. 2020 DualVGR (Wang al. 2021a) Clip-
2018).
HMEMA
(Fan et al. 2019
BERT
LLei et al. 2021I.
Video Captioning. Following the Language Encoder and Video Encoder, we build Multi-oddality Transformer Encoder (BERT-based) to further fuse word embeddings and object features in video.
The training objective consists of parts: optimizing (a) the global and local alignment loss, and (b) masked language modeling loss (by predicting masked tokens). We finetune our pretrained model on MSRVTT Caption dataset with 10epochs.
The compared methods include SupportSet (Patrick et al. 2021 UniVL Luo et al. 2020
VideoAsMT (Korbar et al. 2020), OA-BTG ÇZhang & Pengl 2019). GRU-EVE (Aafqq al. 2019).
SAAT (Zheng et al. [2020), and STG-KD (Pan et al. 2020)_ We provide results using a diverse set of performance metrics, including BLEU4, METEOR, ROUGE-L and CIDEr.
A.4 SUPPLEMENTARY EXPERIMENTS.
To show the generalization of our method, we compare our method with the state-of-the-art re- sults on video-to-text retrieval. Then, we test it on additional downstream tasks: Video Question Answering, including MSRVTT multiple choice, MSRVTT Question Answering, MSVD Question Answering and FGIFF-rramQQA Finally, we pretrain our method with more data to show how it gen- cralize to a larger dataset and give some visualization results to show how region-word alignment is built.
A.4.1 VIDEO-TO-TEXT RETRIEVAL
Considering vision-language is bi-directional task, compare method with the same methods in Section 4.3 on video-to-text retrieval tasks, All results this section is based on the model that samples frame per video during pre-training and samples 8 frames per video during finetuning.
16

---

Under review as a conference paper at ICLR 2023
Video TTxx
Video
Text
Method
Method
R@1 1
R@5
R@ 10
MedR
R@1 1
R R@5
R@ 10
MedR
JSFusion
MMT
27.0
57.5
69.7
3.7
MEE
ActBERT
CE
SupportSet
28.5
58.6
71.6
3.0
MMT
28.6
20.0
AVLNet
AVLNet
TACo
Dig
14.3
33.7
43.6
15.5
ClipBERT
Frozen
Frozen
VTMCE
15.3
34.1
18.0
Ours
32.7
58.4
68.4
4.0
MDMMT
Zero-shot
Ours
23.8
43.6
54.1
8.0
SupportSet
8.7
23.0
31.1
31.0
Zero-shot
Frozen
Ours
16.0
29.3
35.5
40.5
Ours
27.1
45,8
54.5
7.5
(a) LSMDC retrieval
(b) MSRVTT retrieval
Video >Txt
Video
>Txt
Method
Method
R@D
R@5
R@ 10
MedR
R @
R@5
R 10
MedR
S2VT
13.2
33.6
15.0
VSE
15.8
30.2
41.4
14.0
FSE
13.1
33.9
12.0
VSE++
21.2
43.4
52.2
9.0
CE
22.5
52.3
5.0
MCues
31.5
51.0
61.5
5.0
ClipBERT
CE
Frozen
SupportSet
28.7
60.8
2.0
Ours
36.0
65.7
75.6
3.0
Frozen
Zero-shot
Ours
43.9
74.6
83.8
2.0
Frozen
Zero-shot
Ours
29.8
53.1
63.6
4.0
Ours
41.3
67.8
78.3
2.0
(c) DiDeMo retrieval
(d) MSVD retrieval
Table 8: Comparisons with state-of-the-art results on video-to-text retrieval.
Table summarizes the results on four benchmarks. Across all tasks, our method achieves sig- nificant improvement compared with previous methods in both finetuning and zero-shot settings. Considering our method also achieves an obvious advantage on text-to-video retrieval, the results prove that method builds up good alignment between video and language during pre-training and generalizes well on different datasets.
A.4.2
VIDEO QUESTION ANSWERING.
Method
MSRVTT
Method
MSRVTT
MSVD
JSFusion
83.4
Co-Mem
32.0
31.7
ActBERT
85.7
HMEMA
33.0
33.7
ClipBERT
88.2
SSML
35.0
35.1
VideoCLIP
92.1
HCRN
35.6
36.1
MERLOT
90.9
DualVGR
35.5
39.0
VIOLET 91.9
ClipBERT
37.4
Ours
92.4
Ours
38.3
39.5
(a) MSRVTT Multiple Choice
(b) MSRVTT QA and MSVD QA
Table 9: Comparisons with state-of-the-art results video question answering.
Table 9 summarizes the results on video question answering tasks. On MSRVTT MC, our method obtains accuracy of 92.4%, which outperforms prior -tate-of-the--at VideoCLIP (Xu et al.) 2021 by 0.5%. On MSRVTT-QA and MSVD-QA,
method achieves accuracy of 38.3% and 39.5%.
Compared to ClipBERT (Lei et al. 2021), our method brings 0.9% improvement on MSRVTT-QA. It also surpasses VGR (Wang al. 2021a) by 0.5 % on MSVD-QA.
17

---

Under review as a conference paper at ICLR 2023
Method
FGaF-FrameQA
ST-VQA Jang et al.
20017
49.3
Co-Mem Gao et al.
2018
51.5
PSAC Li ct al. 2019
55.7
HMEMA
(2019)
53.8
HCRN Le ct al. (2020)
55.9
QueST Jiang et al. (2020
59.7
ClipBERT Lei et al. (2021
60.3
Ours
60.6
Table 10: Comparisons with State-of-the results on TGIF-FramcQA.
A.4.3 RESULTS ON TGIF-FRAMEQA.
FrameQA requires model to highlight the fact that questions in this task can be answered according to video. The answer comes from a dictionary words type object, number, color and location.
Our method obtains accuracy of 60.6, which outperforms other methods. This experiment a complementary for video QA tasks. The results also verify that our method works well video QA tasks.
Results are shown in Table 10
A.4.4 REGION SELECTION.
To choose the most discriminative regions, we explore two selection strategies: i) Sorted Selection. Top 30 regions with the highest detection confidence are selected. ii) Track Object. We conduct object tracking form tracklet for each object. Only one region feature is reserved for each tracklet. We attempt to remove redundant objects that repeat in continuous frames to cover more object categories with this strategy.
To compare different region selection methods: i) Sorted Selection and ii) Track Object, we give their performance on MSRVTT retrieval. As shown in Table [I] Sorted Selection performs better than Track Object by 2.4 on R@! 1. Because Sorted Selection potentially choose multiple objects the same category and Track Object attempts cover more categories and only select one object category, the results This means that discriminative region features verify that are more important for VLP than covering more object categories.
A.4.5 TEMPORAL MODELLING.
Frozen utilizes TimeSformer (Bertasius et al., 2021) as the backbone, which proposes time- attention layer to model temporal information. In our work, rely temporal position em- beddings and location embeddings to model temporal information. MSRVTT retrieval. As shown in Table [2] time-attention layer cannot bring improvement to method. The results infer that, for VLP of retrieval, temporal position embeddings are enough to model temporal clues.
conduct ablation study on
Text > Video
Text + Video
Method
Method
R @
R 5
R@ 10
R@1
R@5
R@ 10
Track Object
33.6
59.2
69.7
Time attention
35.9
60.8
71.8
Sorted Selection
36.0
61.0
71.8
Temporal position
36.0
61.0
71.8
Table 11: Region selection.
Table 12: Temporal modeling.
A.4.6 RESULTS WITH MORE DATA
Furthermore, verify whether method can generalize larger datasets not, we evaluate our method pre-trained with more data on MSRVTT retrieval. Specifically, we add subset of Concep- tual 12M (Changpinyo et al. 1021) as pre-training data. The subset contains 7M samples that disjoint from CC3M, denoted as CC7M. As results shown in Table [3] CC7M can further improve the results of method. It verifies that our method can extend to massive data achieve better
18

---

Under review as a conference paper at ICLR 2023
Text Video
Data
Data Scale
@
R@5
R@ 10
WebVid
2.5M
27.6
50.4
61.7
WebVid+CC3M
5.8M
36.0
61.0
71.8
CC3M+CC7M
10.3M
38.7
63.8
73.9
WedVid+CC3M+CCMM
12.8M
39.6
64.8
73.8
Table 13: Results on MSRVTT retrieval with more data.
performance. An interesting observation is that adding data from original datasets (i.e., WebVid and CC3M) cannot improve the performance, while adding extra data from a different domain (i.e., CC7M) can. This observation inspires us that the diversity of data plays an important role in VLP.
A.4.7 VISUALIZATION
We also provide qualitative results on Fig. I0 to show the alignment between video and language. In Fig. T0a we visualize the attended regions with respect the most relevant word in the sentence. We observe that not only nouns are connected with corresponding regions, even verbs, such as "see" and "smile", are also aligned with relevant regions. Fig. [Ob] shows the attention heatmap between video frames and corresponding sentence. Regions that are relevant to the sentence description show higher attention weights than others. Visualization verifies that our proposed global-local alignment successfully connects video and language from coarse -grained to fine-grained level.
trainn arriess at station in fogay momning on Friday
waman glasses on train using digitl tablet, shot red epiz
Tolk artist performs Thursday
(a) Attended regions with respect the most relevant word in the corresponding sentence.
(b) Attention heatmap of video frames and corre- sponding sentence. Relevant regions are highlighted.
Figure 10: Visualization of the alignment between video and language.
A. A.4.8 RESULTS ON MSRVTT CAPTION.
MIRVIT
Method
BLEU4
METEOR
ROUGE-L
CIDEr
OA-BTG Zhane Peng 2019
41.4
28.2
46.9
(Aalay
SAAT (Zheli et al 2020
39.9
27.7
61.2
51.0
KDJ
SupponsSrr (Painck al 2021
18.9
28.2
59.8
48.6
VideoA (Korbar 20200
28.5
UniVI. (I.o at ar. 2021)
41.8
28.9
60.8
50.0
Table 14: Results of MSRVTT caption.
Results are shown in Table 14 Our method outperforms other methods in of METEOR and ROUGE-L metrics. On BLEU4 and CIDEr. our method's results are also close to the sfate-of-the-ant Note that different previous works of video captioning which rely on densely sampled raw frames as input, our model only requires the light- weight object features. Results show that our method still achieve satisfying performance with less computation cost.
19

---

Under review as a conference paper at ICLR 2023
A.5 CODE INSTRUCTION.
In this section, we give step-by-step instruction to illustrate how to run the code we provided in supplementary. We first describe how to set up the python environment and prepare the checkpoints to initialize video and text encoder. Then, we introduce how to prepare the pre-trainn vee-training and down- stream tasks, the tools to extract region features are also given. The commands to run the pre-training and downstream task, and explanation of config files are shown at the end.
Algorithm Code Instruction.
Requirements.
conda create -n demovlp pythen=3.8
source activate demovlp
pip install -r requirements
Download Pre-trained Model.
mkdir paetrdined
cd pretrained wget -0 httpg///e/e///ttooooo//aaaaaaaaa
mkdir distilbert- lttpiiiiiiiiiiiieiee
ciit_base_pl__ wttpppp62222------202222 pth
Download all files from huggingface distilbert-base-unnaaed and put them into
Pre-train Datasets.
WebVid: Refer to WehVid
CC3M: Refer to Conceptual Captions Website
Downstream Datasets.
MSRVTT Retrieval Dataset:
wget -c
maabainn/rozeer
zip
Extract Region Features.
To save time and storage consumption, uniformly sample frames for each video in WebVid and extract region features for each frame. There two ways to get region features.
We provide hyper-link (will release after acceptance) to download.
Refer to the scripts in bottom-up-attention
Pre-train.
Specify data_dir and object .di configgs/tt/o2t-----oooo wellocclll-llecttc -lss
cc. json to directories that contain raw videos and region features.
python -m torch.distr:: iit.tiittseeddddeooooooo
distri buted. launch -mproc_per_node
8 --master.port 2912 train.dist.multi.ppp --config
configs/pt/oo2t-00 -ew.oooleeee--osseese-e-ee
-sc 30 40
Downstream Task.
Specify data_dir and object_dir in configs/ http://tt///vttt.ttteeee
json
ec. json directories that contain raw videos and region features.
Specify oad checkpoint nt to the pre-trained checkpoint file.
python -m torch. distributed, launch --nproc.per.node
2 --master.port 2912 train.dist.mull
--config Daafiigs/ttmmsvttt.222 json -sc
Explanation of Config File.
We explain several important terms in the config files for easy run the code.
data_dir: the directory that contains original videos.
object_dir: the directory that contains region features.
object_nn the number of regions extracted from per frame.
num frames: the number of frames sampling from per video.
load.checkpoint: the checkpoint file of pre-training model.
20


Reviews:
[{"review_id": "bomwqnxd0FK", "reviewer": "Reviewer_ymrc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new video-language pre-training (VLP) method which is computationally efficient. The key idea is to reduce (1) temporal visual redundancy by frame sampling and (2) spatial visual redundancy by using region features that are extracted by a pre-learned object detector. To align region features and text tokens, this paper also proposes a bidirectional region-word alignment regulation. The efficiency and performance of the proposed method is empirically proved. \n", "summary_of_the_review": "The method proposed in this work demonstrates significant performance improvement in retrieval tasks. However, the technical novelty of this paper is marginal.\n", "strength_and_weaknesses": "Strength:\n1. VLP requires a huge amount of computational resources, which is not affordable by a lot of researchers. Therefore, it is important to reduce the training time without sacrificing the model performance.\n2. The performance is good.\n3. The paper is well-organized and easy to follow.\n4. Thanks for proving the code which can help readers to reproduce the results.\n\nWeakness:\n1. On page 5, what’s the motivation of this Region-Word Alignment design? Why is it better than existing local alignment methods, such as FILIP? There are lots of work that utilize optical transport (OT) to align regions and words (e.g., UNITER: UNiversal Image-TExt Representation Learning), what’s the advantage of the proposed Region-Word Alignment compared to OT? \n2. To reduce the pre-training time, this paper proposes two strategies, i.e., (M1) reduce temporal visual redundancy by frame sampling and (M2) reduce spatial visual redundancy by using region features. M1 is widely used in exiting VLP methods (e.g., Align and Prompt). In terms of M2, there are some works (e.g., MAE and VideoMAE) that mask 90%-95% patches/tubes/cubes in image/frame/video to reduce the spatial visual redundancy, what’s the advantage of using region features from a pre-trained object detector? Of course, the pre-trained object detector can be used multiple times.\n3. This paper uses a well-trained Faster-RCNN to extract region features then uses ViT to learn interactions between region features. Is that fair to compare with other methods which only use ViT as the video encoder?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KB_L-ipAwyb", "reviewer": "Reviewer_yXUD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for video-language retrieval. The method is based on pretraining a general still image detector and feeding the pooled output of the detector plus position+time features into a transformer to produce a video embedding. A global loss on cls video embedding, as well as local individual token losses (reminiscent of ColBert) are used to train the model. The approach is benchmarked on several datasets and exhibited better performance compared to the baselines and used less pretraining time (though that probably does not include the detector pretraining).", "summary_of_the_review": "It does not seem like the paper offers a big breakthrough in terms of ideas, but the focus of the paper is pretty clear, so novelty does not seem be an issue.\nI think for democratization of the video retrieval the major ingredient is the reproducible code that is easy to run. The submission does have a code associated with it, but it lacks instructions on how to run it, which creates artificial difficulties for a review. \nThe paper also has issues with presentation discussed in the weaknesses, so overall I vote for slightly below the acceptance threshold.\n", "strength_and_weaknesses": "Strengths:\n1) The approach allows getting a higher performance compared to the baselines. Pretrained detector allows to directly inject information from the labeled detection datasets, plus word alignment helps improve the model convergence.\n2) The general idea and solution are presented quite clearly in the paper.\n\nWeaknesses: \n1) No instructions on how to run the code and reproduce the results. This is critical as the paper claims \"democratization\" of the video retrieval. \n2) The paper writing might be improved. There are several references to the equation (10) in the paper, while it is actually in the supplementary, which does not make much sense. The tables 4,5,6 are not clear, particularly what is \"base\". The section 3.3. is hard to follow, the equations lack a word explanation of why they intend to do, there is also no corresponding illustration for those.\n3) The detector network is old and not trained end-to-end. It is mentioned in the text there are no bounding-boxes provided for training, but there is also an option to just backpropogate the final training loss back to the backbone on some architectures.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0jGVG_4sfAj", "reviewer": "Reviewer_nWq8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an efficient video-language pre-training method. The key idea is to employ region-text pairs during the pre-training. Different from recent works that use end-to-end transformers, the authors revisit the object region features extracted from pre-trained object detector. The idea is somehow similar to OSCAR, but extending such use of region-text pairs for video-language problem. Experimental results show that their method achieves better performance on downstream image-text retrieval.", "summary_of_the_review": "Overall, the proposed method achieves nice improvements compared to previous SOTA on image-text retrieval. This shows that prior end-to-end methods have a lot more spaces to improve. However, the proposed method seems very expensive in terms of feature extraction and feature storage. It may have difficulty to scale. In addition, as a pre-training paper, it would be good to verify their method on multiple downstream tasks. Now, the paper only considers retrieval, which is a bit narrow. ", "strength_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to understand. The presentation is clear.\n2. The authors proposed an efficient pre-training approach for video-language tasks. Some discussions regarding video temporal and spatial redundancy are interesting.\n3. The authors provide thorough experiments and show that their pre-training is faster than prior works.\n4. The pre-training objective, including video-sentence alignment and region-word alignment, is technically valid.\n\nWeaknesses:\n1. The proposed method heavily relies on pre-trained object detector. It seems very challenging and not affordable to extract/store all the region features, especially for large-scale video datasets. Such solution seems not appealing as the object detectors are keep improving. The region features need to be re-extracted if newer object detector is available. The feature extraction is very time consuming. Feature storage is very expensive. All video files need to be stored in the system for feature extraction if new object detector is available.  \n2. The proposed method use mixture of video-text and image-text data for pre-training. But it is unclear what is the pre-training objective for the image-text inputs. I wonder if the proposed pre-training mainly learns from image-text inputs or video-text inputs. \n3. The authors only conduct experiments on retrieval tasks. It is unclear if the pre-training is still useful when it comes to video QA and video captioning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject