Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
Authors: Yuchong Sun
Hongwei Xue
Ruihua Song
Bei Liu
Huan Yang
Jianlong Fu
Venue: NeurIPS 2022 Accept (2022)
Keywords: video-language pre-training

Abstract: Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pretraining mainly focus on short-form videos (i.e., within 30 seconds) and sentences, leaving long-form video-language pre-training rarely explored. Directly learning representation from long-form videos and language may benefit many long-form
video-language understanding tasks. However, it is challenging due to the difficulty of modeling long-range relationships and the heavy computational burden caused by more frames. In this paper, we introduce a Long-Form VIdeo-LAnguage pre-training model (LF-VILA) and train it on a large-scale long-form video and paragraph dataset constructed from an existing public dataset. To effectively capture
the rich temporal dynamics and to better align video and language in an efficient end-to-end manner, we introduce two novel designs in our LF-VILA model. We first propose a Multimodal Temporal Contrastive (MTC) loss to learn the temporal relation across different modalities by encouraging fine-grained alignment between long-form videos and paragraphs. Second, we propose a Hierarchical Temporal Window Attention (HTWA) mechanism to effectively capture long-range dependency while reducing computational cost in Transformer. We fine-tune the pre-trained LF-VILA model on seven downstream long-form video-language understanding tasks of paragraph-to-video retrieval and long-form video question-answering, and achieve new state-of-the-art performances. Specifically, our model achieves 16.1% relative improvement on ActivityNet paragraph-to-video retrieval task and 2.4% on How2QA task, respectively. We release our code, dataset, and pre-trained models at https://github.com/microsoft/XPretrain.


Long-Form Video-Language Pre-ningning with Multimodal Temporal Contrastive Learning
Yuchong Sun'; Hongwei R Ruihua Song', f Bei Liu Huan Yang', Jianlong Fu Renmin University of China, Beijing, China, Universityy of Science and Technology of China, Hefei, China,
Microsoft Research, Beijing, China, (ycsun, rsong] (song)@ruc... edu. en, fghh511200ma. ustc. edu, en, [bei. 1iu, huayan, jianf) (aanf)@miiii @mi wwwiicrsof..oom
Abstract
Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pre- training mainly focus short-form videos (i.e., within 30 seconds) and sentences, leaving long-form video-language pre-training rarely explored. Directly learning representation from long-form videos and language may benefit many long-form video-language understanding tasks. However, is challenging due to the difficulty of modeling long-range relationships and the heavy computational burden caused by more frames. In this paper, we introduce a Long-Forn Forg VIdeo-LA LAnguage pre-training model (LF-VILA) and train it on large-scale long-form video and paragraph dataset constructed from an existing public dataset. To effectively capture the rich temporal dynamics and to better align video and language in efficient end-to-end manner, we introduce two novel designs in our LF-VILA model. We first propose Multimodal Temporal Contrastive (MTC) loss learn the tem- poral relation across different modalities by encouraging fine-grained alignment between long-form videos and paragraphs. Second, we propose a Hierarchical Temporal Window Attention (HTWA) mechanism to effectively capture long-range dependency while reducing computational cost in Transformer. We fine-tune the pre-trained LF-VII model on seven downstream long-form video-language retrieval and long-form video question-
understanding tasks of paragraph-to-vine answering, and achieve new state-of-the-art performances. Specifically, our model achieves 16. 1% relative improvement on AetivityNet paragraph-to-ve task and 2.4% on How2QA task, respectively. We release code, dataset, and pre-trained models at https::t//// httee:///iitie /ggtttttttt//gg///g com/ mi wtwsotttttooooooooooooooccctottt wwt.oort//etetrraaiiiii
retrieval
Introduction
In recent years, research on video understanding has attracted extensive attention due to the huge amount of videos available everywhere in daily life. Previous research works video under- standing [14, 15, 42, 46, 60] mainly focus on short-form video (i.e., < 30 seconds) analysis and the semantics are limited to certain types (e.g., actions, scenes). However, there are so many long-form videos (i.e., > 30 seconds) [50] in real scenarios. Human annotated labels (e.g., actions) are difficult cover the rich semantic and dynamic information contained in those videos. On the other hand, the video-language pre-training paradigm provides a way to learn cross-modal representation from
'This work was performed when Yuchong Sun and Hongwei were visiting Microsoft Research as research interns.
Ruihua Song and Bei Liu the corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

One morning had Heron join us out of the blue, fish were constantly visible in the water and there's even a
Later in the evening, would be treated to sunset right front us and for the most part. all felt like very...
In the living room there are the amenities you would hope for. Moving into bathroom, they have really nice
The room itself was of a very high quality with lots of space for people.
0'00"
0'11"
0'28"
0'52"
Figure 1: An example of long-form video-paragraph pair with several clips and sentences. It contains a complicated storyline and a rich temporal dynamic. Each sentence can only describe a short clip, and understanding the whole video needs the ability of long-range spatial-temporal reasoning.
video and language pairs and shows promising results various high-level video understanding tasks joint with language [3, 27, 53, 58]. However, these studies mainly focus on short-form videos. explore directly exploiting long-form video and language pairs for pre- pre-training to
In this paper, benefit a wide range of long-form video-language understanding tasks.
Although long-fom long-form video-language joint learning has been explored in downstream tasks [16, 27, 28, 30, 59, 61], they either use pre-extracted video features which lead to the sub-optimal problem, or utilize image encoder to extract frame features that fail to model the long-range dependency in long-form videos. Recent works [3, 5, 33] have shown that a video Transformer [47] backbone helps capture long-range dependency in end-to-end fashion. An intuitive way for long-form video-language pre-training is to adopt a video Transformer based short-form aadeo-language pre- training model [3, 53] with long-form data. However, there are two main challenges in such design. First, long-form videos often contain more complicated storylines and richer temporal dynamics as shown in Fig. 1. Simply aligning video and paragraph using vanilla contrastive loss like previous models [3, 53] will ignore the temporal relation between clips and sentences, thus hindering the quality of learned representation. Second, feeding more frames in Transformer based video encoder will largely increase computational cost considering the self-attention operation.
To overcome the above challenges, we propose Long- Form "Ideo-LAnguag" pre- vre-trannin model (LF-VILA) with two novel designs. First, to better align long-form video-language pairs and learn temporal relationship between visual and language modalities, we propose Multimodal Temporal Contrastive (MTC) loss that learns temporal alignment between video clips and single sentences. MTC encourages similarity between two modalities be consistent with their temporal relationship. In other words, the embedding distance between a video clip and a sentence closer in time should be smaller than its distance with sentences that are far in time. Combining with global alignment between video and paragraph, MTC ensures the model capture the temporal relation between video clips and single sentences and further helps to improve the quality of joint representation.
Second, to utilize the advantage of the Transformer for capturing long-range dependency while efficiently processing more frames for end-to-end training, we propose Hierarchical Temporal Window Attention (HTWA) mechanism. As shown in Fig. 1, the frames sparsely sampled from a long-form video have large spatial and motion gaps, thus directly computing self-attention all frames in all layers of the Transformer is inefficient and unnecessary. Instead, we only learn the attention between adjacent frames in the first few layers that focus more details of spatial and temporal information. Then we gradually expand the window size in the following layers, where the high-level representation enables the model to better capture the relation between frames far apart. The computational cost is largely reduced with the proposed HTWA mechanism.
We conduct experiments and evaluate LF-VILA on seven downstream long-form video-language understanding tasks of paragraph-to-videe retrieval and long-form video question-answeringg We surpass the ffate-of-the-art models pre-trained short videos by large margin. Our results demonstrate the benefit of modeling long-range dependency for long-form videos. We also verify the effectiveness of our proposed MTC loss and HTWA mechanism through ablation studies.

---

Our contributions are summarized as follows: (1) We are the first to study end-to-end long-form video- language pre-training with large-scale video-paragraph data benefit long-form video understanding. (2) We propose an MTC loss to capture the temporal relationship between clips and sentences while improving the joint representation of long-form video and language. (3) We design an HTWA mechanism for the video Transformer backbone, which can capture the long-range dependency in long-form videos effectively and efficiently. (4) We verify the effectiveness of our LF-VILA model on wide range of downstream long-form video-language understanding tasks. Our model achieves late-of-the-arr performance on four paragraph-to-videe retrieval tasks and three long-form video question question-answering tasks.
2
Related Work
2.1 Video Representation
Most previous video encoders utilize 3D-CNN based backbones [7, 46, 51]. These models show promising performance on short-form video understanding tasks, such as action classification and detection [7, 17]. However, CNN has limited receptive field and cannot effectively capture long- range dependency. Recent works have extended Vision Transformer [13] for video representation and demonstrated the benefit of long-range temporal learning [5, 33J. To reduce the computational cost, TimeSformer [5] introduces factorized spacetime attention, while Video Swin- Tran-forffrrrer [32] restricts self-attention
local 3D window. However, TimeSformer [5] is still computationally expensive when the number of input frames becomes large. Video Swin-Transformer [33] adopts a fix-sized temporal window which not suitable for videos with large duration. We propose hierarchical temporal window attention to effectively learn the long-range dependency in long-form videos while reducing the computational cost.
2.2 Long-form Video Understanding
Long-form video understanding is less explored in previous studies. Some works use long-term context for improving recognition performance [41, 491. Typical long-form video understanding tasks contain shot or event boundary detection [4] and temporal action detection [6]. but these tasks cannot reveal the ability of a high-level understanding of the model. Jointly understanding long-form videos with language is a way to discover the rich semantics contained in videos and many benchmarks are proposed recently, such as paragraph-to-vine
retrieval [1, 2, 23, 37] and long-form video question- answering [27, 28, 30, 57]. Previous works that explore these tasks mostly pre-extracted features, which hinder the performance because sub-optimal fcatures [16, 27, 59, 61]. We study end-to-end long-form video-language pre-training and transfer to long-form video-language understanding tasks.
2.3 Video-Languag Pre-training
Inspired by the success of image-language pre-training [10, 19, 20, 21, 22, 29, 54],, video-language pre-training is also explored recently. However, these works mainly focus on short-form videos [3, 34, 36, 53]. Some works use 3D-CNN as a video backbone [34, 36]. To utilize the advancement of Transformer, some works sparsely sampled frames to reduce the computation requirements [3, 53]. One key factor for learning good representation is using contrastive loss to align multi-modal features [3, 34, 36, 53, 58]. We further design a multimodal temporal contrastive loss conduct fine- grained alignment between long-form videos and paragraphs. The power of the pre-training model the amount of training data, some works built large-scale video-language datasets [36, 53, 58], and we build a long-frop from video-paragraph dataset based on HD-V VOLV-IL--OOO [53]. There are several works have explored long-form video-language pre-training, HERO [27] uses pre-extracted features, while MELORT [58] uses an image encoder to separately encode frames which ignores joint spatial-temporal representation. Different from them, use video Transformer backbone and end-to-end pre-training on large-scale long-form video- aadoo-pargg datasets.
is largely dependent
3 Approach
In this section, we first show the overall architecture of the proposed Long-Form Iideo-LAnguage pre-training model (LF-VILA) in Sec. 3.1. Then explain proposed Multimodal Temporal
3

---

(a) Long-form Video-Language Pre-training
kmi
4pmm
Cross-modal Encdder
yyoobal
Encoder
Video Encoder
Jente
- OD
One
moming Heres
aeer
Eving
room very
svering
there
us
righ
you would.
(b) Multimodal Temporal Contrastive Loss
Multimodal
Paragraph:
O - -
MTC d dis(O
Video: A - - -
Figure 2: The framework of (a) Long-Form VIdeo- Ladgo-LAnguage pre-training model (LF-VILA) and illustration of (b) Multimodal Temporal Contrastive (MTC) learning. (a) LF-VILA consists of a text encoder, a video encoder, and a cross-modal encoder. the text encoder, attention computing is first within each sentence then the whole paragraph. The pink boxes in the video encoder illustrate the proposed Hierarchical Temporal Window Attention (HTWA) mechanism. (b) the MTC loss aligns two sequences of representations (e.g., clip and sentence representations in case), the distance of two element's representations smaller when they are closer in time.
Contrastive (MTC) loss for learning cross-modal temporal relationship in Sec. 3.2, followed by the designed Hierarchical Temporal Window Attention (HTWA) mechanism for efficient video encoder in Sec. 3.3. Finally, we introduce the pre-training pipeline with target pre-training tasks in Sec. 3.4.
3.1 Model Architecture
As illustrated in Fig. 2, proposed Long-Form Video-LAnguage pre-training model (LF-VILA) consists of three parts: a video encoder Ev, a text encoder Ey and a cross-modal encoder Ec. With video and paragraph as input, first pass them the video encoder Ey and the text encoder Er for embedding learning, respectively. Then we concatenate visual and language embeddings as input the cross-modal encoder Ec. where further cross-modal joint learning is conducted. The details of these encoders are as follows.
Text Encoder. sentence-level and paragraph-level encoding. In the first several layers, self-attention is conducted within word tokens from the same sentence, and sentence embedding can be learned individually. In higher layers, we add segment embedding to distinguish each sentence, and the attention computation extended to all word tokens of the paragraph to output paragraph representation.
The text encoder Ë7 is based Transformer network [47]. We divide it into two parts:
Video Encoder. Our video encoder is also stacked by Transformer layers. particular, we design a Hierarchical Temporal Window Attention (HTWA) mechanism for efficient attention computation. Given a long-form video that has M clips, we sample N frames from each clip and divide each raw frame into H X W patches. Then the M X N x И x W patches encoded by Ev. With our designed HTWA mechanism, the temporal window is gradually expanded, so that we can get hierarchical feature maps with different temporal receptive fields. In addition, video features with the same temporal window size as clip frame number in the middle layer can be utilized as the clip representation for fine- mne-graieed alignment with sentences.
Cross-modal Encoder. language embeddings from the output of Ey and Er are concatenated as the input to Ec. Self- attention used to capture the joint relation between visual and language modalities. Eeg outputs the representation of [CLS] token and each textual and visual token.
The cross-modal encoder EG consists of Transformer layers. Visual and

---

3.2 Multimodal Temporal Contrastive Learning
Contrastive learning is widely used in previous multimodal pre-training works to align different modalities such as image-language and video-language. The goal of this loss is to pull the repre- sentation of matched pairs close to each other and push unmatched pairs away from each other. However, when aligning long-form videos and paragraphs, the vanilla contrastive loss neglects the temporal relation between clips and sentences, which is important for capturing the complex temporal dynamics in long-form videos. To better learn the temporal relationship between different modalities, we propose Multimodal Temporal Contrastive (MTC) loss, which can be applied align sequences in different modalities such as video and paragraph.
We assume that the distance of two elements' representations in different modalities should be consistent with their temporal distance. Specifically in our model. MTC loss encourages the video clip embedding be more similar to its neighbor sentence embedding than sentences of long distance in the same paragraph. For example in Fig. 2, given two sequences of representation from the i-th sample, we first sample an anchor set A of k representations from u; then we sample set of representations K. from 44. For each uf in
V {v] ....... 1/ and 4 = {L) (4, ...,,,",,,"" 1,,,,
A, we treat ! as positive, where Ip - jp-atl= = min(Ip - ql), ! E 3. We also randomly sample some representations from i * as A which is used to stable the training. Then the MTC loss is calculated by applying an InfoNCE loss:
exp
1 Σ log ofEAA
Cmeclvi, ti) =
Egex exp (s() + Enjer exp (s()
where f2) f2/T, T is the temperature.
We obtain clip representations of e; and sentence representations t; from the output of the first part of the video encoder and text encoder, respectively. Then the MTC loss can be obtained by:
B
Cyste
1 B
Σ (ur 44), C
1
Σ
Lmte (li, Us),
the overall MTC loss finte is the average of L and Li2e
3.3 Hierarchical Temporal Window Attention
Directly feeding all sampled frames of a long-form video to a vanilla Transformer network [47] for global self-attention learning will heavily increase the computational cost. The cost is quadratic with respect to the number of frames. One possible way is to apply a small fixed temporal window. However, such design will neglect to learn the relationship between different clips of one video, which is essential for long-form video understanding. To capture long-range dependency in long-form videos efficiently and effectively, we propose Hierarchical Temporal Window Attention (HTWA) mechanism by gradually increasing temporal window size in Transformer layers.
A temporal window is used to restrict attention computing between frames in the temporal dimension. Given 3-D input tensor T' H' x W' patches, where T' the number time steps and H' W' denotes the number of spatial patches. We divide the input tensor to Mr windows along T" dimension, where denotes the Transformer layer. Then multi-heal attention is applied within each window, and the output is the combination of all attention windows as follows:
a AH , (1,2, Mi),
(3)
Concal (a1, aa,..., aM,),
where z; is the token embeddings belonging to i-th window, M A multi-head attention, a; the attention weights of i-th window, and a is the attention weights with the shape of T' x H' x W".
Compared with short-form videos, long-form videos have two distinguishing characteristics that we should consider for better understanding. First, there are large motion gaps (dynamics) between
'ince we consider M = 4 in this work, here A and IC are randomly sampled. When we extend this loss for much longer sequences, need to restrict the maximum distance between positive pairs.
5

---

frames that are sparsely sampled. Second, it is important to build long-range relationships to understand the whole video. Considering the above properties, we start build connections between adjacent frames within small window sizes (e.g., 2) in the first few layers. Then the temporal window size gradually increased to capture longer-range dependency as the semanties learned become high-level. Specifically, we use the temporal window size equal to the number of frames in video in the last several layers, so that the full context can be attended to learn the global relationship.
3.4 Pre-training Pipeline
We adopt two-stage pre-training as previous works did [53] since modal podaal-nddeendent design enables to learn powerful single-modality embedding for downstream tasks. In the first stage, the video encoder and text encoder are learned independently with a video-text alignment task. In the second stage, text embedding and video embedding are concatenated as input to the cross-model encoder for joint representation learning. We adopt Masked Language Modeling (MLM) and Video- Matching (VTM) which are widely used as pre-training tasks for learning cross- cross- interaction.
Video-Text Alignment. Specifically, we first train the text-encoder and video-encoder using con- trastive loss align textual and visual representations. In addition the proposed multimodal temporal contrastive loss, we also adopt a standard contrastive loss on the global representation of long-form video and paragraph. The global contrastive loss is calculated as:
(4) where V; and T; the representation of i-th video and paragraph in the batch, respectively. The global alignment loss Lglobal is the average of globat and L global
2 global
1
log Σ exp (V, T2))*
exp (s(V,, T())
L. global
1
E log
exp (s(T,, Vi)
B
B
Σ exp (s(Ti, Vj))
The combination of MTC loss and global contrastive loss is used as the pre-training objective for the first stage:
Estagel Lggobal X Amter
(5)
where a denotes the weight of MTC loss compared with global contrastive loss.
Masked Language Modeling. We follow the previous vision-language pre-training works [20, 26, 54] to mask word tokens and predict the ground-truth labels from the output of the cross-modal encoder, which integrates the context other textual tokens and visual tokens:
Liiim vy log (w; W3, V):
where W denotes the word tokens, V denotes the visual tokens, and w; denotes the masked token. We adopt the same masking strategy and prediction method as BERT [12].
Video-Texx Text Matching. To fuse the textual and visual information and generate a cross-modal representation, we use VTT one pre-training task. VTM predicts whether the input paragraph and video are matched. We randomly replace the aligned video with a sampled negative with a probability of 0.5. We use projection layer on the top of [CLS] embedding to predict two-class matching logits and then compute negative log likely-hood loss as the VTM loss:
Lytm - EEw,y log p (y W)
(7)
In the second stage, we freeze the video-encoder and text-encoder as [53] to accelerate training. The overall loss of stage two the combination of MLM and VTT:
Estage2 Lullm Xg
where X is the weight of VTM in consideration of MLM.
4 Experiments
In this section, we first introduce the pre-training details and then show experiments of utilizing the pre-trained model on downstream paragraph-to-vider retrieval and long-form video question- answering tasks to verify the effectiveness of our proposed LF-VILA. We also transfer LF-VILA for long-form video classification tasks to show the generalization power of the pre-trained model.

---

Table 1: Statistics of LF-VILA-8M and its comparison with existing video-language datasets.
Dataset
Domain
#Video Text Pairs
Avg. Len(sec)
Text Len
Duration(h)
DiDeMo [1]
Flickr
10K
28.0
29.2
87
QuerYD [37]
open
2K
278.0
243.8
200
ActivityNet Captions [23]
action
20K
180.0
48.3
849
Condensed Movie [2]
movie
34K
132.0
18.0
1.3K
WebVid-2.5M [3]
open
2.5M
18.0
12.0
13K
HowTo 100M [36]
instruction
136M
3.6
4.0
135K
HD-V DODVVIA--10O [53]
open
103M
13.4
32.5
372K
LF-VILA-8M
open
8.5M
100.2
307.9
236K
Table 2: Results of paragraph-to-video retrieval on ActivityNet Captions dataset [23].
Method
Pre-training Dataset
R@ 1 f
R@51 f
R@ 1
MedR +
HSE [59]
20.5
49.3
ClipBERT [25]
COCO 19), Visual Genome [24]
21.3
49.0
6.0
HD-VILA [53]
HD-VILA- 1000--00M [53] CC3M, WebVid-2.5MM
28.5
57.4
94.0
4.0
Frozen
28.8
60.9
3.0
Support Set [38]
HowTo100k 00M [36]
29.2
61.6
94.7
3.0
TACo [56]
Hlowoo000% [36]
30.4
61.2
93.4
3.0
LF-VILA (Ours)
LF-VILA-SM
35.3
65.4
95.0
3.0
4.1 Pre-training Details
Pre-training Dataset. To facilitate research on long-form video understanding, We build large-scale long-form video-paragraph dataset based on AID-VILA-I0MM [53], which is an existing large-scale video-language dataset with diverse categories. It contains 100 million clip-sentence pairs derived from 3.3 million YouTube videos. We keep continuous clips with at least 4 clips and construct a dataset with 8.5 million long-from videos and corresponding transcripts, namely LF- LF-VILA-8M. Table shows the statistics of LF-VILA-8), -MM and its comparison with existing video-language datasets. covers -60% of clips of the HD-VILA- 100M dataset. The average duration of each video 100.2 seconds and the average number of words in each paragraph is 307. 99. We provide additional statistics and examples in the supplementary material.
Implementation Details. During pre- tre-training, our model samples 4 consecutive clip-sentence pairs as input. We uniformly sample 8 frames from each clip and resize the frames to 192 320. we use the WordPiece tokenizer like BERT to split each sentence into tokens with max length of 50. For the video encoder, we use Trai--rranformer [32] as the backbone and integrate our proposed HTWA for frame sequence. Temporal window sizes are set to five stages: 2, 4, 8, 16, and 32, respectively. 5. the output feature is down-sampled by 64 times to 3 5. We adopt 12-layer Transformer network for the text encoder, with 8 layers for the first part and layers for second part. We also use 12-layer Transformer network for the cross-modal encoder. The weight of the video encoder is initialized with pre-trained on ImageNet-21IK We use the first 12 layers of BERT-Large to initialize the weight the text encoder, and the last 12 layers initialize the weight of the cross-modal encoder. We provide more detailed model specifications in the supplementary material.
We use 8 patches and use fixed spatial window
We use an AdamW optimizer with a learning rate of 5e-5 5 and warm up the learning rate for 1 epoch, followed by a linear decay, we use a weight decay of 0.05, We train our model with 32 NVIDIA Tesla V 100 GPUs. For stage one, we use batch size of 512 and train for 6 epochs. For stage two, we use a batch size of 1 1,536 and train for another 6 epochs. We use the model from stage one for retrieval tasks since two-stream architecture is efficient for retrieval and is widely used in previous works [39, 53]. Pre-trained model from stage two applied for video QA tasks. We excluded videos that overlap with downstream tasks from the training dataset using YouTube IDs.
7

---

Table 3: Results of paragraph-to-video retrieval two datasets. * denotes results by
implementation.
Result
DiDeMo dataset [1].
(b) Result QuerYD dataset [37].
Method
R@I 1 f
R@st
R@ 101
Method
R@ 1
R@5 f
R@ 10 f
FSE [59]
13.9
36.0
MOEE [35]
11.6
30.2
43,2
ClipBERT [25]
20.4
48.0
69.0
CE [31]
13.9
37.6
78.3
HD-VILA [53]
28.8
57.4
69.1
TeachText [11]
14.4
37.7
50.9
Frozen [3]
31.0
59.8
72.4
Frozen [3]
53.8
75.7
82.7
LF-VILA (Ours)
35.0
64.5
75.8
LF-VILA (Ours)
69.7
85.7
90.3
Table 4: Results of paragraph-to-videe retrieval on Condensed Movie dataset [2] from official
leaderboard https://competittt codall .rgg http://mmpttiiinnna. wtt.iiitoonnnn1///
Method
Geometric Mean f
R@ 1 f
R@5
R@ 10
MoEE [35]
5.88
1.94
7.84
13.38
TeachText [11]
23.15
12.08
27.40
37,45
LF- VILA (Ours)
26.40
13.56
32.47
41.79
4.2 Paragraph-to-Vider Videoo Retrieval
We conduct retrieval tasks four widely-used datasets for paragraph-to-viee viviip retrieval: ActivityNet Captions [23], DiDeMo [1], QuerYD [37] and Condensed Movie [2]. Details of each dataset and implementation in the supplementary material.
Results. Tab. 2- show the results VF-V on four paragraph-to-video retrieval datasets. For the most widely used ActivityNet Captions [23] dataset, surpass the SOTA model TACo [56] by 16.1% % on @@ TACo is pre-trained on How To100M [36] using pre-extracted feature. This demonstrates the benefit of end-to-end training and utilization of long-form video datasets. Compared to VD-VIIA [53] which is trained on 100M short-form video and sentence pairs, we use long-form videos with fewer data (-60% clips of R@I This shows the effectiveness of our model LF-VILA in learning better alignment for long-form video and language. Compared Frozen [3] which is pre-trained
10000) and achieve 23.9% improvement in terms of
suman-annotated dataset, achieve 22.6% % improvement on @@) using relatively noisy but easy-to-get data. For DiDeMo [1], we also observe significant improvement. over the SOTA model Frozen [3]. On Quer YD [37] and Condensed Movie [2], we outperform the previous best models Frozen [3] and TeachText [11] with a relative 29.6% and 12.3% improvement, respectively. These two datasets are challenging due to their long videos. These results show the value of long-form video-language pre- pre-training and our LF-VILA model can better understand the
particular, we obtain 12.9% improvement in
of R @@)
storyline and temporal relations in long videos.
4.3 Long-Form Video Question-Answering
We conduct video question-answeriig tasks on three widely-used datasets for long-form video understanding: ActivityNet-QA [57], How2QA [27] and VIOLIN [30]. Details each dataset and implementation are in the supplementary material.
Results. Tab. 5 shows the results of three long-form video question-answering tasks. For ActivityNet QA [57], we outperform most previous works except MELORT [58]. Note that VQA-T [55] and MERLOT [58] are specifically designed for video QA. VQA-T [55] uses automatically generates 69M video cnestion-answer triplets from narrated videos for training. MERLOT [58] utilizes 180M pairs of data and needs excessive computational cost for training (-30K TPU hours), while we only need ~5K GPU hours. For How2QA [27] and VIOLIN [30], we achieve the new SOTA. This illustrates the reasoning capability our model on long-form videos by better capturing the rongerange dependency between video clips and paragraphs.

---

Table 5: Results of video question-answeriin tasks. We gray out results for fair comparison
because of the use of large-scale video QA triplets or the huge computational cost for pre-training.
(a) AetivityNet QA [57].
(b) How2QA [27].
(c) VIOLIN [30].
Method
Acct
Method
Acct
Method
Acc1
MAR-VQA [61]
34.6
CLIP [39] by [28]
69.3
LXMERT [44]
66.3
CoMVT [40]
38.8
CLIP-SF [28]
72.9
Det-BERT [30]
67.8
VQA-T [55]
38.9
ResNet-SF [28]
74.3
GVE [8]
68.4
MERLOT [58]
41.4
HERO [27]
74.3
HERO [27]
68.6
LF-VILA (Ours)
39.9
LF-VILA (Ours)
76.1
LF-VILA (Ours)
70.9
Table 6: Results of Procedural Activities Classification the COIN [45] dataset. We gray out some results for fair comparison because of the huge computational cost for pre-training.
Model
Pre-training Dataset
#Training Samples
Domain
Acct
ClipBERT [25]
COCO [9]. Visual Genome [24]
5.6M
Open
65.4
MIL.NCEE [34]
HowTol 00M [36]
100M
Instructional
70.2
VideoCLIP [52]
HowTol 10M 100 [36]
100M
Instructional
72.5
SlowFast [14]
Kineties [7]
370K
Action
71.6
TimeSformer [5]
Kinetics [7]
370K
Action
83.5
TimeSformer [5]
HowTol00M [36]
100M
Instructional
85.3
TimeSformer [5]
[36]. wikillow [29]
100M
Instructional
88.9
LF-VILA (Ours)
LF-VILA-8M
8M
Open
85.7
4.4 Transfering to Long-form Video Classification
To demonstrate the improvement of LF-VILA for long-form video representation and broaden the tasks and domains of the evaluation, evaluate LF-VILA COIN [45] and LVU [50]. Details of each dataset and implementation are in the supplementary material.
Tab. 6 shows the result of Procedural Activities Classification COIN [45] dataset. Our model achieves strong performance on this task, although we use out-domain videos for pre-training and our computational cost is smaller than the SOTA method (-2.1K vs. ~7K GPU hours). We largely surpass other video-language pre- vee-training models (c.g.. ClipBERT [25], MIL-NCE [34] and VideoCLIP [52]). Tab. 7 shows the result of video classification SOTA methods largely, especially on Scene and Relation classification. Especially, we outperform the video-language pre- pre-training model VideoBERT [43] largely. The strong performances these two benchmarks show the generalization power of our pre-trained video encoder.
LVU [50] dataset. Our model surpasses the previous
4.5 Ablation Studies
To validate the effectiveness of LF-VILA with MTC loss and HTWA mechanism, we conduct ablation studies with subset of data to save resources. We randomly sample 1M iidoo-prragrah pairs from the whole LF-VILA-8N. 8M for pre-training.
(1) Using more frames is better for long-form video-language understanding. Using the same backbone, compare our model with pre-training with one clip-sentence pair for each sample as in previous models [3, 3J. As shown in Tab, 8, when only using 8 frames, the performance is poor. After increasing the number of frames to 32, there is large improvement. This also indicates that an efficient backbone to support more frames is essential for long-form video representation.
(2) Pre-tree trainirinng on long-form video-language data further improves the performance signif- icantly. When we use continuous clips for pre- training, there also significant performance gain as shown in Tab. 8. The gain is larger QuerYD dataset which consists of longer videos than ActivityNet dataset (278s vs 180s on average).
(3) MTC loss contributes to the performance. As shown in Tab. 8, when combined with the MTC loss, the performance is further improved by 1.7% on ActivityNet dataset and 1.9% on QuerYD dataset in terms of @@

---

Table 7: Results of Content Understanding on the LVU [50] dataset.
Model
Relation (Acc)t
Way of Speaking (Acc)t
Scene (Acc)t
1001-SloFFastNN.. [14, 48)
52.4
35.8
54.7
VideoBERT [43]
52.8
37.9
54.9
Object Transformer [S0]
53.1
39.4
56.9
LF-VILA (Ours)
61.5
41.3
68.0
Table 8: Analysis of the effectiveness of long-form pre-training.
Pretain
ActivityNet [23]
QuerYD [37]
Loss
#clips
#frames total
R@ If
R@5t
R@
R@ 11
R@5t
R@ 101
w/o Pre-training
15.0
40.2
85.8
19.8
46.4
58.4
8
19.3
45.5
87.1
38.4
67.5
78.2
8
20.3
47.0
87.7
41.6
73.5
82.1
Lylobal
32
24.3
53.5
92.1
75.0
83.4
4
32
26.1
56.7
92.7
55.4
79.3
85.5
globbai + fortee
32
27.8
58.3
92.8
57.3
80.9
85.8
(4) HTWA mechanism achieves better computational cost-performance tradeoff. In Tab. 9, we compare our methods with video backbone using fixed window long-range dependency. When apply small fixed window size (e.g..4), the performance is relatively poor. This indicates the limitation of a small attention window for modeling long-range dependency. When we increase the window size 32 cover whole video, there is almost improvement in performance, while the computational cost and training time increases significantly.
Table Analysis of temporal window (TW).
ActivityNet [23]
QuerYD [37]
Method
TW#
Time
R@11 11
R@5t
R@500
R@ It
R@51
R 101
4
0.91
25.1
54.9
91.8
52.8
76.3
84.5
8
0.96x
25.4
54.8
91.9
54.5
78.1
84.4
Fixed
16
1.10x
26.0
56.1
91.5
53.7
78.6
84.3
1.49x
26.2
56.4
92.3
55.6
79.3
85.7
HTWA
[2-32]
1.00x
26.1
56.7
92.7
55.4
79.3
85.5
5
Conclusion
In this paper, we study video-language pre-training on large-scale long-form aidoo-paragraph dataset. To better align long-from videos and paragraphs, we propose Multimodal Temporal Contrastive (MTC) loss to capture the rich temporal relation between different modalities. addition, we design Hierarchical Temporal Window Attention (HTWA) mechanism to be applied with an image Transformer. Our proposed Long-Form "Idgo-LAnguage pre-training model (LF-VILA) combined with MTC and HTWA can learn effective multi-mo multi-modal representation by capturing long- range dependency from long-fome videos efficiently. Experiments on seven long-form video-language understanding tasks verify the effectiveness of our model.
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: Fundamental Research Funds the Central Universities and the Research Funds of Renmin University of China (21XNLG28). Additional revenues related to this work: Internship at Microsoft Research Asia. This work also partially sponsored by Kuaishou Research Collaboration Initiative.
10

---

References
[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. In ICCV, pages 5803-5812, 2017.
[2] Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed movies: Story based retrieval with contextual embeddings. In ACCV, pages 460-479, 2020.
[3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1728-1738, 2021.
[4] Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. Shot and scene detection via hierarchi- cal clustering for re-using broadcast video. In CAIP, pages 801-811. Springer, 2015.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, pages 813-824, PMLR, 2021.
[6] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. ActivityNet:
A large-scale scale video benchmark for human activity understanding. In CVPR, pages 961-970, 2015.
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.
[8] Junwen Chen and Yu Kong. Explainable video entailment with grounded visual evidence. In ICCV, pages 2001-2010, 2021.
[9] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint
2015.
[10] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In ECCV, pages Springer, 2020.
[11] Ioana Croitoru, Simion- Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In ICCV, pages 1583-11593, 2021.
[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, pages 4171-4186, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICML, 2021.
[14] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6202-6211 1211,2212222 2019.
[15] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, pages 1933-1941, 2016.
[16] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox. COOT: Co- operative hierarchical transformer for video-text representation learning. In NeurIPS, pages 22605-22618 88 2020.
[17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pages 5842-5850, 2017.
[18] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[19] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vasion-language representation learning. In CVPR, pages 12976-1299 12955, 2021.
[20] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.008: 2020.
11

---

[21] Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. Wenlan: Bridging vision and language by large-scale multi-modal pre-training. arXiv preprint arXiv: arXiv:2103.06661 2021.
[22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904-4916. PMLR, 2021.
[23] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In ICCV, pages 706-715, 2017.
[24] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome; Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(():32-773 2017.
[25] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara I. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Cliphert for sadeo-and-language learning via sparse sampling. In CVPR, pages 7331-7341, 2021.
[26] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurlPS, 34:9694-999 9705449977,2000 2021.
[27] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hi- erarchical encoder for video+ language onni-representation rre-training In EMNLP, pages 2046-2063 2020.
[28] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, al. Value: A multi-task benchmark for video-and- language understanding evaluation. NeurlPS, 2021.
[29] Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. CVPR, pages 13853-13863, 2022.
[30] Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. Violin: A large-scale dataset for indo-aand-langual inference. In CVPR, pages 10900-10910, 2020.
[31] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv: 1907.19488. 2019.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[33] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
[34] Antoine Miech, lean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, pages 9879-9889, 2020.
[35] Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprini arXiv: arXiv:
2018.
[36] Antoine Micch. Dimitri Zhukov, Jean- DrannBaaptitte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol Learning text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2640- 2019.
[37] Andreea-Mante Oncescu, João Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP, pages 2265- 2269. IEEE, 2021.
[38] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2020.
12

---

[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI,, pages 8748-8763. PMIR, 2021.
[40] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. Look before you speak: Visually contextualized utterances. In CVPR, pages 16877-16887, 2021.
[41] Mykhailo Shvets, Wei Liu, and Alexander C Berg. Leveraging long-range temporal relationships between proposals for video object detection. In ICCV, pages 9756-9764 2019.
[42] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. NeurlPS, pages 568-576, 2014.
[43] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, pages 7464-7473, 2019.
[44] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP, pages 5100-510 10005111,2000 2019.
[45] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, pages 1207-1216, 2019.
[46] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, pages 5998-6008, 2017.
[48] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018
[49]
Chao-Yuaa Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, pages 2019.
[50] Chao- Yuao Wu and Philipp Krahenbuhl. Towards long-form video understanding, In CVPR, pages 1894, 2021.
[51] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning: Speed-accuracy trade-off's in video classification. In ECCV, pages 305-321, 2018.
[52] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. In EMNLP, pages 6787-6800, 2021.
[53] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, 2022.
[54] Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing moder- Visual parsing with self-attention for ssiion-ndd-lngusse pre-training. In NeurlPS, volume 34, pages 4514-4528, 2021.
[55] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning answer questions from millions of narrated videos. In ICCV, pages 1686-1697, 2021.
[56] Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In ICCV, pages 1 11562-11572, 2021.
[57] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127-9134, 2019.
[58] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jac Sung Park. Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. NewrIPS, 34, 2021.
[59] Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modas modal and hierarchical modeling of video and text. In ECCV, pages 374-390, 2018
13

---

[60] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, pages 2914 2017.
[61] Yueting Zhuang, Dejing Xu, Xin Yan, Wenzhuo Cheng, Zhou Zhao, Shiliang Pu, and Jun Xiao. Multichannel attention refinement for video question answering. 7OMM, 16:1-23, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "jBIAsvCTohi", "reviewer": "Reviewer_rUgm", "summary": "In this paper, the authors proposed to pretrain an end-to-end vision-language model especially on long videos. A hierarchical temporal window based attention mechanism is proposed to deal with long temporal dependencies. The effectiveness of the proposed model is validated on extensive empirical evaluations.\n\n", "questions": "Please see the cons for details.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of vision-language modeling on long-form videos is very interesting.\n2. The paper is well presented.\n3. Some strong empirical results are obtained.\n\nCons:\n1. Important ablations are missing in the current evaluations.\na. The most comparable baseline HD-VILA is unfortunately not based on the same backbone model, which makes it hard to understand the source of the performance gain.\nb. The authors entangled number of frames sampled and the temporal length in the current evaluation. What if only short video segments are used but the frames are densely sampled, e.g., using the HD-VILA-100M video segments but sampling 32 frames from each ~13-sec video segment.\n2. Important existing benchmarks on long-form video understanding should have been also evaluated, including [41,5] and [r1, r2, r3]. These well-established benchmarks are important as they require even longer-term video inputs, which could be good test-bed to understand the generalization of the proposed model on longer videos, e.g., 3-4 minutes. More justification on the selection of downstream datasets should have been provided, including the length and complexity analysis of these datasets.\n\n\n[r1] COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis, CVPR 2019\n[r2] Learning To Recognize Procedural Activities with Distant Supervision, CVPR 22\n[r3] Timeception for Complex Action Recognition, CVPR 19", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PbiRIEKwPAQ", "reviewer": "Reviewer_oRe6", "summary": "This paper proposes a method for long-form video-language pre-training by a multimodal temporal contrastive learning that leverages temporal continuity of clips within videos and their corresponding textual descriptions, and a hierarchical temporal window attention for encoding long videos using transformers. In addition, this paper proposes a new dataset with long-form video and paragraphs useful for pre-training video-language models. In experiments, the authors show that their method outperforms baselines in seven downstream tasks.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ New dataset for long-form video-language modeling that will be released for research\n+ New neural network architecture and contrastive pre-training for long-form video-language models\n+ Outperforms baselines.\n\n\nWeaknesses:\n\n- Baselines not pre-trained with proposed dataset\n-- In Table 1, the authors compare the proposed method trained with the proposed dataset outperforming the baselines on ActivityNet Captions. However, the baselines are not pre-trained with the same dataset as the proposed method. This makes the result inconclusive as to whether the performance boost comes from the proposed LF-VILA-8M dataset or the LF-VLP method or both. I suggest the authors to add results from the baselines pre-trained using LF-VILA-8M so that readers can make a stronger conclusion about the proposed method. Also, are all other results using baselines not pre-trained using LF-VILA-8M?\n \n\n- Importance of HTWA.\n-- In Table 5.b, the authors show a comparison between not using HTWA and using fixed size windows of 4 and 32. The difference between fixed 4 and HTWA performance seems marginal. Can the authors clarify how this result validates the effect of HTWA?\n\n- Without contrastive loss on global representation?\n-- In Table 5.a, the authors clearly show that pre-training is important. They also show the difference between using the proposed contrastive learning that uses time continuity to select the negatives and positives. However, I think this table is missing an experiment where the global contrastive learning is not used. This would make it clear if global contrastive learning is an important piece of the equation or if the proposed contrastive learning is good enough on its own.\n\n- Use stage two for retrieval to compare the performance of the cross modal encoder?\n-- The authors mention the representations learned from the model in stage one are widely used and efficient for retrieval. However, it would be good to report whether the cross modal encoder performs as well or better than the stage one model, especially since the cls token of this encoder outputs a score denoting whether the input paragraph and video are matched. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Oe8xOFCusC", "reviewer": "Reviewer_1qEi", "summary": "This paper focuses on pre-training a long-form video-language deep learning model. To handle the challenges in modeling long-form data, the authors propose 1) a large-scale dataset, 2) temporal contrastive loss, and 3) a hierarchical attention strategy. The proposed approach achieves favorable results on the retrieval and VQ tasks.", "questions": "Please see Weakness", "limitations": "Ths authors addressed the limitations and potential negative social impact well.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n1. Except for some indexing issues, the paper is well-written and easy to follow.\n2. The authors build a large-scale long-form video-paragraph dataset and promise to release it to the community. As the field is not well-explored, the dataset could be a critical factor in stimulating further research in this field.\n3. The proposed pre-training algorithm improves the performance on several downstream tasks.\n*Weakness*\n1. The indexing $v_{i, k}$ and $t_{i, k}$ in Section 3.2 are a bit confusing. The authors could consider to change the index to $v^i_k$ and $t^i_k$.\n2. It is unclear how the authors sample the set $\\mathcal{V}$ for the positive instance $t_{i, q^+}$. Specifically, if the set $\\mathcal{V}$ only contains frames that are all far away from the anchor along the time axis, then the instance with index $q^+$ is also far away from the anchor. How does the proposed method handle this situation?\n3. Recently, there have been some papers showing the benefit of leveraging a pre-trained text encoder for visual-language tasks, e.g., Imagen. Can the authors discuss/comment on the potential results of leveraging a well-pre-trained text encoder on tasks this paper focuses on?\nSaharia el at., \"Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HvSH-1UIp5", "reviewer": "Reviewer_ZX7j", "summary": "This paper proposes a method for Long-Form Video-Language Pre-Training, in which the author introduce 1) multimodal temporal contrastive loss for aligning multiple modalities and 2) hierarchical temporal window attention for capturing spatiotemporal dependencies at lower cost. Experimental results demonstrate improvement over the previous state-of-the-art.", "questions": "1.\tThe multi-model in this paper refer to the vision and language. Is audio data useful or complementary in the proposed method?", "limitations": "1.\tThe large-scaled self-supervised pre-training may generate more carbon emission. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper is well written, and the core idea is easy to understand. The proposed method and formulate is clean, straightforward, and easy to re-implement. \n\n2.\tLong-term video modelling and multi-model alignment are unsolved problems in the community. They are challenging due to the video redundancy/complexity and memory burden. It is great to see this work trying to investigate these problems. \n\n3.\tThe proposed hierarchical temporal window attention is an interesting direction for reducing the computation cost of the video transformer, which build local temporal attention window in early stage and global temporal dependencies in later stages when the feature map is smaller.\n\n4.\tThe experimental result looks good, which outperform previous state-of-the-art works.\n\nWeaknesses:\n\n1.\tIn multimodal temporal contrastive loss, is there an assumption that the clip content and sentence will not repeat in the video? Otherwise, the line 50-51 will not make sense.\n\n2.\tLine 157-158, what is the sampling strategy here for K and V. If these two sets are far away from each other, samples in the positive pair may not match each other.\n\n3.\tThe effectiveness of each component is not well analysed. There are some ablation studies in the Table 5. However, it is not clear to see the effectiveness of L_time alone, L_mlm, L_vtm, \\lambda_1 and \\lambda_2. It will be great to show how each part contribute to the final performance.\n\n4.\tMissing details in the comparison: Table 1-3 demonstrate the performance of this paper and previous works. In addition to the pretraining dataset, these details are also important: a). Computational cost. 2) Running Time. 3) Additional dataset (used in pretraining and pretrained model). 4) Input resolution. Combined with the weakness No.3, it is hard to distinguish which part of this paper is the key for the superior performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept