Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Authors: Yuanhao Xiong
Long Zhao
Boqing Gong
Ming-Hsuan Yang
Florian Schroff
Ting Liu
Cho-Jui Hsieh
Liangzhe Yuan
Venue: ICLR 2024 poster (2024)
Keywords: multi-modal learning
video and language
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose S-ViLM to strengthen model's understanding into fine-grained structures such as region-object correspondences and temporal scene changes.

Abstract: Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of importance to downstream tasks requiring temporal localization and semantic reasoning. A powerful model is expected to be capable of capturing region-object correspondences and recognizing scene changes in a video clip, reflecting spatial and temporal granularity, respectively. To strengthen model's understanding into such fine-grained details, we propose a simple yet effective video-language modeling framework, S-ViLM, by exploiting the intrinsic structures of these two modalities. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features, simultaneously. Comprehensive evaluations demonstrate that S-ViLM performs favorably against existing approaches in learning more expressive representations. Specifically, S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks, covering text-video retrieval, video question answering, video action recognition, and temporal action localization.

Published as a conference paper at ICLR 2024
STRUCTURED VIDEO-LANGUAGE MODELING WITH TEMPORAL GROUPING AND SPATIAL GROUNDING
Yuanhao Xiong ,33-
Long Zhao'
Boqing Gong'
Ming-Hsuan Yang' Liangzhe Yuan'
Florian Schroff 'Google Research Google
Ting Liu'
Cho-Jui Hsieh
UCLA
ABSTRACT
Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of im- portance to downstream tasks requiring temporal localization and semantic rea- soning. A powerful model is expected to be capable of capturing region-object correspondences and recognizing scene changes in a video clip, reflecting spatial and temporal granularity, respectively. To strengthen model's 's understanding into such fine-grained details, we propose simple yet effective video- Iado-language model- ing framework, S-ViLM, by exploiting the intrinsic structures of these two modal- ities. It includes two novel designs, inter-clip spatial grounding and intra-clip tem- poral grouping, to promote learning region-object alignment and temporal-awe aware features, simultaneously. Comprehensive evaluations demonstrate that S ViLM performs favorably against existing approaches in learning more expressive repre- sentations. Specifically, S- S-ViLM surpasses the state-of-the th-tt--rrtt methods substan- tially on four representative downstream tasks, covering text-video retrieval, video question answering, video action recognition, and temporal action localization.
INTRODUCTION
Videos are composed of groups of pixels spanning spatially and temporally. Semantically related groups of pixels form the objects in the visual scenes and their changes through space and time vividly show the action and interactions of physical world. Scene switching further complicates the video story line and finally depicts attractive video stories. The similar structures also appear in the paragraphs when they come to describe the vidcos. Captions are built from the basic grammar components such as nouns and verbs, and sentences are concatenated describe complex scenes.
Modern rideo-languae models (VLMs), however, mostly neglect the fine-grained structures of such video-text pairs during the development. Video-language pre-training typically follows the pipeline: (1) encoding video and text pairs into latent representations, (2) modality fusion, and (3)
pre-training on specific objectives. Existing methods typically optimize these three components in
the pre-training pipeline by designing expressive encoders (Bain et al.) 2021 Li et al. 2022
Ge
ctt al. 2022al Nagrani et al. 2022; Ma et al. 2023), ) fusing two modalities via a cross cross-cncoder
GGe
iet al. 2022al Li al. 2022 Lei et al. 2021 Li ct al. 2020; Luo et al.) 2020 Xu et al. 202 la
Zhu
& Yang 2020), or adopting combination of various pre- traintaining tasks such as contrastive learning
and masked modeling (Li al. 2022) Ge et al. 2022a Fu et al. 2021 Zellers al. 2022 Cao
jet al. 2022 Ge et al. (002b)). While these modifications benefit the pre- traitedined model, their lack of
local discriminative modeling poses challenges for VLMs to further understand complex videos.
has been shown that most video-language pre-training methods merely perform well on learning holistic representations match (video, caption) pair while neglect fine-grained information such as region-object correspondences. or scene/action changes along the time in video (Akbari et al. 2021 Bain al. 202T Lei et al. (202 Li et al. 2020 Luo et al. 2020 Miech al. 2019 Xu al. 202 Nagrani et al.) (022). However, such regional or temporal fine-grained information has been demonstrated to play a vital role localization and reasoning tasks (Li et al. 2022) Ge et al. 2022a]
Work done student researcher at Google Research.

---

Published as a conference paper at ICLR 2024
Zhang et al. 2022 Ma et al. 2023 Yuan et al. 2022).
Motivated by aforementioned observations, we revive the strong connectivity between basic components of video clips and languages during self-supervised iideo-language pre-training. We approach the video-language pre-training task from a different perspective with a focus on exploiting spatiotemporally fine-grained structures.
In this work, we incorporate structured video-language interactions into the pre-training stage and propose a novel framework, namely Structured Video-Language Modeling (S- ViL,,, with tempo- ral grouping and spatial grounding. S-ViLM encourages instance-level video-caption alignment, fine-grained region-object alignment, and learns temporal-aware video representations, simultane- ously, As shown in Figure S- S-ViLM consists of three primary training objectives: inter-clip spatial grounding, intra-clip temporal grouping, and global contrastive learning. Given video-caption pair as the input, a classical dual-encoder model leveraged extract the representation for each modal- ity, respectively. Videos are pre-processed with the cut-and-paste operation, inspired by (Zhang) al., 2022, Yun et al.) 2019), i.e., pasting one clip in a video onto the other background video, to explicitly introduce temporal scene changes. We further adopt grouping blocks (Xu et al. 2022 Yu ct al. 2022) to aggregate semantically similar video patches to represent regions without ofe-the- shelf detectors via a set of group tokens shared among all videos. In inter-clip spatial grounding, we align grouped video tokens with objects represented by nouns in the caption by minimizing our designed grounding loss. In intra-clip temporal grouping, we improve features temporal granularity by distinguishing foreground and background representations within one clip. Finally, the model is trained by global video-caption contrastive loss to match instance-level video-caption pairs. We evaluate our proposed method comprehensively on four representative tasks, including text-video retrieval, video question answering, video action recognition, and temporal action localization. Our strong experimental results demonstrate that exploiting fine-gr mee-griieed video-text text structures during pre- training effectively improves VLM's s video understanding and reasoning capabilities.
Our key contributions are summarized as follows:
We propose M-,,LM, a dual-encoder video-language modeling framework, making use of structured video-caption interactions learn more expressive spatiotemporal features.
We leverage cut-and- paste operation to introduce scene changes into videos during pre- training, and propose an intra-clip grouping module to learn more temporal-aware aware features.
We design an inter-clip spatial grounding module to capture fine-grained correspondences by aligning objects from the caption and regions from the video in a self-supervised manner.
Experimental results have demonstrated the effectiveness of S-ViLM on four downstream tasks, including text-video retrieval, video question answering. video action recognition, and temporal action localization. For example, S- ViLM outperforms SOTA by 3% in R in zero-shot video-text retrieval on MSR- VTT and 5% in accuracy in action recognition UCF101, showing its advantages over both multi-modal and single-modal tasks.
2 RELATED WORK
Video Video-language pre-training. Video-language pre-training is emerging research area that aims to develop machine learning models capable of jointly understanding visual and textual content. Representations learned from large scale noisy datasets such HowTo100M, (Miech et al.) 2019), Web Vid (Bain et al., 2021), and VideoCC (Nagrani et al. 2022) have demonstrated great potentials in adapting downstream tasks, including but not limited to text-video retrieval, video question answering, and video captioning. Elaborately designed pre-training objectives ranging from gener- ative (Chen al. 2020b Fu al., |021 Li et al. 2019 Liu et 2022) discriminative (Bain et al.) 2021 Lei et al.] 2021] Akban et al. 2021 Sun et al. 2022] Li et al., 2022,, Ge et al.) [022a Ma et al. 2022; Wang et al. 2023a blc) have been proposed, among which contrastive learning is prevalent and widely adopted to attract paired video-caption instances and repelling unpaired ones.
However, their primary focus is still on learning holistic global representations to align instance-level (video, caption) pairs. Recently, some approaches have been proposed to leverage finer-grained in- formation such as nouns/verb phrases from a caption. ALPRO (Li et al. [022) extracts pseudo entity labels by feeding noun prompts into frozen model and use contrastive objective align cropped visual regions and the corresponding textual labels. In Ge et al.) (2022a), MCQ recovers randomly masked noun/verb tokens via resorting global video features, which implicitly improves text entity
2

---

Published as a conference paper at ICLR 2024
association in visual encoding. LAVILA Zhao et al. 2023 constructed temporally dense captions by automatic annotation from large language models to describe activities more comprehensively. In addition, TemPVL (Ma et al. 2023) enables temporal and semantic alignment such that the trained model can accurately perceive temporal boundaries in videos given the text description. Despite these efforts, correspondences between visual regions and objects from noun concepts in captions and temporal scene shifts in a video are still neglected and not modeled explicitly in existing video- language pre-training methods. In this work, we propose two novel designs, spatial grounding and temporal grouping, to leverage tne-grained information in the pre- pre-training stage.
Vision language grounding. The goal of visual grounding (VG) to locate the most relevant object or region in visual input based on a natural language query (Fang et al.] [015]] Robrbach al. 2016, Fukui et al., 2016 Ghiasi et al., 2022: Gupta et al., 2020). Recently, visual ground- ing has been adapted to pre-training tasks in self-supervised manner for open-vocabulary image segmentation (Ghiasi et al., 2022] Xu et al. 2022). For example, OpenSeg (Ghiasi et al.) (2022) se- mantically aligns a caption with extracted image regions via a grounding loss. Moreover, without the off-the-shelf object detectors, GiT (Xu et al. 2022) learns to group together semantic regions from text supervision by contrastive learning. Note that visual grounding is mostly discussed in the image domain and its success motivates us to extend visual-semantic alignment to video-language pre-training. To achieve this, we integrate novel spatial grounding module in our framework to promote visual and textual entity correspondences in a self-supervised manner.
Video temporal modeling. In contrast to images, videos contain a sequence of dynamic frames and how to model temporal information is critical in video understanding (Feichtenhofer et al.)
2019 Bertasius
2021] Tran et al., [2014] Alwassel et al. 2021] Zhang et al. 2022; Qian
et al. 2022). Specifically, TSP (Alwassel et al. (022)) learns temporal information via predict-
ing clips inside or outside the action with substantial annotations. PAL (Zhang et al.
2022) aligns
features of pasted pseudo action regions from two synthetic videos. BSP (Xu et al.
2021c) intro-
duces a novel boundary-sensinn
pretext task via classifying the boundary types of synthetic videos.
These techniques are elaborately designed for training models on long videos such as movies or TV dramas, which contains natural scene changes. However, few of them have been considered in video-language pre-training since the majority of video-language datasets contains short videos with repeated frames and are lacking in temporal differences. Instead, we develop a temporal grouping method to learn temporal-aware clip features in self-supervised manner. We show that features extracted from explicitly temporal modeling achieve significant improvements in not only temporal action localization tasks, but also coarse-grainee reasoning and understanding tasks such as video question answering and video action recognition.
METHOD
3.1 OVERVIEW
The framework of S-ViLM is presented in Figure [ We adopt the dual encoder architecture for video-language pre-training, and there are three primary objectives used in the pre-training stage: (1) inter-clip spatial grounding, (2) intra-clip temporal grouping, and (3) global contrastive learning.
As shown in Figure [ temporal changes are first artificially introduced into training examples through cut-and-paste. Then the pre-processed video together with learnable group tokens are fed into the video encoder. Specifically, group tokens aggregate semantically similar video tokens via grouping blocks and are then aligned with object concepts by spatial grounding. promotes region- object groundingness, which indicates the alignment between a region in the video and an object in the caption, e.g., as illustrated in Inter-clip Spatial Grounding Figure 1, the red region corresponds exactly to the word "pins" in red. In contrast to previous methods where regions are extracted with pre-trained object detectors (Cai et al. 2022 Li et al. 2022. Yan et al. 2021), these learnable group tokens can cluster and organize semantically similar regions in a self-supervised manner, which is more effective and reduces the artifacts of any detectors. For the language branch, the original cap- tions are tokenized into sequence of text tokens, which are then fed into text encoder to extract the corresponding representation from the preceding [CLS] token. Noun tokens representing objects extracted in the same way given a set of prompts.
3

---

Published as a conference paper at ICLR 2024
Intra-clip Temporal Grouping
Inter-clip Spatial Grounding
Repel
Repel
Attract
man pins bowling ball
Repel
video tokens
group tokens
caption tokens
Contrastive Learning
Video
Text Encoder
Cut & Paste
A video
video contoins pins.
of man.
throws bowling ball into vartous pins.
Noun Extraction & Prompting
input video clip
input caption
Figure 1: Illustration of S-VILM pre-training. Three proposed training objectives promote struc- tured video-language interaction: (I) temporal grouping learns temporal-aware features by distin- guishing whether clips are from background or foreground; (2) spatial grounding focuses on local correspondences between regions and objects; (3) global contrastive learning matches instance-level (video, caption) pairs.
To promote temporal awareness, we use masks derived from the cut-and-paste operations as the ground-truth for temporal grouping. Furthermore, we model the interaction between region features and noun tokens using inter-clip spatial grounding loss. Finally, global contrastive loss is computed between the video and the caption representations to match the instance-level (video, caption) pair.
3.2 ANTRA-CIP TEMPORAL GROUPING WITH CUT- OUT-AND-PASSE PASTE
Commonly-used video-language pre-training data usually consist of short video clips with repetitive scenes. To simulate scene shifts, we design cut-and-paste operation inspired from image augmen- tations YYun et al., 20199] Zhang al. 2022) to introduce temporal changes manually to further improve video representations.
Given target video v; with 7' frames as the foreground and a randomly sampled video Ep, with the index as the background from the same batch of size B, we divide each video into N; = T clips with the temporal window size We then sample the start and end clip indices and from N₂), and paste the corresponding region from o; into the background video Up. to form blended video 6z For the clip sampling procedure, we first uniformly sample the duration of foreground video d from [N,/2, N₂) to guarantee it is the majority of the blended video. Then we sample the start index s from [0, .r, - d), and the end index e was computed naturally as e = s + We included this detail in our latest version. We define the foreground-be bargground-backgroundd mask m, E Ig., {1() [s, e]ií E [0, N.)). where 1(.) is the indicator function. This operation is illustrated in Figure 1
A video is first flattened into N nveeoverlg voxels. After projected by linear layer, these voxel tokens are fed into the transformer encoder to obtain transformed tokens 2," Nxd where
is the feature dimension. To obtain clip-level representations clip E R.N.x X. we average-pool over 2 along the spatial dimension after recovering the feature map's sup's 3D shape. Two cluster centers, 2f for the background and z/ for the foreground, are further computed by averaging features from 2,' on the corresponding position based on the mask m,. To assign each clip to either background or foreground, we compute a, via cosine similarity with element-wise sofimax function applied on the last dimension, where (s } is cosine similarity and 7 is the temperature to scale logits:
a; Softmax clip [2)
= =(1))/> +
N.2
4

---

Published as a conference paper at ICLR 2024
Finally, the temporal grouping loss can be computed within a batch between a, and the ground-truth one-hot masking m, using mean squared error as
1 B B
4=
Σ (CCE(a, One-hot(m,)).
(2)
Note that we have also tried the binary cross entropy loss which performs comparably to MSE. Thus, we select relatively simple MSE loss for temporal grouping.
3.3 INTER-CLIP SPATIAL GROUNDING WITH GROUP TOKENS
Observing the correspondences between visual regions video and noun phrases (objects) caption, we model such fine-grained alignment for more expressive encoders. In practice, it is in- feasible to pool tokens of interest as cluster centers since we do not have ground-trutt segmentation. Thus, we adopt M learnable group tokens to cluster semantically similar regions in self-supervised manner. Note that group tokens are randomly initialized and shared among different videos. The detailed structure of grouping block is presented in Appendix [A] and multiple grouping blocks are placed at different layers of the video encoder to update group tokens progressively. The final group tokens denoted ent regions in the video u. Compared with using off- shell region proposal networks, our design of token grouping more computationally efficient, and can be adapted the pre-training dataset without region annotations in a self-supervised manner dynamically and flexibly. For each caption G; of a video, we extract K noun phrases using noun chunking with a set of handcrafted sentence templates, e.g., "A photo of a {noun)". Such prompted noun phrases are fed into the text encoder to extract noun tokens {n 11
{gghh m=1 aggregate semantically similar voxels and represent differ-
spaCy and prompt each of them
Jk=1
We define the notation for softmax on a vector x at the i-th element as: oxx; Σ; exp(xs)) //> where T is the temperature to scale logits. The similarity of all group tokens 9 with respect to a noun token n² is defined as s(G, n²* = [(',,="), nr)),,,,,, (gd n²)] E R.M where (:-) is the cosine similarity. Since the ground-truth correspondences between regions and nouns are inaccessible, we compute the grounding similarity between all group and noun tokens by:
exp(x.)/r
M
G(v,c)
1 K
n
Σ a (s(G. m gm
m==1
G(u, c) encourages each noun to be grounded to one a few regions and avoids penalizing regions that cannot find any relevant nouns.
G Similarity scores over batch of size B are computed as: G(V.c) () Glvn, ci)] B and G(v,, C)- = (G),,, Ci).... G(v,, RR where V = (u and C {c.) denote the set of videos and captions in batch, respectively. Inter-clip spatial grounding loss Cg is then defined to enable nouns to be matched with regions for each positive (video, caption) pair: Lg = f fV-+c + Lg Le-+" consists of video-to-caption grounding loss and a caption-to- to-iieoo grounding loss
B
B log a (GV,,)))).
C
Σ
log (G (w,
C 5
1
Σ 1=1
(4)
B
Recall that the cut-and-pass operation indicates that ú, has another positive caption cp, besides its original 4, and the loss weights positive indices are W E BB = {wf,) which satisfy
i=
w
1 B, = Pr
(5)
0,
otherwise
where B, = (e s)/N, is the ratio of the foreground in the cut-and video 0, From the perspective of captions, can obtain We (W*)² We can derive the augmented grounding loss fag with the video-to-caption loss and and the caption-to-video loss:
-- -L w log a (G C)) C
ΣE Ef wh, log a (V.ca)) (6)
https://spacy.1nt
5

---

Published as a conference paper at ICLR 2024
3.4 OVERALL PRE-TRAINING OBJECTIVE
We include global contrastive learning objective for instance- alignment, fi" the video repre- sentation of Bg. extracted from average-pooles group tokens and ff the caption representation c₁, is computed from the [CLS] ] token of the original caption. Instance similarity scores are defined as: s(V, (4) = JJ)) ......., (BB: J()) E B and s(0;, C) : (f),..)).... (s [ā]] E TR A global contrastive loss is defined as Leomtrast L and the caption-to-vicco views:
C combination of the video-to-caption
C contrast
B - د w log (s((i, C),,
CC- contrast
21-1 2--1 wf, log (ss())()))))))))))))))))))((())) (7)
The overall pre-training objective a weighted sum grouping loss, grounding loss, and global contrastive loss: L = W1 C + wyCg + WyLcontratt We set three weights, w1, Wg. and wa, be equal to one in our experiments for simplicity.
4
EXPERIMENTS
4.1 DOWNSTREAM TASKS
Text-video retrieval. We adopt the widely used text-video retrieval benchmark MSR-VTT Xu al. 2016) for evaluation. It consists of 10K YouTube video clips with 200K captions. We conduct experiments in both zero-shot and fine-tuning settings. For fine-tunne tuning setup, we follow Bain et al. (2021) and Ge et al.] (2022a), and train and test the model on the split of 9K and 1K videos.
Video question answering (VQA). We consider open- open-ended VQA settings with two representative datasets: (1) MSRVTT-QA (Xu et al., 2017) with 1,500 answer candidates and (2) MSVD-QA (Xu 2017) with 2,423 answer candidates.
et al.
Video action recognition. We select HMDB51 Kuehhne et al. 2011) containing 766 videos with 51 categories and UCF101 (Soomro et al. 2012) containing 13,320 videos with 101 categories. Both linear probing and fine-tuning the whole model are explored.
Temporal action localization (TAL). TAL aims at predicting the temporal extent and the labels of action instances. We evaluate the performance on ActivityNet (Heilbron et al., (203)). an action understanding dataset of 19,994 temporally annotated untrimmed videos with 200 action categories.
4.2 IMPLEMENTATION DETAILS
Input. Following Nagrani et 2022 sample 32 frames for each video and resize them into
224 x 224 with the same augmentations. Each caption is tokenized into 32 tokens including [CLS].
K noun phrases are extracted for each caption and then prompted with a of prompt templates such as "It is a video of [nown)" " We include the full list of prompt templates in Appendix A
Model architecture. We
12-layer ViT-base model with the patch size of 2 16 x 16 as
the video encoder and initialize it with weights pre-trained on Kinetics-400, We adopt 32 learn-
able group tokens and grouping blocks featuring K-means attention (Xu al. 2022 Yu al. 2022). Grouping blocks are inserted at the 6th, 9th and last layers of the video encoder (Xu et al. Yu al. 2022). The text encoder initialized from the pre-trained BERT-base base model. All representations are projected into the common space with the dimension of 256.
2022
Pre-training datasets. We pre-train S- ViLM with the VideoCC (Nagrami al. 2022 dataset, which contains about 3.3M video-caption pairs. We also include Activity Net-Caption (Kishhna et al. 2017) with 20K well-aligned pairs into the pre-training corpus. We note the commonly ussed Web Vid (Bain) et al. 2 is unavailable to us due to the restricted data access policy. To illustrate the effectiveness of our proposed method and how the pre- pre-training datasets contribute to the final results, we designed fair studies on dataset impacts. Details could be found in Section 4.3.5
Pre-training and fine-tuning setups. We implement S- S-ViLM in JAX and train all models on TPU accelerators. During pre- mre-trannng,, SGD with momentum 0.9 and initial learning rate 0.1 is used for optimization. We train S- ViLM for 10 epochs with a batch size 1024 and adopt a cosine learning rate decay schedule with warmup ratio 0.05. It takes about one day for the whole pre-training stage. In

---

Published as a conference paper at ICLR 2024
Table 1: shoo (top) and fine-tuning evaluation (bottom) of text-video retrieval on MSR-VTT test set with 1K videos. Higher R @@k and lower MedR (Median Rank) indicate better performance.
Method
Video Encoder Input
Dalaset
Pairs
RØ3
RØ
MedR
MIL-NCE. Miech et al
Miich
2019
Raw Videos
HowTol00M
120M
9.9
24.0
32.4
29,6
VATT
HoWToL00M, AudioSet
29.7
VId
Xu et al 20216
S3D
HowTolO0M
110M
10.4
22.2
30.0
Pairicc Set
20120
R(2+1)D-34
27.5
36.2
24.0
Frozen (Bam e Fil 2044
Raw Videos
CC3M. WebVid 2M
5.5M
18.7
39.5
51.6
10.0
(Rovdncenen.o ctat
ReaNeXt-10
1000M
40.8
50.7
DemoVVL
Caa el 20724
Raw Videos
CC3M. Web 2M
5.5M
24.0
44.0
52.6
8.0
ALPRO
Videos
CC3M. WebVid-2M
55.4
MCQ (GS de 20204
Raw Videos
CC3M, WobVid-2M
5.5M
26.0
46.4
56.4
7.0
VCC
(Nagrani
Videos
VideoCC
37.5
S-VILM
Raw Vidcos
VideoCC, ActivityNet
3.3M
28.6
53.6
65.1
5.0
UniVL (Lug
S3D
MMT
Gateer of cue RU20
S3D
HowTo100M
120M
26.6
57.1
69.6
4.0
ClipBERT
2014
VisGenome
59,9
AVnnet (Rouditciennoo c 2021
ResNeXt- 101
HowTo100M
120M
27.1
55.6
66.6
4.0
Supportser
R(2+1)D-34
HoWTo100M
120M
30.1
58.5
69.3
3.0
VideoCLIP X et 0 20216
S3D
HowIooDOM
110M
30.9
55.4
66.8
Frozen
Raw Vidcos
CC3M. WebVid-2M
5.5M
59.5
70.5
3.0
DemoVi (Cal er ml 2012
Row Videos
CC3M WabVid-2M
5.5M
36.0
61.0
71.8
3.0
ALPRO
Raw Vidcos
CC3M. WebVid-2M
5.5M
60.7
73.2
3.0
MCQ GG dral 2i1z4
Row Videns
CC3M. WahVid-2M
5.5M
37.6
64.8
75.1
3.0
VIOLETVE (lu
2025
Raw Videus
CC3M. WebVid 2M
5.5M
37.2
64.8
75.8
All-in-One WWang al 20123
Raw Videos
How 1owTL. Web-id-MM
112M
37.1
66.7
75.9
VCC
Nagram
Raw Videus
VideoCC VideoCC, ActivityNet
3.3M
35.0
63.1
75.1
S-VILM
Raw Videos
3.3M
38.4
65.7
76.3
2.0
terms of fine-tuning, different tasks are trained independently with their own set of hyperparameters on the target dataset and more details can be found in Appendix A For temporal action localization, we fix weights of the pre-trained video encoder and its grouping blocks to extract video features, which are then evaluated by G-TAD (Xu et al. 2020). commonly used method for TAL.
4.3
EVALUATION RESULTS
4.3.1 TEXT- VIDEO RETRIEVAL
We evaluate S-ViLM for the text-video retrieval task on MSR-VTT under both zero-shot and fine- tuning settings, and compare with existing prevalent methods in Table S- V-VLLM outperforms other methods significantly for zero-shot evaluation with R @@I 10 of 65.1, yielding approximately 9% improvement over the best-perforning baseline MCQ. The superior results demonstrate that our pre-trained model builds up a good alignment between video and language and generalizes well to unseen datasets. M-V also achieves performance gain when the model is fine-tuned on the target MSR-VTT dataset, which further validates advantages of the pre-trained model. Note that S- S-ViLM performs favorably against existing methods despite the much smaller size of the pre-training data used in S-VILM than those in baselines, such How ToT.oM and WebVid-2M.
4.3.2 VIDEO QUESTION ANSWERING
VQA results on two open-ended datasets are shown in Table 2
To enable S-ViLM deal with the VQA task, we add a fusion head adapted from BUTD (Anderson et al. 2018) by integrating video and text features with simple linear layers. Then a classifier is inserted after the fusion module to perform question answering as classification problem. Compared with previous methods which leverage particular architectures for VQA or include complicated fusion encoder, S-ViLM is the most efficient and flexible for various vision- tasks. S- S-ViLM achieves better performance than competing methods with the accuracy of 43.5% (+1.4%) and 46.4% (+0.5% MSRVTT-QA and MSVD-QA, respectively.
4.3.3 VIDEO ACTION RECOGNITION
For video action recognition, we only keep the video encoder together with its grouping blocks to extract single-modality video representations for evaluation. Two evaluation settings are considered: (1) linear probing where the backbone encoder is frozen and only the last linear classifier trained and (2) end-to-end fine-tuning where both the backbone and the classifier are trained. Top-1 aceu-
7

---

Published as a conference paper at ICLR 2024
Table 2: Top-I accuracy (%) of Video Question Answering on MSRVTT-QA and MSVD-QA.
Method
PT Dataset
MSRVTTQQA
MSVD-QA
HGA Diane Ham
2020
35,5
34,7
QUEST
HCRN Lx al
2010
35.6
36.1
ClipBERT Let al.
2021
COCO, VG
37.4
SSML Amirant
How T0100M
35.1
35,1
CoMV ISst
39.5
DemoVLi CarE PIZ4
CCSM, WobNid-2M
38.3
39.5
ALPRO
CCSM, WobNid 2M
42.1
45.9
S-VILMI
VideoCC, ActivityNet
Table Experiments of action recognition UCF101 and HMDB51 with linear evaluation (Lin) and fully fine-tuning evaluation (FT).
Table 4: Comparison SOTA methods on temporal action localization (TAL).
Method
Modal
Lin
UCF101
PT
HMDBSI
Lin
FT
Method
mAP@0.5
TAL Task (G-TAD)
@0.75
@0.95
CoCLR
al.
77.8
CoCLR (Han
2020
47.9
31.9
MVCGC FRom
2021
78.0
90.8
XDC (Alwassel et
2020
18.4
32.6
7.5
32.3
XDC R AAwasse et di 2020
A
80.7
88.8
49.9
61.2
(Caza MoCo-v2
20200
46.6
30.7
30.3
XDC.K K.
85.3
91.5
56.0
63.1
VideoMaC
MIL-NC
834
89.1
$4.8
RSPNet (Ches er
47.1
31.2
7.1
30.9
Prozen
Iaiin ar MM1
T
87.8
89.8
61.3
06.3
Acl Weare E18 2011S
44.1
28.9
5.9
28.8
VATT
(Atban
2021
A.
89.2
63.3
SpeodNet Benain
ELO
MMY
MMV
Piererov
KlaVes et
2070
A
T
OF
86.8
77.1
55.1
53.6
PAL (Zhang Q 21/i
BSP
TAC Atrera 2020
48.5
50.9
50.7
35.5
35.6
31.9
7.2
34.8
32.5
34.6
MMV
91.8
95.2
67.1
Loti
50.4
35.4
34.4
MCQ
Le al 20225
T
89.1
92.3
65.8
69.8
TSP (Afunsse 20 ii 2121
51.3
37.1
9.1
38.8
S-VILAI
7
94.8
96.5
70.0
76.9
S-VILMI
51.7
36.4
9,7
35.6
racy UCF101 and HMDB51 is reported in Table We observe that in linear probing, S-ViLM outperforms other baselines, with 3.0% and 2.9% higher than current SOTA, MMV that leverages audio and text modalities in addition on UCF101 and HMDB51. S-ViLM also achieves consis- tently superior performance under the fine-tuning evaluation. Outstanding performance of S-ViLM demonstrates that leveraging lne-grained video language structures during pre-training contributes to meaningful video representations, This aligns with our intuition because finer mner-grained video-text alignment improves video understanding.
4.3.4 TEMPORAL ACTION LOCALIZATION
We report the mean average precision (mAP) under different temporal Intersection over Union (tloU) thresholds on ActivityNet in Table For temporal action localization, the model is pre-trained on HowTo100M only, which is observed be beneficial TAL compared with VideoCC Activi- tyNet (see the ablation study below). We directly use pre-trained models to extract video features as the imput G-TAD and do not further train the encoder. S- S-ViLM consistently exceeds other self-supervised competitors and even fully supervised approaches such as LoFi and BSP. This ob- servation again consolidates the conclusion that vision-language pre-training can not only be applied to specific VI. problems like text-video retrieval, but also benefit single-modal downstream tasks.
4.3.5 ABLATION STUDIES
Pre-training datasets. To analyze of effects of pre-training datasets, we report the model per- formances on selected downstream tasks in Table [ In particular, the same model pre-trained on VideoCC achieves the best performance in zero-shot retrieval on MSR-VTT, compared with HowTo100M and WebVid-2MM These results coincide with findings in Nagrani al. (2022), where HowTo100M has been pointed out not appropriate for vision-language tasks requiring strong align- ment. S-ViLM trained on VideoCC alone significantly outperforms VCC on both tasks, showing the effectiveness of our proposed techniques. In particular, when pre-trained the same VideoCC dataset, S- ViLM leads to better performance than MCQ. The significant improvement over MCQ shows that techniques do help to learn better features for downstream tasks. It is also worth noting that pre-training on VideoCC and ActivityNet performs consistently better than using only one dataset, and thus we choose this setup in the main experiments.
Training objectives. Without loss of generality, the model in this ablation is pre-trained on VideoCC only. For better understanding S-ViLM, we start with the contrastive baseline represented in Sce-
8

---

Published as a conference paper at ICLR 2024
Intra-clip Temporal Grouping
Inter-clip Spatial Grvending
Caption: A main was driving Eractor land reclamattion.
CH
20
RAT
CAN
60
81
620
buseline
lempural-sware
Caption: The bost makes turn the right
Figure 2: Visualization of M. Left: Similarity scores features derived from the baseline and our method. Right: Attention maps between region and object with spatial grounding.
Table Effects on different choices of pre- pre-training datasets.
Method
PT Dataset
M-RVTT-ZS R@5
R@
mAPe0.5
TAL.
0.95
Arg
HowTo10MM
10.4
22.2
30.0
VCC
WebVid
15.4
33.6
44.1
VideoCCC
37.5
49,9
34.3
8.7
33.7
MCQ
VideoCC
22.5
43.8
51.8
HowToLO0M
9.4
22,9
31.3
51.7
36.4
9.7
35.6
ActivityNet
14.4
33.5
44.0
50.5
35.3
8.7
34.5
VideoCCC
24.7
47.4
50.5
35.0
9.2
VidoOCC, ActivityNet
28.6
53.6
65.1
50.8
35.6
9.3
31,7
nario in T Then we add our proposed spatial grouping module during the pre- pre-training phase. This module is driven by the grouping loss L. in Scenario 2, and we observe consistent improve- ments on all tasks across the board comparing to Scenario 1. Similarly, we introduce the temporal grouping module in C, to encourage more temporal discriminative video representation. After com- paring Scenario 3 to Scenario 1 in Table 6 we also observe noticeable improvements on different downstream tasks. These phenomenons suggest both spatially and temporally fine-graid fine-grained features improve video understanding tasks. After combining everything together in Scenario 4, we show significant performance improvements on all tasks, which demonstrates the effectiveness of S-ViLM pre-training. Moreover, visualize effects of temporal grouping and spatial grounding in Figure can be observed from similarity scores among frames that with temporal grouping, features from different scenes much easier distinguish. Besides, attention maps from spatial grounding indi- cates the alignment between the region and the noun phrase has been learned during the pre-training stage without any fine-grane mne-grained annotations. More examples can be found in Appendix C
Table 6: Ablation study training objectives. We validate that our proposed spatial grounding loss and temporal grouping loss both benefit downstream tasks.
MSRVTT-ZS
MSVD-QA Acc
UCF101
TÄL
Scenario
Couatint
4g
R@1
R@B
R@10
Acc
mAP@0.5
0.75
0.95
Avg
22.7
45.9
57.0
43.6
90.5
49.9
34.3
8.7
33.7
2
/
23.3
46.6
58.6
44.1
90.6
50.2
34,7
8.7
34.0
24.2
46.7
58.2
43.9
90.9
50.1
34.6
34.0
24.7
47.4
59.0
44.9
91.0
50.5
35.0
9.2
34.2
CONCLUSION
In this paper, we present novel video-language pre-training framework, named S- V-M,M, that aims to utilize fine-grained structures in video and languages to learn region-object correspondences and temporal-aware features simultaneously. Spatial grounding and temporal grouping are introduced to achieve the goal of local region-object alignment and temporal distinction in a self-supervised manner. The proposed framework outperforms existing methods significantly on downstream tasks, including text-video retrieval, video question answering, video action recognition, and temporal action localization. The superior performance validates our design and our method could be easily scaled up, it is self-contained and does rely other artifacts.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank the reviewers for their invaluable feedbacks. Cho-Jui Hsich is partially supported by NSF 2331966, 2325121, 2330830, 2244760, 2008173, and ONR 20230936.
REFERENCES
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In Advances in Neural Information Processing Systems, volume 34, pp. 14206-242211 2021.
Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self- ceff-superviee mul- timodal versatile networks. In Advances Neural Information Processing Systems, volume 33, pp. 25-37,2 2020.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering, In Advances in Neural Information Processing Systems, volume 33, pp.
2020.
Humam Alwassel, Silvio Giancola, and Bernard Ghanem. Tsp: Cemporally-sensitiee pretraining of video encoders for localization tasks. In IEEE International Conference on Computer Vision, pp- 3173-3183, 2021.
Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. Noise estimation using density estimation for supf-super multimodal learning. In AAAI Conference on Artificial Intelligence, volume 35, pp. 6644 6652, 2021.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, 2018.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, pp. 1728-1738, 2021.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness videos. IEEE Conference Computer Vision and Patter Recognition, pp. 9922-9931. 2020.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In International Conference on Machine Learning, volume pp. 4, 2021.
Guanyu Cai, Yixiao Ge, Alex Jinpeng Wang. Rui Yan, Xudong Lin, Ying Shan, Lianghua He, Xiaohu Qie, Jianping Wu, and Mike Zheng Shou. Revitalize region feature for democratizing video-language pre-training arXiv preprint 2731v:2003.07720, 2022.
Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. Loevtp: Video-text pre-training for temporal localization. European Conference on Computer Vision, pp. 38-56, 2022.
Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. Rspnet: Relative speed perception for unsupervised video representation leamn ing. In AAAI Conference on Artificial Intelligence, volume 35, pp. 1045-1053, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04223 04297, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan. Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pp. 104-120, 2020b.
10

---

Published as a conference paper at ICLR 2024
Hao Fang. Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1473-1482, 2015.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In IEEE International Conference Computer Vision, pp. 6002-6226222222 0221,,21,111 2019.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Vi- olet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprini arXiv:211 121681, 2021.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. An empirical study of end-to-end video-language transformers with masked visual modeling. In 2023.
IEEE Conference Computer Vision and Pattern Recognition, pp.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.012 2016.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In European Conference on Computer Vision, pp. 214-229, 2020.
Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li. Ying Shan, Xiaohu Qie, and Ping Luo. Bridging video- text retrieval with multiple choice questions. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 16167-16176. 2022a.
Yuying Ge. Yixiao Ge, Xihui Liu, Jinpeng Wang, Jianping Wu, Ying Shan, Xiaohu Qie, and Ping Luo. Miles: visual bert pre-training with injected language semantics for video-text retrieval. In European Conference on Computer Vision, pp. 691-708, 2022b.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open- open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557, 2022.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Con- trastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision, pp. 752-768, 2020.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representa- tion learning. In Advances in Neural Information Processing Systems, volume 33, pp. 5679-5690, 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In IEEE Conference Computer Vision and Puttern Recognition, pp. 961-970, 2015.
Yuqi Huo, Mingyu Ding, Haoyu Lu, Nanyi Fei, Zhiwu Lu, Ji-Rong Wen, and Ping Luo. Compressed video contrastive learning. In Advances in Neural Information Processing Systems, volume 34, pp. 14176-14187, 2021.
Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. Divide and conquer: Question- guided spatio-temporal contextual attention for video question answering. In AAAI Conference on Artificial Intelligence, volume 34, pp. 1 11101-11108, 2020.
Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In AAAI Conference on Artificial Intelligence, volume 34, pp. 11 11109-11116, 2020.
Ranjay Krishna. Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. IEEE International Conference on Computer Vision, pp. 706-715, 2017.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In IEEE International Conference on Com- pufer Vision, pp. 2556-2563 2011.
11

---

Published as a conference paper at ICLR 2024
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1972-9981. 2020.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for andeo-and-langugee learning via sparse sampling. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7331-7341, 2021.
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Nicbles, and Steven Hoi. Align and prompt: Sndeo-and-language pre-training with entity prompts. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2022.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training, In Empirical Methods in Natural Language Processing, pp.
2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557. 2019.
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In European Conference on Computer Vision, pp. 319-335, 2022.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: unified video and language pre-training model for multimodal under- standing and generation. arXiv preprint arXiv:2002.06353, 2020.
Fan Ma, Xiaojie Jin, Heng Wang, Jingjia Huang, Linchao Zhu, Jiashi Feng, and Yi Yang. Temporal perceiving video-language pre-training. arXiv preprint arXiv:2301.07463, 2023.
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. ACM International Conference on Multimedia, pp. 638 647, 2022.
Antoine Miech, Dimitri Zhukov, Jean- Baa-tttt Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howto 100:: Learning text-video embedding by watching hundred million narrated video clips. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 2630-2640, 2019.
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid, Learning audio-video modalities from image captions. In European Conference on Computer Vision, pp. 407-426, 2022.
Tian Pan, Yibing Song, Tianyu Yang. Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 11205-11214, 2021.
Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Hen- riques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. arXiv preprint arXiv:2010.02824 2020.
AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unsupervised video representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 133-142, 2020.
Rui Qian, Yeqing Li, Liangzhe Yuan, Boqing Gong. Ting Liu, Matthew Brown, Serge Belongie, Ming-Hsuan Yang, Hartwig Adam, and Yin Cui. Exploring temporal granularity in self- supervised video representation learning. In British Machine Vision Conference, 2022.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Computer Vision, pp. 817-834, 2016.
12

---

Published as a conference paper at ICLR 2024
Andrew Rouditchenko, Angie Boggust. David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuchne, Rameswar Panda, Rogerio Feris, et al. Avlnet: Learning audio- visual language representations from instructional videos. In Interspeech, 2021.
Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. Look before you speak: Visually con- textualized utterances. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 16877-16887, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:1212.0444 2012.
Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. In NIPS, volume 35, pp. 38032-38045, 2022.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. C3d: generic features for video analysis. CoRR, abs/1412.00677, 2:8, 2014.
Haixin Wang, Xinlong Yang, Jianlong Chang, Dian Jin, Jinan Sun, Shikun Zhang, Xiao Luo, and Qi Tian. Parameter-effiieen tuning of large-scale multimodal foundation model. In NIPS, 2023a.
Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al. All in one: Exploring unified video-language pre-training. In CVPR, pp. 6598-6608, 2023b.
Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to- fine alignment for video-text retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2816-2827, 2023c.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William Freeman. Learning and using the arrow of time. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8052-8060, 2018.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM International Conference Multimedia, pp. 1645-1653, 2017.
Hu Xu, Gargi Ghosh, Po- Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichten- hofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic video-language model pre- training for video understanding. In Findings of the Association for Computational Linguistics, pp. 4227-4239, 2021a
Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zeulemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Empirical Methods in Natural Language Processing, 2021b.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. IEEE Conference on Computer Vision and Pattern Recognition, pp. 18134-18111 18144, 2022.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In IEEE Conference Computer Vision and Pattern Recognition, pp. 5288 5296, 2016.
Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem. G-tad: Sub-graph localization for temporal action detection. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 10156-10165. 2020.
Mengmeng Xu, Juan- Manuel Pérez-Rúa, Victor Escorcia, Brais Martinez, Xiatian Zhu, Li Zhang, Bernard Ghanem, and Tao Xiang. videos. In IEFE International Conference Computer Vision, pp. 7220-7230, 2021c.
pre-training for temporal localization in
13

---

Published as a conference paper at ICLR 2024
Mengmeng Xu, Juan Manuel Perez Rua, Xiatian Zhu, Bernard Ghanem, and Brais Martinez. Low- fidelity video encoder optimization for temporal action localization. In Advances in Neural Infor- mation Processing Systems, volume 34, pp. 9923-999999 9935,,955111 2021d.
Rui Yan, Mike Zheng Shou, Yixiao Ge, Alex Jinpeng Wang, Xudong Lin, Guanyu Cai, and Jinhui Tang. Video-text pre-training with learned regions. arXiv preprint arXiv:21 194, 2021.
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chieh Chen. k-means mask transformer. European Conference on Computer Vision, pp. 288-307, 2022.
Liangzhe Yuan, Rui Qian, Yin Cui, Boqing Gong, Florian Schroff, Ming-Hsuan Yang. Hartwig Adam, and Ting Liu. Contextualized spatio-temporal contrastive learning with self-supervision. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 13977-13986, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE International Conference on Computer Vision, pp. 6023 6032, 2019.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. IEEE Conference on Computer Vision and Pattern Recognition, pp. 18375-16387, 2022.
Can Zhang, Tianyu Yang, Junwu Weng, Meng Cao, Jue Wang, and Yuexian Zou. Unsupervised pre-training for temporal action localization tasks, In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14031-14041, 2022.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6586-6597, 2023.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In JEEE Conference Computer Vision and Pattern Recognition, pp. 8746-8755, 2020.
14

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 TEXT PROMPT TEMPLATES
we extract noun tokens by prompting noun phrases with pre-defined templates. We randomly select one from 12 templates to generate the prompt and details are pre- sented in Table
As mentioned in Section 3.3
Table 7: Prompt templates used for generating noun tokens.
Template
Prompts for noun phrases
A footage of {
2
A footage of the ()-
footage one 0-
4
A video a {}-
5
A video of the {)-
6
A video of one ()-
7
A portrait of a 0.
8
A portrait the ().
9
A portrait one {
10
A video footage of a (--
11
A video footage of the (-
12
A vidco footage of one {).
A.2 STRUCTURE OF GROUPING BLOCK
We demonstrate the structure of grouping block in Figure attention layer, in which attention scores are computed between group tokens as query and video tokens as value. The cluster assignment is computed via gumbel softmax over group tokens and converted into one-hot hard assignment.
3 It features a K-means clustering
Feed forward network
Gumbel
softmax
Multi head self attention
Lincar
Linear
Linear
K K-means clastering attention
000
0000
group
Figure 3:
The structure of grouping block. It is inserted at different layers of the video encoder to update group tokens by merging semantically similar video tokens.
A.3 DOWNSTREAM TASKS
Implementation details of fine-tuning the pre-trained model on downstream tasks are described in this section. During fine-tuning, resize video frames to 224 224 and sample 32 frames for each video. The maximum length for each caption is 32 by default, the same as the value in the pre-training stage. Specific optimization settings for each dataset are shown in Tables 8 I1
15

---

Published as a conference paper at ICLR 2024
Table 8: End-to-end fine-tuning configurations on MSR- VTT for text-to-video retrieval.
Config
MSR-VTT
optimizer
SGD
base learning rate
2.5e-1
optimizer momentum
0.9
learning rate schedule
cosine decuy
batch size warmup ratio
0.1
training epochs
Table 9: End-to-end fine-tuning configurations on MSRVTT-QA and MSVD-QA for VQA.
Config
MSRVTT-QA
MSVD-QA
optimizer
SGD
SGD
base learning rate
Ic-1
optimizer momentum
0.9
0.9
learning rate schedule
cosine decay
cosine decay
batch size
64
warmup ratio
training epochs
30
Table 10: Fine-tuning configurations on UCF101 and HMDB51 for video action recognition.
Config
UCF101
HMDBS1
optimizer
SGD
SGD
base learning rate
le-1
le-1
optimizer momentum
learning rate schedule butch size warmup ratio training epochs
cosine decay
cosine decay
64
0.1
30
Table 11: Fine-tuning configurations on ActivityNet for temporal action localization.
Config
ActivityNet
optimizer
Adam
base learning rate
4e-3
weight decay
lc-4
optimizer momentum
0190.9 9,09999
learning rate schedule batch size trannigg epochs
decay by -=0.1 every epochs
16
10
16

---

Published as a conference paper at ICLR 2024
B ADDITIONAL RESULTS
Number of frames. The number of frames used in pre-training vary among different methods. We follow the setting Nagrani et al. (0022) sample 32 frames for each video. Note that S-ViLM uses a vanilla vision transformer with temporal patch size 2, which effectively downsamples the video frames the first layer. Since no downsampling happens ALPRO and MCQ's encoder, their computational cost of 16 frames is comparable to S- S-ViLM with 32-frame input. In Table [2 we report 16 and 32-frame results and S- V-ViLM outperforms ALPRO and MCQ both settings. most methods use 16 frames during evaluation, we thus select to sample 32 frames to ensure fair comparison.
Table 12: Results with different number of frames.
MSRVTT-ZS
UCF101
Method
Input
R@1 1
R@5
R@ 10
Lin
FT
ALPRO
8- 8-frame
24.1
44.7
55.4
ALPRO
16-frame
24.7
55.0
MCQ
16-frame
89.1
92.3
S-ViLL
16-frame
28.5
53.5
64.0
93.1
96.0
S-ViLM
32-frame
28.6
53.5
65.1
94.8
Spatiotemporal action localization. We report experiment results of spatial temporal action local- ization task AVA v2.2 dataset. Results are presented in the table below:
Table 13: Results of spatiotemporal action localization on AVA v2.2. Two columns indicates using Detected and Grounded-truth boxes respectively.
mAP@0.5
Method
Detected
Ground-truth
SlowFast
23.80
MVTT-B
24.50
S- S-ViLM (contrastive only)
21.95
26.55
S-VILM
25.00
30.15
We can observe that S- S-ViLM outperforms other models and the variant with contrastive loss only when either detected boxes or ground-truth boxes are used for evaluation. This experiment demon- strates the effectiveness of the spatiotemporal learning design of our method.
C VISUALIZATION
We present the visualization of spatial grounding in Figure For each example, we choose the group token which has the maximum similarity score of the target noun phrase, and compute the attention heatmap based on corresponding video tokens assigned to that group token. It can be observed that the alignment between the region and the noun phrase has been learned during the pre-training stage without any fine-grained annotations. In addition, more comparisons between similarity scores of baseline features and temporal- anwerel--wware features provided in Figure 5 With temporal grouping, features from different scenes are much easier to distinguish.
17

---

Published as a conference paper at ICLR 2024
Cagtion: A mun throws bowling ball into various pine..
Caption: The beat makes turn to the right.
Caption: spins
Cuption:
Figure 4: Visualization of spatial grounding. The attention feature map of each example is computed from the corresponding regions assigned to the group token which achieves the highest similarity score with respect to the target noun phrase.
I
peneil, one green pencil, on craser on the table.
Caption:
Captions out water
background
AUT
10
A22
10
6/1
Similarity scores of baseline feutures
Similarity scores of somporal features
Similarity scores of baseline feutures
Similarity seores of temporal festurus
Caption: A man was driving tractor for land reclamation.
Caption: A penson cut the paper to make giä box.
10
7.58
Simularity scores of baselime feutunes
Similarity sccces of lempural-swar ficttures
Similarity scoces of buseline fenturrss
Similarit scores of temporal comporal-aware features
Figure 5: Visualization of temporal grouping.
18


Reviews:
[{"review_id": "k1NQaiVzSN", "reviewer": "Reviewer_adH9", "summary": "The paper addresses improving video-language alignment. The paper introduces two novel pre-training tasks, namely, inter-clip spatial grounding and intra-clip temporal grouping. The inter-clip spatial grounding aims to associate relevant image regions with text. This task is weakly supervised, without explicit correspondences between regions and nouns. Instead, it employs learnable 'group tokens' to cluster semantically similar image regions.The intra-clip temporal grouping optimizes the model features to be able to distinguish a video clip (start/end time) from a background clip, akin to a metric learning loss approach. The benefit of these tasks is that the supervision can be generated automatically using random cut & paste and pre-processing the captions. The experimental results show that the proposed method outperforms VCC (Nagrani et al., 2022) on multiple benchmarks. The ablation studies on the two introduced losses indicate that each contributes to improving the original method. However, when combined, their combined effect results in a marginal improvement compared to employing each loss separately.", "questions": "Q1. See W2 (1). \nQ2. In Eq 1, were z_i’s L2 normalized?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. The proposed additional pre-training tasks (inter-clip spatial grounding and intra-clip temporal grouping) are reasonable. The supervision can be generated automatically using random cut & paste and pre-processing the captions for extracting the nouns. \n\nS2. I liked that the authors provide Table 5 that shows the effects on different choices of pre-training datasets. This allows comparing the proposed method and VCC trained on the same dataset (HowTo100M and VideoCC). It is good to know that the proposed method is on par with VCC when trained on HowTo100M, but significantly outperforms VCC when trained on VideoCC on MSRVTT-ZS. \n\nS3. The authors provide an ablation study on combinations of their proposed losses.\n\nS4. The paper presents visualizations of the affinity scores of learned features, alongside the attention map of S-ViLM in Figure 2. These visualizations are interesting and further validate that the model was trained as intended.", "weaknesses": "W1. The authors state that most video-language pre-training methods neglect scene/action changes along the time in a video. However, there are works like LaViLa [a] that takes temporal and patch-level information into account. \n[a] Learning Video Representations from Large Language Models, Zhao et al., CVPR 2023\n\nW2. Some design choices are not obvious but no ablation study was presented in the paper.\n(1) Instead of Eq 1-2, I am curious if the authors tried BCE loss on z_i clip using the mask m_i. If so, how did the performance differ?\n(2) Instead of using grouping blocks, one can opt for existing region proposal networks or segmentation networks to obtain semantically similar regions, and then aggregate visual features within those regions to compute relevance against the nouns. \n\nW3. The writing could be further improved for enhanced clarity\n\n- It would be nice if \"interaction\" is clarified in “c) modality fusion and interaction” in the second paragraph of Section 1. \n- The term \"region-object groundingness\" is ambiguous in “Specifically, group tokens aggregate semantically similar video tokens via grouping blocks to promote region-object groundingness and video tokens are utilized in temporal grouping to improve temporal awareness.”\n- I suggest using “semantically similar regions” in “Thus, we adopt M learnable group tokens to cluster semantic similar regions in a self-supervised manner.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCg2ukP5l8", "reviewer": "Reviewer_tkWv", "summary": "The paper proposed a new framework, namely S-ViLM, for video-language modeling. \n\nThe core ideas include:\n\n- Perform intra-clip temporal grouping by appending clips from other videos to the start and end (cut-and-paste) and classifying each clip as foreground / background in a self-supervised manner based on feature similarity\n\n- Perform inter-clip spatial grounding with the help of grouping blocks and tokens, based on feature similarity between prompts of nouns identified by spaCy and grouped tokens\n\n- Joint pretraining by learning intra-clip temporal grouping,  inter-clip spatial grounding, and general video-text contrastive matching\n\nPretrained on VideoCC and ActivityNet-Caption, the model achieved competitive performance against previous state-of-the-arts on four downstream tasks including text-video retrieval, VQA, video action recognition and temporal action localization. Ablation studies demonstrate some important design choices and present some details on the effects of each part.", "questions": "Please address my concerns according to the Weaknesses part.\n\nBesides what are listed there, I also have some other questions:\n\n- For video action recognition, I wonder if the authors have results on the Something-Something / Something-Else dataset. UCF101 and HMDB51 are both quite biased towards objects presented in the videos and therefore may benefit a lot from the inter-clip spatial grounding. I'm very curious on S-ViLM's performance on a more different action recognition dataset.\n\n- How are start and end clip indices s, e sampled?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed methods are well-motivated and bring something new to video-text modeling. The intra-clip temporal grouping with cut-and-paste is an effective way of better capturing scene changes and the inter-clip spatial grounding with grouping blocks and tokens is an interesting way to enhance object (noun) understanding.\n \n- The comparison with previous state-of-the-arts on four different downstream tasks suggest the effectiveness of the proposed method on learning more representative features.\n\n- Ablation studies in Fig. 2 show that intra-clip temporal grouping helps the model better distinguish different scenes and inter-clip spatial grounding  leads to features that are sensitive to objects, both contributing to performance improvement as in Tab. 6. \n\n- Good details are provided, which can make reproduction easier", "weaknesses": "- Extra knowledge about determining noun chunks from spaCy is introduced during pretraining, which may lead to some level of unfair comparison with other methods\n\n- The concern regarding benefits from pretraining set are not fully addressed. Although Tab. 5 shows that S-ViLM obtains better results than VCC when both pretrained on VideoCC, it is still unclear how much benefits are brought when comparing with other methods in those downstream tasks. This may also result in unfair comparison to some extent.\n\n- The authors claim that Eq. 3 \"encourages each noun to be grounded to one or a few regions and avoids penalizing regions\nthat cannot find any relevant nouns.\" I want to hear more explanations and discussions on this. In particular, how it can prevent n_k being similar to many regions, or even all regions uniformly.\n\n- I suggest not listing all baseline names in the main text but just citing them in the Table to save space for more important discussions and details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n1vAsAf0ot", "reviewer": "Reviewer_Dxbj", "summary": "Compared with existing video-language pre-training tasks, the authors focus on instance-level alignment with spatial and temporal granularity instead of global contrastive learning. Specifically, the authors propose two pre-training tasks, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features simultaneously. The experiment results empirically demonstrate the effectiveness of the proposed framework.", "questions": "Please see the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The design of pre-training tasks sounds technically works. And the details of it are comprehensive.\n2. The comparison and ablation study are comprehensive. The visualizations clearly present the actual contribution of inter-clip spatial grounding and intra-clip temporal grouping.\n3. The overall presentation is clear and easily understandable.", "weaknesses": "1. The novelty may be somewhat weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h292cV6FX0", "reviewer": "Reviewer_7b3w", "summary": "The paper proposes a new framework S-ViLM for video-language pre-training, which aims to improve the fine-grained understanding of both videos and text. The framework consists of two novel components: inter-clip spatial grounding and intra-clip temporal grouping. The former learns to align video regions and text objects across different video clips, while the latter learns to group video frames and text tokens within the same clip based on temporal coherence. The paper evaluates S-ViLM on four downstream tasks that require temporal localization and semantic reasoning, such as text-video retrieval, video question answering, video action recognition and temporal action localization. The paper shows that S-ViLM outperforms existing methods on these tasks and achieves state-of-the-art results. The paper also provides an extensive ablation study to demonstrate the effectiveness of each component of S-ViLM.", "questions": "- The methods being compared are outdated. It is recommended to compare them with state-of-the-art (SOTA) methods, particularly in the MSRVTT and TAL tasks. The authors can refer to https://paperswithcode.com/sota for comparing with more recent methods.\n- The text may involve multiple concepts. How can we extract k noun tokens from all sentences?\n- The number of group tokens remains constant during training. How do the group tokens align with different nouns in different sentences?\n- The visualization results for group tokens and noun tokens should be included.\n- It may be beneficial to include a grounding task since spatial alignment is claimed to be achieved.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The manuscript effectively conveys the concept of aligning fine-grained video and text features.\n- The use of group tokens in the video encoder to align with the concepts in the text is a noteworthy approach.\n- The proposed loss function significantly enhances zero-shot retrieval performance on the MSRVTT dataset.", "weaknesses": "- The representation of spatial concept features by the group tokens is unclear.\n- The compared methods are outdated and do not compare to state-of-the-art methods.\n- The claimed cut-and-paste operation has been widely used in previous studies.\n- The performance of the current methods on several tasks is inferior to many recent works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)