Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Authors: Zhenhailong Wang
Manling Li
Ruochen Xu
Luowei Zhou
Jie Lei
Xudong Lin
Shuohang Wang
Ziyi Yang
Chenguang Zhu
Derek Hoiem
Shih-Fu Chang
Mohit Bansal
Heng Ji
Venue: NeurIPS 2022 Accept (2022)
Keywords: video-language
few-shot
language model
in-context learning

Abstract: The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining or finetuning on any video datasets. We use image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into a temporal-aware template.  We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility of prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models in understanding videos on a wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the-art supervised models trained on large-scale video datasets.
Code and processed data are publicly available for research purposes at https://github.com/MikeWangWZHL/VidIL. 

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Zhenhailong Wang'; Manling Li", Ruochen Xu', Luowei Zhou", Jie Lei', Xudong Lin", Shuohang Wang", Ziyi Yang', Chenguang Zhu?, Derek Hoiem', Shih-Fu Chang', Mohit Bansal', Heng Ji' UIUC MSSR 'UNC *Columbia University edu
Abstract
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples. Existing few-shot video-language learners focus exclusively
the encoder, resulting in the absence of video-to- text decoder to handle generative tasks. Video captioners have been pretrained large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidII. few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining finetuning on any video datasets. We use Image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into temporal-awarc template. We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models understanding videos on wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the- supervised models trained on large-scale video datasets. Code and processed data are publicly available for research purposes at httph:://// /htb.tt...u/.. wtttttu..oooom..teee www/Miikoaaaaaaaaaaccc /ww.///eeaaanng//aiil.
1 Introduction
One major gap between artificial intelligence and human intelligence lies in their abilities to generalize and perform well on tasks with limited annotations. Recent advances in large-scale pre-trained generative language models [45, 6, 71, 24] have shown promising few-shot capabilities [72, 43, 63] in understanding natural language. However, few-shot video-language understanding is still in its infancy. A particular limitation of most recent video-language frameworks [28, 21, 61, 68, 67, 25, 64, 34] is that they are encoder-only, which means they do not have the ability generate
from videos for purposes such as captioning [62, 57J, question answering [60], and future prediction [23]. Meanwhile, unified video-language models [36, 49] that are capable of language decoding still rely heavily on finetuning using large number of manually annotated video-text pairs. therefore cannot adapt quickly to unseen tasks. Few-shot video-to-text decoding challenging because the natural language supervision for learning video-language representation is typically based on subtitles and automatic
"Equal contribution. Currently Google Brain.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

speech recognition (ASR) transcripts [39, 68], which differ significantly from downstream tasks in terms of distribution and may have poor semantic alignment across vision and text modalities.
We propose to address this problem by harnessing the few-shot power of frozen large-scale language models, such as InstructGPT [40]. Our inspiration is derived from the fact that humans are excellent visual storytellers [15]. with the ability to piece together coherent story from few isolated images. To mimic this, we propose VidIL, a few-shot Video-language Learner via Image and Language models, to use image models to provide information about the visual content in the video (as well as optionally use ASR to represent speech), and then we instruct language models to generate a iideo-based summary. answer, or other target output for diverse video- aado-language tasks.
The main challenge of understanding videos is that, videos contain rich semantics and tem- poral content at multiple granularities. Unlike static images which depict objects, attributes and events in snapshot, the temporal dimen- sion of videos further conveys the state changes of the objects, actions, and events. For exam- ple, in Figure 1, the individual frame captions of the video clip only describe static visual fea- tures such as "a person holding a green object in hand". In contrast, correct video-level de- scription would be "a woman makes realistic looking leaves and flowers for a cake", which involves reasoning over a collection of objects and events that occur at different timestamps in the video clip, such as "cake decorating" and "flowered design". Hence, to inform video-level description and queries, we need to represent all of this information and its temporal ordering.
Input Video
Image Laage-Language Model
Visual
Objects
decorating, sugar poste. city animation, payy-doh...
Token
Events
outing
shaped
Rlwored
Attributes made Fondontt edging, nibbingg paper green goo
Frame
Level
Frama
Captions
green ofjectin
person holding
hand
green leat on baby's a head
penson is putting
place of paner with pair ssors
person cutting
Language Model
Gurs: a perton making a cake out of fondant and day, and then docorazing with doll os and leaves
Video
Video
Ground Truth 1: women mekees mecibeli looking leaoes and nowers for cole
Captions
Ground Truth 2: woman creating londart buby and flower
Figure 1: Multiple levels of information in videos.
To address the unique challenges of videos, we propose to decompose video into three levels: the video output, frame captions, and visual tokens (including objects, events, attributes). One major benefit from this hierarchical video representation is that we can separate the visual and temporal dimensions of video. We leverage frozen lmage-language foundational models lower levels to collect salient visual features from the sparsely sampled frames. Specifically, we first leverage pretrained image-language contrastive model CLIP [44] to perform visual tokenization, based on the similarity score between frames and tokens of objects, events and attributes. The tokenization is done under the guidance of semanties role labeling [14], which provides us with candidate events with involved objects and related attributes. Next. in order to capture the overall semantics at the frame level, we employ the pretrained image captioner in the image-language model BLIP [26] to obtain frame captions. Finally, instruct a pretrained large language model using in-context learning [40, 13, 51, 48] to interpret visual tokens and frame captions into the target textual output. In detail, we temporally order visual tokens and frame captions using specially designed prompts such as "First... The....
to instruct the pretrained language model to track the changes of objects, events, attributes and frame semantics along the temporal dimension.
Without pretraining or finetuning on any video datasets, we show that our approach outperforms both video-languse and image-language state-of-the-art baselines few-shot video captioning and question answering tasks. Moreover, on video-language event prediction, our approach significantly outperforms sully-supervised models while using only 10 labeled examples. We further demonstrate that our generative model can benefit broader video-language understanding tasks, such as text-video retrieval, via pseudo label generation. Additionally, we show that our model is highly flexible in adding new modalities, such as ASR transcripts.
2 Related Work
2.1 Image-Language Models and Their Applications on Video-Language Tasks
Large-scale image-language pretraining models optimize image-text matching through contrastive learning [44, 17] and multimodal fusion [65, 27, 58, 66, 35, 52, 8, 29, 73, 70, 18, 16]. Recently,

---

BLIP [26] proposes a bootstrapping image-language pretraining framework with a captioner and filterer which has shown promising performance on various image-language tasks. However, video-language pretraining [25, 36, 28, 38, 3, 1, 42, 33] is still hindered by noisy and domain-specifee video datasets [74, 22, 39]. Naturally, researchers start to explore transferring the rich knowledge from image models to videos. Different from the traditional way of representing videos by 31) dense features [12], recent work [21, 25] proves that sparse sampling is an effective way to represent videos, which facilitates applying pre-trained image-language models to video-language tasks [37, 11]. Specifically, the image-language model BLIP [26] sets new state-of-the-e on zero-shot retrieval-style video-language tasks, such video retrieval and video question answering. However, for generation- style tasks such as domain-specific video captioning, lideo-language model UniVL [36] still leads the performance but highly rely on fine-tuning. In this work, we extend the idea of leveraging image-lang language models to a wide variety of video-to-text generation tasks. We further connect image- language models with language models which empowers our model with strong generalization ability. We show that the knowledge from both image-languaal pretraining and language-only pretraining can benefit video-language understanding in various aspects.
2.2 Unifying MultiModal Tasks with Language Models
The community has paid much attention to connecting different modalities with a unified representa- tion recently. Text-only generation models, such as T5 [46], have been extended to vision-language tasks by text generation conditioned on visual features [9, 53, 50, 75, 55]. In order to fully leverage the generalization power from pretained language models, [63] represents images using text in fully symbolic way. [32] includes more modalities such as video and audio, but requires annotated video-text data to jointly training the language model with the video and audio tokenizer. In this work, we propose a temporal -aaprr aware hierarchical representation for describing video textually. To our knowledge, we are the first work to leverage prompting a frozen language model for tackling few-shot video-language tasks with unified textual representation. Concurrent work Socratic [69] uses a zero-shot language-based world-state history to represent long videos with given time stamps, while our model can quickly adapt to different video and text distributions with few examples. Furthermore, we show that by injecting temporal markers to the prompt we can make a pre-trained language model understand fine- grained temporal dynamics in video events. Compared with the concurrent work Flamingo [2], which requires dedicated vision-language post-pretraining. our framework does not require to pretrain or finetune on any video data. Our framework is simple and highly modulated where all the components are publicly available. Additionally, our framework is more flexible on adding new modalities, c.g., automatic speech recognition, without the need for complex redesigning.
3 Method
We propose a hierarchical video representation framework which decomposes a video into three levels, i.e., visual token level, frame level and video level. The motivation is to separate the spatial and temporal dimension of video in order to leverage image-Language and language-only foundation models, such as CLIP [44] and GPT-3 [6]. All three levels use a unified textual representation which enables us to leverage the powerful few-shot ability from pretrained language models.
3.1 Frame Level: Image Captioning
Following [21] we first perform sparse sampling to obtain several video frames. Unless otherwise specified, we sample frames for frame level and frames for visual token level. We then feed each frame into pre-trained Image-language model to obtain frame level captions. An example can be found in the blue part of Figure 2. In our experiments, we use BLIP [26], a recent image-language framework containing both -oogee-goonded encoder and decoder, for generating frame captions. We follow [26] to do both captioning and filtering on each frame. However, as mentioned in Section 1, videos contain rich semantics and temporal contents at multiple granularities. It is not enough to generate video-level target text such as video captions solely based on frame captions. Thus, we further perform visual tokenization for each frame to capture features at finer granularity.
3

---

Image- Language
Objects
Events
Attributes
Visual
Model
Tokenization
Frnt: bath caddy, baih loy, loy chest, push & pull on Frm2: adhesive bandage, band aid, bady soddler shoe FmmN: shon cara, adhasiva handage, snakubbiie..
Visual Token Level
Semantic Role Labeling
Frame Caption
Frame Caption
Image- Language Model
Generation
Frmt: loddler playing a bathtub filled with toys Frm2: toddler playing in bathtub with toys
Frame Level
Sampled Frames
and Filtering
Frm3: <filered out>
FrmN: boy sitting the floor with his foot toilet
Task: Video Captioning
Task: Video Question Answering
Sparse Sampling
Task Instruction
Genorate caption based onjects, events, subtite and frame captions. Example:
Answer the question based on the objects, events, subtill frame captions. Example:
Objocts: First, bath toy. Then, toddler. After mhat adhesive bandage
Events: First, playing with
Then, wearing sandal
Video Level
Few shot Context
Attributes: First, red loy. Then, disper chenging. After that, band aids
Frame Captions: First, loddler playing in bathtub filled toys. Then, toddler playing in
bathubb with Finalty, bay sitting floor with his foot
Subtitle: <ASR Transcript>
Input Video
Task Query
Caption:
Question: what a littl kid playing with his toys?
caxample captions None
Answer: Kestampl answerk Nore
ASR
Language Model
Input Speech
Oulpul
Target
a boy in the tub with toys and getting his foot bandaged
Output Caption
bathtub
Answer
Output
Temporal-Aware Few-shot Prompt
Figure 2: Overview of VidIL framework. We represent a video in a unified textural representation containing three semantic levels: visual token level, frame level, and video level. At visual token level, we extract salient objects, events, attributes for each sampled frame. At frame level, we perform image captioning and filtering. At video level, we construct video representation by aggregating the visual tokens, frame captions and other text modalities such as ASR, using a few-shot temporal- aware prompt. We then feed the prompt to pre-trained language model together with task-specific instructions to generate target text for variety of video-language tasks, Examples of the full prompt for different tasks can be found in Appendix ??.
3.2 Visual Token Level: Structure-Aware Visual Tokenization
At this level, we aim to extract the textual representations of salient visual token types, such as objects, events and attributes. We found that pre-defined classes for classification, such as those in ImageNet [10], are far from enough for covering the rich semantics in open-domain videos. Thus, instead of using classification-bast adopt retrieval-based visual tokenization approach by leveraging pre-trained contrastive image- language models. Given a visual token vocabulary which contains all candidate object, event, and attribute text phrases, we compute the image embedding of frame and the text embeddings of the candidate visual tokens using a contrastive multi-modal encoder, CLIP [44]. We then select top visual tokens per frame based on the cosine similarity of the image and text embeddings. An example of the extracted object tokens can be found in the green part of Figure 2.
methods for visual tokenization as in previous work [32, 63],
Unlike in images where objects and attributes already cover most visual features, events are more informative in videos. In order to discover events from video frames, we construct our own event vocabulary by extracting event structures from Visual Genome [19] synsets using Semantic Role Labeling. Specifically, we first select the phrases that contain at least one verb and one argument as events. Then we remove highly similar events based on their sentence similarity using Sentence- BERT [47] embeddings. For object vocabulary, we adopt OpenImage [20] full classes (~~20k), instead of using the visually groundable subset (~600) as in concurrent work [69]. We found that using large but noisy vocabulary is more effective than using small but clean vocabulary in our retrieval- based setting with CLIP. For attribute vocabulary, we adopt visual genome attribute synset. In Section 4.6, we provide ablation study on the impact of different types of visual tokens. The statistics of visual token vocabulary be found in Appendix Table ??.
We use the keys in Visual Genome [19] object synsets which contains frequent <verb,object> pairs.

---

change ordering of the objects and frame captions
Static Prompt
Objects: sun moving night
Captions sun oceen, shining al right
Objects sky,
Frame Daplions: shining
Video Caption:
Video Caption
generation:
The sun is setting over the ocean and the stars are shining night.
The sun is setting over the ocean the stars start to shine in the night sky.
Temporal-
Objects: First, aun moving: Then, right sky.
Object:: Capliara: Caption:
moxing.
Aware
Frama Captaan:: First, over an ocaan. Then, stars shining night.
shining right.
Prompt
Video Caption:
generation:
sun setting over the ocean, and the stars are shining night, The stars are shining night, and sun rising oieer the ocean.
Expected caption: Sunset
Expected caption: Sunrise
Figure 3: Temporal-awe -waare prompt successfully distinguishes the Sunset and Sunrise scenarios based on the temporal ordering change of objects and frame captions, while the static prompt fails.
3.3 Video Level: Temporal-Aware Few-shot Prompting
Once we obtain the textual representation from frame level and visual token level, the final step is to put the pieces together to generate video level target text. The goal to build model that can be quickly adapted to any video-to-tex generation task with only a few examples. To this end, we propose to leverage large-scale pre-trained language models, such as GPT-3 [6], with a temporal-aware few-shot prompt. As shown in Figure 2, our framework can be readily applied to various video-to-text generation tasks, such as video captioning and video question answering, with a shared prompt template. The proposed prompting strategy enables a language model to attend to the lower level visual information as well as taking into account the temporal ordering.
Here, we use the video captioning task depicted in Figure 2 to illustrate the details. The few-shot prompt consists of three parts: instruction, few-shot context, and task query. The instruction is concise description of the generation task, e.g., "Generate a video caption based on the objects, events, attributes and frame captions. Example:" which is proved to be effective in zero-shot and few-shot settings [6, 59]. The few-shot context contains the selected in-context examples as well as the test video instance. Each video instance is repre- sented by the aggregated visual tokens", e.g., "Objects: First, bath toy. Then, the frame captions, such as "Frame Captions: First, toddler playing in a bathtub and the ASR inputs if available, e.g., "Subtitle: <ASR Transcript>". Finally, the task query is a task-specific suffix indicating the target text format, e.g. "Video Caption:" " For in-context examples (omitted here for simplicity), the task query is followed by ground truth annotation, while for the test instance, the generation starts at the end of the task query.
filled with toys. Then,...
Formally, we denote the instruction line as t, few-shot context as c, the task query as q, and the target text as y, where y = (y1, .2.... Vr.) The generation of the next target token y can be modeled as:
y/1 arg max p(y|s, C, V<))
In order to capture the temporal dynamics between frames and visual tokens, we further propose to inject temporal markers to the prompt. As shown in the few-shot context in Figure 2, each visual token and frame caption is prefixed with a natural language phrase indicating its temporal ordering. e.g., "First, "aarss,,"TTen,, ", and "Finally, We found adding the temporal marker can make the language model conditioned on not only literal but also temporal information of the context. We show an example in Figure 3, where compare our temporal-aware prompt with static prompt on video captioning using InstructGPT. Again, the in-context examples are omitted here, which can be found in Appendix ??. In this example, the only difference between these two contexts is the ordering of the visual tokens and the frame captions. For the context on the left, where "sun moving" appears before "night sky", we are expected to see story talking about sunset, while for the context on the right, we are expected to see sunrise. We can see the static prompt generates captions about sunset for both contexts, while the temporal-aware prompt can capture temporal ordering correctly and generate sunrise for the context on the right.
"To obtain video level visual tokens, the visual tokens extracted from each frame are further ranked and ordered based on frequency and frame index. More details can be found in Appendix ??.
5

---

Experiments
4.1 Experimental Setup
To comprehensively evaluate our model, show results four video-language understanding tasks in few-shot settings: video captioning, video question answering (QA), video-language event prediction, and text-video retrieval. We compare our approach with state-of-the-ee art approaches on five benchmarks, i.e, MSR- VTT [62], MSVD [7], VaTeX [57], YouCook2 [74], and VLEP [23]. The
statistics of the datasets can be found in Table For more details please refer Appendix ??.
Implementation Details. We use CLIP-L/14 as our default encoder for visual tokenization. We adopt BLIP captioning checkpoint fine-
Table I: Statistics of datasets in our experiments
Split Count train eval
Dataset
Task
tuned on COCO [31] for frame captioning. We use InstructGPT [40] as our default language model for generating text conditioned on the
MSR- VTT [62]
Captioning: QA
6,513 / 2,990
MSR-VTT [62]
Retrieval
7,010/ / 1,000
MSVD [7]
Question Answering
30,933 / 13,157
few-shot prompt. To construct event vocabulary, use the semantic role labeling model from AllenNLPS. The experiments are conducted on 2 NVIDIA V100 (16GB) GPUs. All few-shot finetuning on baselines and semi-supervised training are performed on 2 Nvidia V100 16G GPUs.
VaTeX v1.1² [57]
Captioning: Retrieval
25,991 6,000
YouCook2 [74]
Captioning
10,337 / 3,492
VLEP [23]
Event Prediction
20.142 / 4,192
In-context Example Selection. From our preliminary experiments, we find that the generation performance is sensitive to the quality of in-context examples. For example, for QA tasks such as MSVD-QA where the annotations are automatically generated, the <question, answer> pair in randomly selected in-context examples can be only weakly-correlated with the video context. Thus, instead of using a fixed prompt for each query, we dynamically filter out the irrelevant in-context examples. Specifically, given a randomly sampled M-shot support set from the training set, we select a subset of N-shots as in-context examples based on their SentenceBERT [47] similarities with text queries. Furthermore, score to account for the recency bias [72] in large language models. For QA tasks, we choose the most relevant in-context examples by comparing with questions. While for captioning task, compare with frame captions. not otherwise specified, we use M=/0 and N=5, which we consider 10-shot training.
reorder the selected examples in ascending order based on the similarity
4.2 Few-shot Video Captioning
We BLEU-4 [41], ROUGE-L METEOR [5], and CIDEr [54] scores on three video caption- ing benchmarks covering both open-domain (MSR-VTT VaTeX) and domain-specific (YouCook2) videos. We compare with both state-of-the-art video captioner (UniVL [36]) and image captioner (BLIP [26]). In order to implement the BLIP baseline for few-shot video captioning, we extend the approach used for text-video retrieval evaluation in [26] to video-language training. Specifically, we concatenate the visual features of sampled frames and then feed them into the image- grounded text-encoder to compute the language modeling loss. This is equivalent to stitching the sampled frames into a large image and then feeding it to BLIP for image captioning. We found that this simple approach results in very strong baselines.
As shown Table 2, existing methods have strong bias on certain datasets. For example, Vn performs well YouCook2 but fails MSR-VTT and VaTeX, while BLIP performs the oppo- site. This is because Vn is pretrained on HloToo00MM which favors instructional videos, i.c., YouCook2, while BLIP is pre-training i.., MSR-VTT and VaTeX. On the contrary, our model performs competitively on both open-domain and instructional videos, and significantly outperforms the baselines on the average CIDEr score across all three benchmarks. This indicates that by leveraging language models, we can maintain strong shot ability regardless of the video domain or the target caption distribution.
image-caption pairs which favors description-style captions,
wttpe//aaaccccoooceeeeeeeee co/ co/ppppaal/lllp--t---ag.....eeee 7 /nttps://github http://hw...ttttaaaoe/ee/eeeeaaa chw.kooinnccccccccccccccccec "nttps://docs.allll http://www..aaaaaaa.///////tt/ta srl/
"nttps://buggingface

---

Table 10-shot video captioning results. indicates concurrent work. The reported Flamingo [2] results using 16 shots. #Videopr represents the number of videos used for pre-training. B-4, R-L,, M, C represents BLEE-4, ROUGE-I, METEOR and CIDEr. Avg C represents the average CIDEr score across all available benchmarks. ASR indicates whether the model has access the ASR subtitles. BLIP and BLIP cup use the pretrained checkpoint and the finetuned checkpoint on COCO captioning. All results are averaged over three random seeds.
MSR-VTI VTT Caption B-4 R-L
YouCook2 Caption B-4 R-L M B-4
VaTTex Caption R-L
Method
#Videorr ASR
Avg
M C
M
C
Few-shor
UniVL BLIP
1.2M
No
2.1
22.5
9.5
3.6
25.3
11.6
34.1
1.7
15.7
8.0
2.1
13.3
No
27.7
43.0
23.0
39.5
0.7
9.0
3.4
11.5
13.5
39.5
15.4
20.7
23.9
BLIP cap VidlI[ours)
0
No
21.6
48.0
22.7
30.2
3.7
8.6
3.8
9.4
20.7
41.5
17.4
28.9
22.8
No
26.0
51.7
24.7
36.3
2.6
22.9
9.5
27.0
22.2
43.6
20.0
36.7
33.3
UniVL
1.2M
Yes
4.3
26.4
12.2
48.6
2.7
17.7
10.2
3.4
26.0
(ouls(
0
Yes
10.7
35.9
19.4
111.6
23.2
44.2
20.6
38.9
75.3
Flamingo-3B(16)
27M
No
73.2
57.1
Flamingo-80B(16)
27M No
84.2
62.8
Fine-tuning
UniVL
1.2M No
42.0
61.0
29.0
50.1
11.2
40.1
17.6
127.0
22.8
38.6
22.3
33.4
70.2
UniVL
1.2M
Yes
16.6 45.7
21.6
176.8
23.7
39.3
22.7
35.6
106.2
As discussed in Section 1, video captions describe the content in various semantic levels. The N-gram based metric may not fairly reflect the models' performance in capturing the video-caption alignment. We further verify this hypothesis in Section 4.5. Thus, in addition automatic metrics, we include qualitative examples illustrated Figure 4. More examples are in Appendix ??.
Additionally, for most existing methods and also concurrent work, e.g.. Flamingo [2], adding new modality often requires a dedicated model redesign or retraining. However, the nature of our framework, where we use a unified textual representation for each level, makes it highly flexible for incorporating new modalities. As shown in row Table, our model can effectively utilize extra information from ASR to obtain significantly better few-shot performance certain datasets such as YouCook2.
MSR-VTT Caption
YouCook2 Caption
VaTex Caption
drink up
Objects:
First, interview: Then, cable
Objects:... Events:... Attributes:... Captions:... Subtitle:
Objects:
Attributes: First, taoging. Then, woodburning. 'Aller that, wood burning. luming
Events:
lelevision. After That. television program.
going saueages prelly out time.
Finally, sports commentator.
Captions: piace wood with words drink written
Events: ... Attributes: ... Frame Captions:
we're going to now, my cat gravy as source.
UniVL: man is playing
with a man
add sausages Ours: take the sausages out of the pan
UnIVL: you re ready to decorate your cako
BLIP: man in suit and sitting on a couch
BLIP: a person holding a string with small object in
Ours: an interview with a sports
gravy plate
Ours: front them
making Up" with wood burning kit.
"Drink
commentator
Ground Truths:
Ground Truth:
Someone
burning
men are discussing sports on talk show
remove sausages from pan
tool to burn design into slice of wood and begine to brush polyurethae
being interviewed tv show
Figure 4: Qualitative examples video captioning. Grey boxes contain part of the video repre- sentation from our model. Blue boxes contain caption generation from different models. Green boxes contain ground truth annotations. Bold green text highlights the correct information that captured in baseline outputs which can be reasoned from our visual tokens and frame captions.
4.3 Few-shot Video Question Answering
We compare the test accuracy of our approach with few-shot pretrained BLIP, BLIPvQa [26], and concurrent work Flamingo [2] on two video question answering benchmarks, ASSR-VTT. _A and MS. BLIP VQA represents finetuned BLIP VQA [4] dataset, which is the previous SOTA
7

---

on zero/few-shot video question answering. In order to have fairer comparison with BLIP QA. we reduce the shot number to 5 and report the average accuracy on three sets of randomly selected 5-shot examples. As shown in Table 3, our method outperforms previous SOTA by a large margin. Comparing with concurrent work Flamingo, which is post-pretrained on a large number of video-text data, our model is training-free and did not observe any video data. However, with only image- language and language-only knowledge, our 5-shot model is able to outperform 8-shot Flamingo-3B and achieve on-par performance with 4-shot Flamingo-80B.
Table Video QA results. BLIP QA is finetuned on VQA [4]. in- Table 4:
Accuracy (%) on
dicates concurrent work. PT, FT indicates pretraining and finetuning.
VLEP hidden test set.
Method
#vidcoer
#videoFr
MSR-VTI
MSVD
Method
#videoyr
Ace
BLIP
0
0-shot
0.55
0.45
VI_P [23]
20142
67.5
BLIP
5-shot
0.84
0.53
MERLOT [68]
20142
68.4
BLIPvQA [26]
0-shot
19.2
35.2
VidlL(ours)
0
5-shot
21.2
39.1
VidIl
10-shot
72.0
Flamingo-33 [2]
27M
4-shot
14.9
33.0
Human
90.5
Flamingo-3B [2]
27M
8-shot
19.6
37.0
Flamingo-800 -0B [2]
27M
4-shot
23.9
41.7
Flamingo-80B [2]
27M
8-shot
27.6
45.5
ALPRO [25]
2M
full-shot
42.1
45.9
4.4
Few-shot Video-Language Event Prediction
In this section, we show that our model not only can answer questions about the video visual features but also answering "What is more likely to happen next?". Given a video with associated subtitle transcript as premise, the video-language event prediction (VLEP) task is predict the most likely future event. The original VLEP [23] paper formulates the problem where the model will be chosen from two possible future event candidates. Instead, we formulate this problem as another video-to-text generation problem to fit into our framework. Figure 5 depicts an example with the same format in Figure Similar to the evaluation setting in QA, the generated free-form text will first be mapped to one of the two candidate answers using SentenceBert [47], and then calculate the accuracy. In Table 4, we report accuracy on the hidden test set of VLEP [23]. To our surprise, our 10-shot model outperforms state-of-the-att fully-supervised baseline, i.e., MERLOT [68], by a large margin (~ 4%). This shows that our model has strong few-shot ability not only on video- language understanding but also prediction. Since event prediction tasks rely heavily temporal ordering, we show that with the proposed temporal-ay cemporal-wware prompting, language models can guided capture temporal dynamics between historical and future events.
binary classification problem
Task: Video Language Event Prediction (VLEP)
Instruction
Predict what is more likely happen
the captions and dialogue. Example:
Few-shot
Frame Captions: First, a man a black shirt is serving food. Then, group of people in kitchen preparing food
Context
Dialogue: took portion of it you can smell the onions. You can amell kind of spice like cinnamon little bit
Task
Question: What more likely to happen next? A: He puts the hot sauce on the food. B: The chef will serve the food.
Query
Answer: The chef will serve the food.
Figure 5: Prompt for VLEP task.
4.5 Semi-supervised Text-Vides Retrieval
In addition video-to-text generation tasks, we show that broader range of video-language tasks can benefit from our few-shot video captioner from data perspective. Here, we consider low- budget semi- supervisd setting where we only have a few labeled video-caption pairs and large amount of unlabeled videos. The idea is to leverage our video captioner to generate pseudo labels for training any given vision-language models. As a case study, we evaluate on two text-video retrieval benchmarks, i.e., MSR-VTTT and VaTeX. We use greedy decoding to generate pseudo caption for each video in the training set. We then train an identical base model, i.e., ILII, using different pseudo labeled data as well as ground truth annotations. We report Recall @ 1 and 5 for both video-to-text
8

---

further select Table 5: Semi-supervised text-video retrieval with 10 labeled examples. Viaell
V Vnlaaeel the number labeled and unlabeled videos, respectively. I t_R/ and r r_R denote video-to-text Recall and 5. v RI and v_R5 denote text-to-video Recall @ and 5.
MSR-VTT
Retrieval
VaTex Retrieval
Model
Pseudo Label
label// unlabel
t_1
v_R1 v_R5
Vabell unlabel
t_1
t_5
v_R1 Y R5
BLIP
33.2
57.2
40.5
62.8
28.2
53.4
34,0
58,6
BLIP
UniVL BLIP
10/7010
33.1
57.3
33.6
57.7
10622685
25.5
47.7
26.1
49.1
BLIP
10/7010
35.6
60.8
39.8
60.4
10 22685
26.3
50.5
29.3
53.6
BLIP
BLIP cap (odIL((ours)
7010
35.3
58.0
39.1
63.3
10 22685
23.9
46.8
27.5
49.7
BLIP
10 7010
39.6
64.5
40.8
65.2
10 22685
33.3
59.1
33.7
59.5
BLIP
Ground Truth
7010/0
43.6
66.2
43.1
67.2
22685 /0
40.1
66.4
40.1
66.6
ALPRO [25]
Ground Truth
140200/ /
32.0
60.6
33.9
60.7
DRL [56]
Ground Truth
180000 /
54.1
77.4
52.9
78.5
pseudo labels, we achieve significant improvements compared with zero-shot BLIP. We also show that the performance gain not simply a result of training more data, since finetuning other baselines (UniVL, BLIP) is less effective and can even hurt the performance. Furthermore, on MSR-VTT Recall @ 5 we can even achieve comparable performance against BLIP model finetuned on full ground truth annotations.
and text-to-video retrieval. Table 5 shows that through training
the pseudo labels generated by
Another interesting observation is that, compared with the video captioning results in Table 2, found that the gain of our model over baselines on text-video retrieval is more visible than on captioning. A key factor performing well text-video retrieval tasks is to learn good video-text multi-modal alignment. This result shows that our pseudo labels capture richer video-text alignment that benefit the retrieval-style downstream task. The N-gram based generation metries, e.g.. BLEU, may not be able to fully reflect the alignment information, due to the variety semantic levels in video captions. Furthermore, from data perspective, our video captioner can be viewed as a data augmentation tool which is capable of generating or augmenting any open-domain video- language pretraining datasets with minimal human effort. As result, video-language pretraining by constructing cleaner and more diverse video-text corpus.
potentially improve
Table Impact of visual tokens and temporal dimen- sion.
Table 7: Impact of shot selection. #ICE in-
dicates the number of in-context examples in the prompt. Details of in-context example selection are in the Appendix.
Video Representation
Avg
Std
Frame
39.6
3.7
Frame+
40.3
2.9
w/o selection
w/ selectioon
#shot
Visual Token
Framt-eOject-Event
39.9
2.8
#ICE
Avg
Std
#ICE
Avgł Std
Frame+Objectt
2.9
5
5
38.4
2.1
5
40.4
1.2
Frame+OOjectt-Event+ Attribuut
40.8
2.4
10
10
41.3
3.6
5
40.8
2.4
Reduce to one frame
38.5
2.4
20
20
42.6
3.3
5
42.2
2.0
Temporal
Reverse temporal order
40.7
1.7
30
30
40.0
2.9
5
41.1
1.9
4.6 Ablation Studies
We perform comprehensive ablation studies on our few-shot prompt including the impact different video representation, number of shots and in-context selection. All the ablation results evaluated on MSVD_ __Q validation set, and we report the mean and standard deviation of each setting on three sets of randomly sampled shots. For the cases with in-context example selection, examples as in-context examples from the sampled shots, while for the cases without in-context selection, all shots will be feed into the prompt. In Table 6, we show adding visual tokens consistently improves not only the model accuracy but also the model variance. lower standard deviation indicates that the model is less sensitive the few-shot sampling.
To further demonstrate the impact of the additional temporal dimension videos, we perform two setting. First, reduce the number of
ablations on the "Frame+Ob

---

frame captions and visual tokens to be one" for each video. We found that the performance drops significantly compared with using the default four frames, which indicates the model's ability to incorporate information from multiple timestamps. Further, we found that fine-grained temporal modeling is rarely required for performing well on current lideo-language benchmarks. As shown in the ablation result where we reverse the order of all visual tokens and frame captions, the performance decreased only marginally, which indicates that current benchmarks may not be sufficient reflecting the benefits from better temporal ordering.
In Table 7, we first show that, with the same context length, namely, 5 in-context examples, in-context example selection significantly increases the performance as well as the robustness. At 10-shot, and 20-shot, directly fitting more shots into the prompt results better performance. In-context selection achieves slightly lower performance but with significantly better efficiency due to shorter context. Interestingly, at 30-shot, in-context selection with 5 examples outperforms directly adding all 30 shots into the prompt. This is showing that in-context selection can help the model utilize larger number noisy video examples. Nevertheless, we still observe that the benefit of adding more shots saturated at around 20 to 30 shots, even if with in-context selection. we view this remaining challenging on how to make language models benefit from longer contexts.
5 Conclusions, Limitations and Future Work
This paper proposes VidIL. a few-shot Video- Video-language Learner via Image and Language models. It demonstrates the strong ability of large-scale language models performing video-to-text tasks when frame features are provided as unified text representations using image-language models. We propose a temporal order aware prompt by decomposing videos into a hierarchical structure, which is able to plug in multiple levels of frame features, along with speech transcripts. Without pretraining on videos, our model outperforms vision- lision-language models learned from large-scale video datasets on a variety of few-shot tasks, such as domain-specific captioning, question answering, and future event prediction. One limitation of using unified textual representation is that we might lose low-level visual features which can be essential for some specific tasks, such as fine- fine-grained spatial visual question answering. We also observe that current lideo-language benchmarks rarely require explicit temporal tracking on the frames and visual tokens. Future work will focus on leveraging large-scale language models for learning script knowledge from long videos where temporal dynamics are better emphasized.
6
Broader Impact
An open-domain few-shot video-langusee learner has wide range of beneficial applications for society, such as automatically detecting violent or mature content in videos and helping people with vision impairment understand videos. However, since the language model is pretrained on massive internet-scale text data, there might be unexpected output that can have potential negative impact on the society, such bias against people of a certain gender, race sexuality. Future work and dedicated collaboration from the community are needed to alleviate the potential negative societal impact of large language models.
Acknowledgements
We thank the anonymous reviewers helpful suggestions. This research based upon work supported in part by U.S. DARPA AIDA Program No. FA8750-18-2-0011 and U.S. DARPA KAIROS Program Nos. FA8750- 19-2-1004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
9 we the frame caption and visual tokens from the middle frame.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances Neural Information Processing Systems, 34, 2021. 3
[2] Cean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. ArXiv preprint, Abs/2204.144999 2022. 7,8
[3] Bran-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovie, Jason Ramapuram, Jef- frey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. In Hugo Larnchelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. 3
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In 2015 JEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-22422 2233.243..1 IEEE Computer Society, 2015. 7,8
[5] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, 2005. Association for Computational Linguisties. 6
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. 1, 5.5
[7] David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies. pages 190-200, Portland, Oregon, USA, 2011. Association for Computational Linguistics.
[8] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. Earopean conference on computer vision, pages 104-120. Springer, 2020. 2
[9] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying aaiion-and-language tasks via text generation. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1931 1942. PMLR, 2021. 3
[10] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248 255. IEEE Computer Society, 2009. 4
[11] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. ArXiv preprint, abs/2106. 1097, 2021.
[12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019. Seoul, Korea (South), October 27- November 2, 2019, pages 6201-6210. IEEE, 2019. 3
[13] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3834 Online, 2021. Association for Computational Linguistics. 2
[14] Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi. Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS) pages 1 1-6, Melbourne, Australia, 2018. Association for Computational Linguistics. 2
11

---

[15] Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He. Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende. Michel Galley, and Margaret Mitchell. Visual storytelling. In Proceedings of the 2016 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1233-1239, San Diego, Califoria, 2016. Association for Computational Linguistics. 2
[16] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitiin, pages 12976-12985, 2021. 2
[17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904 49 16. PMLR, 2021.
[18] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-ane iidionand--aae waeea--nd--aaggand language transformer without convolution or region supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594. PMIR, 2021.
[19] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73,
2017. 4
[20] Alina Kuznetsova, Hassan Rom, Neil Alidrin, Jasper Uijlings, Ivan Krasin, Jordi Juste, Shahab Ka- mali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7): 1956-1981,, 2020. 4
[21] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg. Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and and-language learningvia sparse sampling. In CVPR, 2021. 1,
[22] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. TVQA: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369 1379, Brussels, Belgium, 2018. Association for Computational Linguistics. 3
[23] Jic Lei, Licheng Yu, Tamara Berg, and Mohit Bansal, What is more likely to happen next? vidco- and languagee future event prediction. In Proceedings the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 8769-8784, Online, 2020. Association for Computational Linguistics. 1, 6, 8
[24] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising srpuence-to-squenne pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, 2020. Association for Computational Linguistics.
[25] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C.H. Hoi. Align and prompt: Video-and- Video-and-language pre-training with entity prompts. In arxiv, 2021. 8,9
[26] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping Eiguage-image pre- ve-training for unified rision-langual understanding and generation. ArXiv preprint, abs/2201. 12086, 2022. 2, 3, 7, 8
[27] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021. 2
[28] Linjie Li, Yen-Chun Chen, Yu Cheng, The Gan, Licheng Yu, and Jingjing Liu. HERO: Hicrarchical encoder Video+Language represeeprsnnntaioon pre-training, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2046-2065, Online, 2020. Association for Computational Linguistics. 3,3
[29] Xiujun Li. Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Ubject-semantte aligned pre-training for lasion-language tasks. In European Conference on Computer Vision, pages 121-135, Springer, 2020. 2
12

---

[30] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 605-612, Barcelona, Spain, 2004.
[31] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 6
[32] Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani, Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7001-7011. 1. IEEE, 2021. 3,
[33] Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13853-13863, 2022. 3
[34] Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, and Shih-Fu Chang. Towards fast adaptation of pretrained contrastive models for multi multi-channel video-language retrieval. arXiv preprint X206 02082, 2022. 1
[35] Jiasen Ln, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining agnostici visiolinguistic representations for risionaand-language tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue Buc, Emily B. Fox, and Roman Gameet, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13 2019. 2
[36] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li. Jason Li, Taroon Bharti, and Ming Zhou. Univi: A unified video and language pre-training model for multimodal understanding and generation. ArXiv preprint, abs/2002.06353, 2020. 1, 3,6
[37] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An empirical study of clip for end to end video clip retrieval. ArXiv preprint, abs/2104. 04.008860 2021.
[38] Antoine Miech, Jean- Baan-Baptiste Alayrae, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9876-98866 IEEE, 2020. 3
[39] Antoine Miech, Dimitri Zhukov, Bantisttetteeeeee Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning tixt-video embedding by watching hundred million narrated video clips. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Scoul, Korea (South), October 27 November 2, 2019, pages 2630-2640. IEEE, 2019. 2,
[40] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. ArXiv preprint, abs//203..0 02100015555 2022. 2,
[41] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics. 6
[42] Mandela Patrick, Po-Yao Huang, Yuki Markus Asano, Florian Metze, Alexander G. Hauptmann, João F.
Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 3
[43] Baolin Peng, Chenguang 7hu, Chunyuan Li, Xiujun Li, Jinchan Li, Michael Zeng, and Jianfeng Gao. Few- shot natural language generation for task-oriented dialog. In Findings of the Association for Computational Linguistics: EMNIP 2020, pages 172-182, Online, 2020. Association for Computational Linguistics. 1
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021. 2, 2,
13

---

[45] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv preprint, abs/1910.10683. 2019.
[46] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified eext-to-text transformer. Journal of Machine Learning Research, 21:1-61 2020. 3
[47] Nils Reimers and Iryna Gurevych. Sentence- BERT: Sentence emheddings using Siamese -ERT-nsttwrks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China, 2019. Association for Computational Linguistics. 4, 6,
[48] Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages Linguistics. 2
Online, 2021. Association for Computational
[49] Paul Hongsuck Sco, Arsha Nagrani, Anurag Amab, and Cordelia Schmid, End-to-end generratvee pretraining for multimodal video captioning. ArXiv preprint, abs/2201 .08264, 2022.
[50] Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang. Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. ArXiv preprint, abs/2205.002655. 2022. 3
[51] Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1980-4991, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. 2
[52] Hao Tan and Mohit Bansal. LXMERT: Learning mross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 5100-5111, Hong Kong, China, 2019. Association for Computational Linguistics. 2
[53] Maria Tsimpoukelli, Jacob I. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212 11:000-11..00 2021.
[54] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-base based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566-4575. IEEE Computer Society, 2015, 6
[55] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li. Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. ArXiv preprint, abs/2202.03052, 2022. 3
[56] Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xian- Sheng Hua. Disentangled representation learning for tixt- retrieval. ArXiv preprint, abs/2203.071111 2022. 9
[57] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang. and William Yang Wang. Vatex: A large- scale, high-quality multilingual dataset for vidco-and- language research. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Scoul, Korea (South), October 27 November 2, 2019, pages 4580 4590. IEEE, 2019. 1, 6
[58] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Can. Simylm: Simple visual language model pretraining with weak supervision. ArXiv preprint, abs/2108. 10904, 2021. 2
[59] Jason Woi, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. ArXiv preprint, abs/2109.0166 2021. 5
[60] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017, pages 1645-1652 2017.
14

---

[61] Hu Xu, Gargi Ghosh, Po-Yau Yao Huang. Dmytro Okhonko, Armen Aghajanyan. Florian Metze, Luke Zettlemoyer. and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7787-6800, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
[62] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging vidco and language. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5288 5296. IEEE Computer Society, 2016. 1, 6
[63] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot wowweddge-basee vqa. ArXiv preprint, abs/2109 00000000514.,1. 2021. 1, 4
[64] Ziyi Yang, Yuwci Fang, Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mci Gao, Yi-Ling Chen, et al. i-code: An integrative and composable multimodal learning framework. ArXiv preprint, abs/2205 01818, 2022.
[65] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emice-vl: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208-3216, 2021.
[66] Yuan, Dongdong Chen, Yi- Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: new foundation model for computer vision. ArXiv preprint, ahs/2111. 1432, 2021. 2
[67] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjac Yu, Yanpeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. ArXiv preprint, abs/2201 1.02699, 2022. 1
[68] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjac Yu, Jac Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34, 2021. 1,2,8 8
[69] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Secratic models: Composing zero-shot multimodal reasoning with language. ArXiv preprint, abs/2204.0059% 2022. 3, 4
[70] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5579-5588, 2021. 2
[71] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205.0100 01068, 2022. 1
[72] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate hefore use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR, 2021. 1,
[73] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision- languge pre-training for image captioning and VQA. The Thirty-Fourth Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Phirty-Secona Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pages 13041-13049. AAAT Press, 2020. 2
(74] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty- Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 7590-7598. AAAI Press, 2018. 3, 6
[75] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. ArXiv preprint,
2021. 3
15


Reviews:
[{"review_id": "jBBE5Og-RLa", "reviewer": "Reviewer_12Dy", "summary": "This paper presents a few-shot learning approach for video and language tasks.\nThe approach relies on pre-trained and frozen models: first, vision models are used to extract for each video frame: (A) a caption and (B) a visual attributes (Objects, Events, Attributes) from a set of pre-defined ones.\nA text-only prompt is then constructed showing few-shot examples of how to perform a vision-language task utilising information the generated captions and attributes.\nFinally, the prompt is fed to a large language model (InstructGPT) to solve the task defined by the few-shot examples.\nThe model is evaluated on several video benchmarks such as: video captioning, video question answering and next event prediction.", "questions": "1. About the use of temporal marker: how does it work when you have more frames than temporal markers?\n2. I was curious to know the impact of the choosen language models. Why did you choose InstructGPT over GPT-3? Is it because InstructGPT is much more lightweight to run? Or is it because InstructGPT is performing better than GPT-3 at following the few-shot examples?\n\ntypo:\nL 276 : ‘video-captain’\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n\n- The approach is relatively easy to follow\n- One strength of the approach is that it does not rely on video data for training. More precisely, it does not require any training at all (in the sense of gradient descent training) as the approach mostly builds on top of pre-existing pre-trained/foundational models.\n- The VLEP (Next event prediction task) results are strong and they provide a new state-of-the-art number on that task.\n- The method is more interpretable than standard VLM models.\n- The approach is highly modular. It is straightforward to update it when improved captioning models or language models are available.\n\n**Weaknesses**:\n\n- I believe the approach to be highly bottlenecked by the expressiveness of the vision foundational models. As a concrete example, I do not see how this approach can work well on traditional action recognition benchmarks (i.e. Kinetics or Something Something) when the task requires low-level temporal understanding of actions. Also the approach does not work for task where there is no useful signals to leverage from frame captions and visual attributes. For instance such task could be fine-grained spatial visual question answering. MSVD-QA and MSRVTT-QA are not really fine-grained VQA dataset, so there are no questions requiring spatial understanding such as \"'What is on the left of the car?' or 'What is on bottom the table?'. Right now I do not see how the provided frame captions and attributes are enough to answer fine-grained question about the visual content.\n\n- Why are the ablation studies only reported on MSVD-QA? The results would have been more convincing with several evaluation datasets.\nFor example in Table 7, we do not see clear improvement when the number of shots increase. As the ablation study is only reported on one dataset, I'd assume the conclusion were not even better on the other datasets? Also I find that the number of in-context examples VS the true few-shot number in this Table 7 is not clear. It would have been clearer to have two columns, one for both. It also look like the in-context selection is not really beneficial for more than 5 shots.\n\n- Table 3: Clarify the real number of shot used in that table. What is the number of in-context vs the number of shots the retrieval system has access to. Is it 5-shot in the sense of 5 in-context shot?\n\n- Table 6: It looks like frame captions are enough to get most of the useful signal, as it only improves the MSVD-QA top-1 accuracy of 1.2% when adding the different visual attributes. This is disappointing knowing all the emphasise around the need for the visual attributes in the approach section.\n\n- In the Related work section, the authors wrote L 110:\n‘We are the first work to leverage prompting a frozen language model for tackling few-shot video-language tasks with a unified textual representation’.  I believe this is an overclaim as [1] have already done something in a similar spirit. It looks to me this paper, mostly extend the work of [1] with non-vqa tasks and video instead of images.\n\n[1] Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022, June). An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 3, pp. 3081-3089).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_EZo63P-vL9", "reviewer": "Reviewer_Y7bx", "summary": "This paper introduces a method for few-shot learning for video understanding tasks. In a nutshell, it works by first extracting information from a given video using pre-trained image and language models: it uses CLIP to extract \"visual tokens\" for objects, events and attributes, as well as a captioning model (BLIP) to extract image description of a few frames of the video. These automatically generated annotations are then assembled together to form a textual description representing the video by notably keeping some information about the order of events. Since videos are represented by simple text, it allows to use the few-shot capability from an existing language model (InstructGPT in this case) to do few-shot learning on video understanding tasks: video captioning and video question answering. The authors evaluate this approach on five different benchmarks. The model is also used as a pseudo labeler to improve video-text retrieval system.\n", "questions": "**Clarifications**\n\n- L144: \"We then select top 5 visual tokens\": is it top 5 per frame, or top 5 total? In the Figure 2, it seems there are only 3 for the first frame. Can the authors clarify this? It is also said that 8 frames are used but when I check the Appendix, I can see that there are at most 4 temporal description (First, Then, After that, Finally). Is there some filtering happening on top of the top 5?\n\n- L224: For the BLIP baseline, is there additional training involved or is the proposed technique of \"stitching the sampled frames\" is a pure inference only trick? This was not clear to me as the term \"loss\" was mentioned. Can the authors clarify what is meant here?\n\n- Table 2: As far as I know UniVL and BLIP are not \"few-shot\" methods. How do the authors obtain 10-shot results for these baselines?\n\n- Table 2: How come VidIL can be less good than BLIP (for MSRVTT) knowing that the BLIP captions are used as input in the prompt? Is that because the language model is not able to properly use the information provided by the BLIP model in that case?\n\n- When using in-context selection, are you also ordering the samples in a certain way to account for the recency bias in big language models? (putting the most similar last in the prompt might help)\n\n**Additional ablations**\n\n- How important is the \"temporal aware prompt\" technique quantitatively? Unless I missed that ablation I could only see the qualitative example of Figure 3. Since this trick seems one of the contribution of the paper, I believe it would be important to properly demonstrate that it brings something.\n\n- How does the performance change with the number of sampled frames?\n\n- How does the performance evolve with the number of shots? What is the maximum number of shots you can fit in the usual prompt of the model?\n\n**Questions about experiments**\n\n- In Table 6, adding event seems to hurt performance (Line 3). Can you provide the numbers for \"Frame+Object+Attribute\"?\n- Table 7: is that possible to put a column \"Prompt size\" and a column \"Number of shots\" to better distinguish between these two parameters. I think it will make it clearer. How come in-context selection can actually hurt?\n", "limitations": "Some important limitations are not mentioned:\n\n- Performance potentially highly bottlenecked by the image level \"text\" extraction\n- Replacing videos by text in the first place is limited and may not scale well when moving to longer videos (the approach would always be limited by the size of the prompt of the language model). Having Large Language models that could directly attend high level features instead could help alleviate this issue. \n\nThe authors do discuss the societal risk, notably the ones inherited from large language models that might be amplified here by using them on videos.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly written and easy to follow\n- Few-shot learning in multimodal setting is a very important endeavor \n- The proposed method is a good example of repurposing a large language model to do few-shot learning for videos in a simple way that does not involve any training with video and language dataset. It provides a very strong baseline for such models.\n\n\n**Weaknesses**\n\n- A few clarifications are required around the generation of the prompt (see Question section)\n- Conceptually, the approach is quite similar to the PICA paper [61]. It would be good to better clarify the difference with that approach? Is it mainly the video setting and the fact that one has to deal with temporal information? If yes, then would it be possible to show what happens if only a single frame from each video is used instead of multiple ones?\n- Some ablations are missing (see Question section for specific suggestions)\n- Some results seems a little bit inconsistent and raise some questions (see Question section for more details)\n- Should Flamingo be also compared against in Table 2 for consistency (since they provide results for YC2 and VATEX as well?)\n - It would be interesting to see how the performance evolve with number of prompts for all benchmarks but mainly 5 shot results are given (see request in the Question section)\n- Limitations of the approach: \n  - This approach is brittle as it is bottlenecked by the extraction of visual entities and the captioning model. \n  - Such approach may not scale well to long term video understanding (mentioned in the conclusion) since one would be quickly limited by the size of the prompt of Language Models. Also temporal information is only given with words like \"First\", \"After\"... How would that be scale to longer videos? Could the authors comment on that?\n\n\n**Summary of review**\n\nOverall the paper introduces an interesting method and has good results on video benchmarks. However there are a few important questions that are important to address in the rebuttal to help confirm my current weak accept decision.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GqwOPo6NG-", "reviewer": "Reviewer_6Be3", "summary": "This paper proposed a new model for video-language tasks with few-shot ability. The task is decomposed into hierarchical level, to first translate objects, frame captions and their relations using CLIP model. Then a langauge model is instructed with prompts to compose the video frame. ", "questions": "1- Line 28: Only the encoder model with bidirectional attention are limited to understanding tasks, but with causal attention, they are capable of both understanding and generation. In fact, autoregressive models have shown promising few-shot performances, because they are more parameter efficient in compare to encoder-decoder models \n\n2- Is the few-shot ability arises from using language model at the end, or from decomposing the tasks to two models?\n\n3- Line 116: it is mentioned that the proposed model is not required to be pretrained on any video, compare to Flamingo. However, it is mentioned otherwise in other parts of the paper!\n\n4- is there an ablation on the decomposition of video contents into frame, visual token level, video level on the few-shot performance?\n\n5- Figure 2: in visual token level, each frame includes multiple objects. How the template format organizes objects of one frame in order. how the order of objects is defined for each frame? (refer to “few-shot context” box). Is there an ablation on the ordering of visual tokens?\n\n6- is there an ablation on different task instruction formats? how does wording of the template affects performance?\n\n7- Table 7: increasing shots also increase standard deviation. why model performance variance increase by more training examples?", "limitations": "The proposed model is not end-to-end. It is based on using CLIP and InstructGPT3. Therefore, there is minimal novelty in the architecture.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed model is not end-to-end. It is based on using CLIP and InstructGPT3. The approach is simply based on extracting candidate objects and captions frame by frame using CLIP encoder, and then creating a prompt to feed to InstructGPT3 using in-context learning to achieve few-shot performance. Therefore, there is minimal novelty in the architecture", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5qn3uoOxeEW", "reviewer": "Reviewer_An3y", "summary": "This paper proposes a few-shot Video-language Learner termed VidIL based on image- language modeling. A temporal order aware prompt is also proposed by decomposing videos into a hierarchical structure. Without pre-training on videos, the proposed VidIL is evaluated on a variety of few-shot video-to-text tasks, such as domain-specific captioning, question answering, and future event prediction. ", "questions": "See my questions in weaknesses.", "limitations": "Yes. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ A few-shot Video-language Learner termed VidIL is proposed based on image- language modeling.\n+ A temporal order aware prompt is proposed by decomposing videos into a hierarchical structure.\n+ A variety of few-shot video-to-text tasks are included for performance evaluation. \n\nWeaknesses:\n- The technical novelty of this paper is very limited. CLIP, BLIP, and InstructGPT are respectively deployed for visual tokenization, frame captioning, and conditional text generation with few-shot prompt. Excluding this, the contributions made by the authors are rather minor. I even have doubt that the performance of VidIL is mainly due to the deployment of CLIP, BLIP, and InstructGPT. \n- The proposed VidIL can still be considered as prompt learning for large-scale cross-modal pre-training models. Therefore, other prompt learning methods should be included in the main experiments. Some examples are:\\\n[a] Align and Prompt: Video-and-Language Pre-training with Entity Prompts, arXiv:2112.09583, 2021.\\\n[b] CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models, arXiv:2109.11797, 2021. \n- The method part should be reorganized in a more formal way. Although the source code is given, an outlined algorithm would help the readers easier to capture the proposed method.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept