Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Authors: Antoine Yang
Antoine Miech
Josef Sivic
Ivan Laptev
Cordelia Schmid
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video Question Answering
Zero-Shot
Vision and Language
Computer Vision

Abstract: Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA. It also demonstrates competitive performance in the few-shot and fully-supervised setting. Our code and models are publicly available at https://github.com/antoyang/FrozenBiLM.

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Antoine Yang 1.2 Antoine Miech Josef Sivic Ivan Laptev 1.2, Cordelia Schmid
Inria Paris
Département d' d'informatique de I'ENS, CNRS, PSL Research University
'DeepMind 'CIRCC CTU Prague
https://antoyang. gittubb.ioffrozeneee..... html
Abstract
Video question answering (VideoQA) is a complex task that requires diverse multi- modal data for training. Manual annotation of questions and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answe answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data multi-modie inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-mode data, and finally (iii) perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MS. ActivityNet-QA. and TVQA. It also demonstrates competitive performance in the few-shot and fully-supervised setting. Our code and models publicly available at [1].
TGIF-Frameds
How2QA
Cross-modal Training
Zero-Shot VideoQA
Training
data:
FrozenBiLM Pretrained BILM
Test data:
FrozenBiLM
Pretrained BILM
Web-
*
scraped
Video
Caption
Video
Little cute toy poodle dog running fast on the beach.
Question
What the dog doing?
Answer: Running
Our model FrozenBiLM builds pretrained and frozen bidirectional language model (BiLM), and is trained from Web-scraped video-caption pairs. FrozenBiLM excels in the zero-shot video question answering task without using any explicit visual question-answer supervision.
Figure I:
Introduction
Video question answering (VideoQA) is a challenging task that requires fine-grained multi-modal understanding. State-of-the-art approaches VideoQA [43, 107, 109] rely on large video datasets manually annotated with question question-answer pairs. Yet, collecting such annotations is time consuming, expensive and therefore not scalable. This has motivated the development of zero-shot VidcoQA approaches [101, 102, 110], that use no visual question-answer annotation for training, see Figure 1.
*Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Recently, a promising line of work builds on frozen large autoregressive language models [19, 68, 91, 96, 104, 111] for zero-shot visual question answering. This has been motivated by the findings from GPT-3 [8] which exhibits strong zero-shot text-only question answering abilities from large autoregressive language models. Such models [8, 72, 82, 92] can predict an arbitrarily long sequence of text, one token at each step from left to right. However, they usually require billion parameters to work well, making them computationally expensive to train, and challenging to deploy in practice,
In contrast, recent work in natural language [65, 76, 77, 87] demonstrates strong zero-shot perfor- mance for lighter bidirectional language models (BiLM). Such models [17, 25, 35, 42, 61, 75] can predict a few masked tokens in an input sequence given left and right context in a single forward pass. These works cast downstream tasks in cloze form [90], similar to the masked language modeling task (MLM) [17] solved by these models at pretraining. This motivates us to tackle diverse zero-shot multi- modal tasks (open-ended VideoQA [98], multiple-choice VideoQA [46] and fill-in-the-blee [66]) by formulating them in cloze form and leveraging the text-only knowledge of pretrained BiLM.
To adapt pretrained BiLM multi-modal inputs, we combine it with frozen pretrained visual backbone and a set of lightweight additional modules including adapters [28]. We train these modules on Web-scraped video-text data using a simple visually-conditioned MLM loss. We preserve the uni-modal knowledge of a BiLM by freezing its weights. To our knowledge, our approach is the first to explore the zero-shot visual- lingusise capabilities of frozen non- aotreeroreerrsivvee language models.
We show that our approach largely improves the state of the art on various zero-shot VideoQA benchmarks. Furthermore, we demonstrate that frozen bidirectional language models perform better while being cheaper to train than frozen autoregressive language models [91]. Moreover, our ablation studies show (i) the ability of our model effectively perform zero-shot multi-modal reasoning using both visual cues and speech transcripts. (ii) the importance of adapters combined with frozen pretrained language models, (iii) the impact of multi-modal data scale, (iv) the impact of the language model size and of bidirectional modeling. Our approach also performs competitively in the fully- supervised setting. Indeed, we show the benefits of freezing the weights of BiLM when using VideoQA training data, while updating considerably less parameters compared to alternative methods. Finally, we introduce a new few-shot VideoQA task in which we finetune our pretrained model on a small fraction of the downstream training dataset, and show promising results in this setting.
In summary, our contributions are three-fold:
(i) We present FrozenBiLM, a framework that handles multi-modal inputs using frozen bidirec- tional language models and enables zero-shot VideoQA through masked language modeling. (ii) We provide extensive ablation study and demonstrate the superior performance of our framework in the zero-shot setting when compared to previous autoregressive models.
(iii) Our approach improves the state of the art in zero-shot VideoQA by a significant margin. FrozenBiLM also demonstrates competitive performance in the fully-supervised setting and shows strong results in the few-shot VideoQA setting which we introduce. Our code and trained models are publicly available at [1].
2 Related Work
Zero-shot VideoQA. A vast majority of VideoQA approaches rely on relatively small, manually
annotated VideoQA datasets [3, 9, 10, 13-15, 20, 23, 24, 30, 33, 34, 36-39, 43, 44, 47, 58, 60, 69, 70, 74, 78, 79, 83, 89, 97, 100, 103, 105, 112, 116]. Recently, few work [101, 110] have explored zero-shot approaches for VideoQA, where models are only trained on automatically mined video clips with short text descriptions. In contrast to VideoQA annotations, such video-text pairs are readily- available at scale on the Web [6, 67, 109]. In particular, Yang et al. [101] automatically generate VideoQA training data using language models [72] pretrained on a manually annotated text-only question-answer corpus [73]. Reserve [110] uses GPT-3 [8] to rephrase questions into sentences completed by a multi-modal model. In contrast to these prior works [101, 1 100, our method does not require any kind of explicitly annotated language dataset or the use of data generation pipelines for zero-shot VideoQA. Note that BLIP [53] studies a related setting where a model trained on manually annotated pange--uestion-answsee Also note that VideoCLIP [99] considers a related zero-shot multiple-choice video-to-tex retrieval task as VideoQA, but in this setting the model is not provided with natural language questions.
triplets is transferred to VideoQA, which is less challenging task.
""Cloze test" is an exercise test where certain portions of text are occluded masked and need to be filled-in.

---

Visual language models. As language models require large amounts of training data to perform well [27]. recent works have studied transferring pretrained language models [8, 94] to image-text tasks. VisualGPT [11] and VC-GPT [64] showed the benefit of initializing the weights of an image captioning model with pretrained autoregressive language-only model. Recent work pushed this idea further by freezing the weights of pretrained autoregressive language model for tackling vision and language tasks [2, 19, 68. 91, 96, 104, 111]. Our approach also leverages a frozen pretrained language model. Similar to MAGMA [19], we also use adapter layers [28, 29]. However, we differ from these approaches as we propose to instead use lighter bidirectional masked language models, instead of autoregressive ones, and rely on masked language modeling objective (MLM) instead of an autoregressive one. Moreover, our model is specifically designed for videos, for which high-quality visual question answering annotation is even more scarce compared to still images [19, 68, 91, 104]. We also explore the use of the speech modality, and tackle tasks which challenging for autoregressive language models such video-conditioned fill-in-the-bland [66]. Finally we show in Section 4.3 the superior performance of frozen bidirectional language models in comparison with autoregressive ones [91].
Masked Language Modeling in vision and language. The MLM objective was initially introduced in natural language [17, 42, 61] to pretrain bidirectional transformers and learn generic representations. This approach achieved state-of of-the-art results in many language tasks after finetuning on downstream datasets. Its success inspired numerous works to adapt it to train multi mottll transformer models on paired visual- -sngul-iin data [12. 21, 22, 26, 31, 40, 48, 51, 56, 54, 59, 52, 50, 62, 63, 80, 81, 85, 86, 88, 93, 95, 106, 109, 114, 115]. However, these works typically use it to learn generic visual-linguistic representations by updating the transformer weights, and then use expensive manual supervision to train randomly initialized task-specific answer classifiers for VQA [12, 22, 51, 52, 59, 62, 80, 81, 85, 88, 95, 106] or VideoQA [21, 48, 50, 93, 109]. In contrast, we tackle zero-shot VideoQA, i.e. without using any manual annotation. Moreover, we do not update the transformer weights during cross-modal training, but instead exhibit the benefits of freezing these weights after text-only pretraining, for both zero-shot and fulev-supervised VideoQA (see Sections 4.2 and 4.5).
3 Method
This section presents our approach tackle zero-shot video question answering. Here, zero-shot means that we do not use any visual question answering annotation and only rely on scalable data from the Web. Our approach starts with two strong pretrained components: (i) text-only bidirectional masked language model (BiLM) pretrained on data from the Internet, which has the capability of zero- shot question answering but is not capable of visual reasoning, and (ii) a vision encoder pretrained to map images to text descriptions, but which does not have the ability to perform visual question answering. We aim at connecting these two components while keeping the language component frozen to avoid catastrophic forgetting [16]. where the large language model would specialize to a new task while forgetting its initial capabilities. The end-goal to design a unified model having the best of both worlds: visual understanding capabilities of a powerful visual encoder and question answering capabilities of a powerful language model. This requires several technical innovations, which are described in the rest of this section. First, we explain in Section 3.1 how we augment frozen pretrained bidirectional masked language model with new layers to enable joint video and language reasoning, see Figure 2. Second, we present in Section 3.2 how we train these layers on video-text data scraped from the Web [6]. Finally, we describe in Section 3.3 how we enable zero-shot predictions for several video-language downstream tasks, including open-ended VideoQA, by casting them in a cloze form, similar to the masked language modeling task solved during training.
3.1 Architecture
The proposed architecture, illustrated in Figure 2, brings together a powerful frozen pretrained bidirectional language model with a strong visual encoder. The difficulty lies in enabling multi-modal reasoning while keeping the large language model frozee To address this challenge, we unify these two models via a visual-to-text projection module together with small adapter modules inserted within the frozen language model. Next, we describe in more detail the three main components of the architecture: (i) the frozen pretrained bidirectional language model, (ii) the pretrained video encoder and (ii) the lightweight modules that scamlessly connect the two components.
3

---

Frozen
cuT
Cake
wedding
Trained
MUMTTE
MUMIT
MLMMIn
Adapter A
Transformer Encoder (Language Model) with Adapters f
Feed-Forward
Visual Text Projection
Visual Text Projection
Myt
4
6
Add & Norm
XN
Visual Backbone high
Visual Backbone lag
Token Embeddder 90
*
Adapter A
Self-Attention
The bride and the groom [MASK] the [MASK] at the [MASK]
Caption x
Frame Y/2
Frome yr
Figure 2: Our training architecture consists of large frozen bidirectional language model (BiLM) and frozen pretrained visual encoder (in blue). complemented with additional lightweight trainable modules (in orange): (1) visual-to-text projection module P (on the left), which maps the frozen visual features the joint visual-text embedding space and (2) set of small adapter modules A (on the right) between the frozen transformer blocks. The pretrained normalization layers in the BiL.M (on the right) are also finetuned.
Frozen Bidirectional Masked Language Model. Our method starts from a pretrained bidirectional language model based on Transformer encoder [92]. The input text is decomposed into sequence of tokens æ = {x,}{ E [1, v12 by a tokenizer vocabulary size V. The language model, pa- rameterized by 0, makes use of embedding function Go which independently transforms each token into D-dimensional continuous embedding t = =f = {x)();)}{ E RLXD a Trans- former encoder fo which computes interactions between all input tokens and outputs contextualized representations = H' and a masked language modeling (MLM) classifier head ma which independently maps the D D-dimensional continuous embedding for each token E to vector of logits parameterizing a categorical distribution over the vocabulary V. This distribution is referred to by log (o) (mo (f)) RLV. We assume that the language model is pretrained, i.e. O has been optimised with a standard MLM objective [17] on large dataset of text from the Web. We show in Section 4.2 that this text-only pretraining has crucial importance for zero-shot VideoQA.
Pretrained Video Encoder. The video represented by a sequence frames y = {v:) Each frame is forwarded separately through a visual backbone hg. which outputs one feature vector per frame {u,) = (w:)) E TT..D.. In detail, the visual backbone is CL.IP VIT-I/14 [18, 71] at resolution 224 x 224 pixels, pretrained to map images to text descriptions with a contrastive loss on 400M Web-scraped image-text pairs. The backbone is kept frozen throughout experiments. Note that a CLIP-baselin for zero-shot VideoQA results in poor performance, see Section 4.4.
Connecting the Frozen Language and Frozen Vision components. The video features are in- corporated into the language model prompt [49, 57, 113]
length T (Figure 2, left). This prompt is obtained by linearly mapping the visual features u to the text token embedding space via a visual-to-text projection
IR RRuu i.e. v {w: = (P(u.))f The prompt then concate- nated with the text embeddings before being forwarded to the transformer encoder that models joint disual-lingisttc interactions. We show in Section 4.2 that incorporating the input video considerably improves zero-shot VideoQA results. In addition, to learn powerful multi-modal interactions while keeping the transformer encoder weights frozen, we equip the transformer encoder with lightweight adapter modules A [28] (Figure right). We use adapter which transforms the hidden state with a multi-layer perceptron transformation and residual connection, i.e. A(z) = + W uP vi W/doonn) with Wyoown JR Dx Ds., Wup E D. D the hidden dimension of the transformer, Dd, the bottle- neck dimension, and v a ReI.U activation function. Dr, is typically set to be smaller than D such that the adapters are lightweight. detail, we add an adapter module before the layer normalization. after each self-attention layer and each forwwrd layer of the transformer encoder.

---

3.2 Cross-modal training
We wish to train the newly added modules introduced in the previous section (shown in orange in Figure 2) for the VideoQA task. This is hard because we assume that no explicit manual annotation for the VideoQA task is available, such annotations being expensive and therefore hard to obtain at scale. Instead we train our architecture using only readily-available video-caption pairs scraped from the Web. Such data is easy to obtain [6, 67, 109], ensuring the scalability of our approach.
During training, we keep the weights of the pretrained BiLM and pretrained visual backbone frozen as previously explained. We train from scratch the parameters of (i) the visual-to-text projection module P and (ii) the adapter modules A. We show in Section 4.2 the importance of freezing the BiLM weights combined with training the adapter modules. Note that all normalization layers [5] of the pretrained BiLM are also updated to adjust to the new distribution of the training data. We denote all the trainable parameters of our model by the subscript j.. In practice, they sum up to about 5% of the BiLM parameters, hence the training of our model is computationally efficient.
We use conditodee masked language modeling objective (MLM), in which some text tokens } } are randomly masked and the model has to predict these tokens based on the surrounding text tokens and the video input. Formally, we minimize the following loss:
1 M Σ
(x(
log Pp PT (T, y) TT
(1)
where à is the corrupted text sequence, y is the sequence of video frames, ( hm is the probability for the (masked) m-th token in T to be Im. and M is the number of masks in the sequence ă. In detail, we follow [17] and corrupt 15% of text tokens, replacing them 80% of the time with a mask token, 10% of the time with the same token and 10% of the time with randomly sampled token.
3.3 Adapting to downstream tasks
After training, our model is able to fill gaps in the input text given an input video together with left and right textual context as part of the input text. We wish to apply our model out-of-the-box to predict an answer given question about a video. The video can optionally come with textual subtitles obtained using automatic speech recognition. To avoid using manual supervision, we formulate the downstream tasks cloze form [76, 90], i.e. such that the model only has to fill-in a mask token in the input prompt similarly to the MLM objective optimized during training. The adaptation to the downstream tasks brings several challenges, as described next. First, we describe how we formulate the input text prompts for several downstream tasks. Then, we explain how we map the mask token from the input text prompt to an answer via a frozen answer embedding module. Finally, we present how finetune architecture in supervised setting.
Input prompt engineering. We describe how we design the input text prompts for several down- stream video-language tasks. Each downstream task is formulated as a masked language modeling problem. This allows us to apply FrozenBiLM out-of-the-box. A [CLS] token and [SEP] token are respectively inserted at the start and the end of each sequence following [17].
Open-ended VideoQA. Given a question and a video, the task is to find the correct answer in large vocabulary A of about 1K answers. Answers are concise, i.e. the great majority of answers consist of one word [32, 98, 101, 108]. We design the following prompt: "[CLS] Question: <Question>?
Answer: [MASK] Subtitles:
<Subtitles> [SEP]
Multiple-choice VideoQA. Given a question and a video, the task is to find the correct answer in small number of candidates C. typically up to 5 choices [46, 54]. We set the vocabulary to A = [Yes,No and compute a confidence score for each candidate by using the following prompt:
[CLSS] Question: <Question>? Is it <Annser Candidate> [MASK] Subtitles:
<Subtitles> [SEP]"
We choose the best option by selecting the candidate with the highest Yes logit value.
Video-conditioned fill-in-the-blank task. Given video and a sentence with blank space, the task is to fill in the blank with the correct word from a vocabulary A of about 1K answers. We replace the blank in the sentence with mask token, and design the following prompt:
[[CLS] <Sentence with [MASK] token>. Subtitles: <Subtitles> [SEP]"
5

---

Note that all prompts are prepended with the video prompt (see Section 3.1) before being forwarded the transformer encoder.
Answer embedding module. For each downstream task, we wish to map the mask token in the input text prompt an actual answer prediction in the set of possible answers A. described above. For this we use the frozen MLM classifier head mg. However, mg E RVDD covers V different tokens where V >> N and N s 1, 000 is the size of A. Therefore, answer classification head which linearly maps a contextualized mask representation 1 to a vector of logits parameterizing a categorical distribution over the vocabulary A, i.e. I E RNXD, We set the weights of this answer module with the corresponding weights of the pretrained MLM classifier me for one-token answers. In the case of mulli-token answers, we average the weights of their different tokens. We, hence, enable cero-shot inference at test time. We also discuss other alternative strategies to handle multi-token answers in the Supplementary Material.
introduce task-specific
Fully-supervised training. To evaluate our approach on sully-supervised benchmarks, we also explore finetuning our model on datasets that provide manual annotations for the target task. To this end, we train the same parameters explained in Section 3.2, while keeping the transformer weights and the answer embedding module frozen. For open-ended VideoQA and video-conditioned fill-in-the- blank, we use cross-entropy loss on the task-specifie vocabulary A. For multiple-choice VideoQA, we use a binary cross-entropy loss applied each answer candidate. We show Section 4.5 the benefit of freezing the language model weights during uully-supervised training.
Experiments
This section demonstrates the benefits of our FrozenBiLM framework and compares our method to the state of the art. We first outline our experimental setup in Section 4.1. We then present ablation studies in Section 4.2. Next we compare our bidirectional framework to its autoregressive variant in Section 4.3. The comparison to the state of the in zero-shot VideoQA and qualitative results are presented in Section 4.4, Finally, we finetune our model on the VideoQA task in Section 4.5, where we show few-shot and superviede results.
4.1 Experimental setup
Frozen bidirectional language model. We use tokenizer based on SentencePiece [41] with V = 128, 000, and bidirectional language model with 900M parameters, DeBERTa-VV-XII.ae trained with the MI.M objective on a corpus of 160G text data. We also show how our approach generalizes other MLM-pretrained bidirectional language models such as BERT [17] in Section 4.2.
[25],
Datasets. For training we use the publicly available VedViM dataset [6], which consists of 10 million of video-text pairs scraped from the Shutterstock website where video captions are obtained from avaiiall alt-text descriptions. We evaluate results on eight downstream datasets covering wide range of textual and video domains (e.g. GIFs, YouTube videos, TV shows, movies), and multiple VideoQA paradigms: open-ended VideoQA (iVQA [101], MSRVTT. QA [98], MSVD-QA [98], Activity Net-QA [108) and TGIF-QA FrameQA [32]), multiple-choice VideoQA (How2QA [54] and TVQA [46]) and video-conditioned fill-in-the-blank (LSMDD-Fill- in-the-blank [66]). Unless stated otherwise,
report top-1 1 test accuracy using the original splits for training, validation and test. For How2QA, we report results on the public validation set for comparison with prior work [78, 101, 107]. For TVQA, we report results on the validation for the ablation studies and on the hidden test set for the comparison to the state of the art. More details are included in the Supplementary Material.
Implementation Details. The training for 2 epochs on WebVid10N, lasts 20 hours on 8 Tesla V100 GPUs. We give further details in the Supplementary Material.
4.2 Ablation studies
In this section, we evaluate the zero-shot performance of different variants of our method. By default, the frozen pretrained DeBERTa-V22 -2-XLaa- language model and train the visual-to-text-

---

1. Pretraining
Frogen
Adapters
Fill-in-the-hlan hlank
Open-ended ended
Multiple-choine
LSMDC
iVQA
MSRVIT-QA
MSVD-QA ActivityNet-QA TGIF-QA
How2QA
TVQA
K
0.5
0.3
0.1
0.0
0.5
0.0
32.4
20.7
X
37.1
21.0
17.6
31.9
20.7
30.7
45.7
45.6
50.7
27.3
16.8
32.2
24.7
41.0
53.5
53,4
/
$1.5
26.8
16.7
33.8
25.9
41.9
58.4
59.2
Table 1: The effect of initializing and training various parts of our model evaluated on zero-shot VideoQA. All models are trained on WebVidIOM and use multi-modal inputs (video, speech and question) at inference.
Fill-in the blank LSMDC
Open ended MSVD-QA
Multiple-chene choice
Visual
Speech
iVQA
MSRVIT-QA
ActivityNe-QAA
TGIF-QA
How2QA
TVQA
I.
47.9
11.0
6.4
11.3
22.6
32.3
29.6
23.2
2.
x
/
49.8
13.2
6.5
11.7
23.1
32.3
45.9
44.1
3.
50.9
26.2
16.9
33.7
25.9
41.9
41.9
29.7
4.
51.5
26.8
16.7
33.8
25,9
41.9
58.4
59.2
Table 2: Impact of the visual and speech modalities on zero-shot VidcoQA. Rows 1 and 2 report results for
pretrained language model without any visual input. Rows 3 and give results for FrozenBiL.M model pretrained on WehVidIOM.
projection layer together with adapters for epochs WebVid10M. We refer to this default model as FrozenBiLM. This model uses three input modalities in terms of video, question, and speech.
Ablation of the model training. We ablate the effect of initializing parameters of the language model, freezing its weights and training adapters in Table 1. We observe that the language model pretraining is crucial. Indeed, model with randomly initialized language weights (row 1) performs poorly compared to models initialized with language pretrained weights (rows to 4). Moreover, the model which updates the language model weights (row 2) during cross-modal training performs considerably worse compared to variants that freeze them (rows and 4). This shows the benefit of freezing the language model for zero-shot VideoQA. We also notice the benefit of the adapter layers by comparing rows and 4, especially for multiple-choice datasets. Finally, we note that training variants with the frozen language model is twice faster compared to updating all parameters, as there is a significantly lower number of parameters to be trained.
Impact of modalities. Table shows the impact of the visual and speech modalities the zero-shot performance of our model. First, we evaluate the text-only performance our model using neither visual input speech input in row 1. We observe that adding speech (row 2) marginally improves the results and that the importance of speech highly depends on the dataset. When adding vision (rows 3 and 4), the performance increases significantly, e.g. +13.6% accuracy iVQA and +22. % on MSVD-QA between rows 4 and 2. Finally, the model with vision also benefits from the speech, e.g. +16.5% accuracy How2QA and +29.5% accuracy on TVQA (compare rows and 4).
Note that in practice. speech is missing for many videos, as we obtain the speech directly from the YouTube API and many videos no longer available. Exceptions How2QA and TVQA for which the authors [46, 55] provide speech for all videos. Consequently, we have speech data for only 44.3%, 14.2%, 8.2%, 7.1% and 25.3% of test samples in LSMDC-FiB iVQA, MSRVTT-QA. MSVD-QA and ActivityNet-QU respectively. GIFs in TGIF-QA do not contain speech.
Size of the cross-modal training dataset. Zero-shot results of FrozenBiLM after training for fixed number of iterations on different fractions of WebVidIOM are shown in Table 3. We construct these subsets such that larger subsets include the smaller ones. We find that performance increases mono- tonically with more multi-modal training data.
Training Data
MSVD-QA
How2QA
WebViaTK
13.6
33.0
WebVid1OK
22.7
54.9
3.
WehVid200K
27.8
56.0
WebVid2M
30.1
57.4
5.
WebVidlOM
33.8
58.4
Size of the language model. In Table 4, we ablate the im- portance of the language model size for the zero-shot per- formance. Note that when comparing different language models, we use no adapters to avoid biases related to the choice of the bottleneck dimension hyperparameter [28]. We find that using the 900M- parameeer DeBERTA- V2-XI (row 6) outperforms the 300M a0a-aaaaeeeer BERT-Large (row 5) which also improves over the 100M-parameter BERT-Bast -ase (row 4).
Table 3: Dependency on
of the
training set. Zero-shot results are pre- sented for different fractions of the We- bVid10M dataset used for training.
7

---

Train time (GPUH)
Method
I Language Model
L.M params
VQQ MSRVIT-QA MSVD>-A ActivityNet-O TGIF-QA
1. GPT-Nco--L. L338
133
200
6.6
4.2
10.1
17.8
14.4
Autorogressive 2. GPT-Neo-2.78
2.7B
360
9.1
7.7
17.8
17.4
20.1
3. GPT-J-6B
820
21.4
9.6
26.7
24.5
37.3
4. BBRT-BBuse
110M
24
12.4
6.4
11.7
16.7
23.1
Bidirectional 5. BERT-Large
340M
60
12.9
7.1
13.0
19.0
21.5
6. DcBERTe-V2-XLaage
890M
160
27.3
16.8
32.2
24.7
41.0
Table 4: Comparison of autoregressive language models (top) and bidirectional language models (bottom) for
zero-shot VideoQA. All variants trained on Ved for the same number of epochs.
Method
Training Data
Speech
Fill in the blank LSMDC
IVQA MSRYTT-QA MSVD-QA Net-vityNetQQA TGIF-QA
Open enieed
Multiple choiie How2QA TVQA
Random
0.1
0.1
0.1
0.1
0.1
0.1
25
20
CLIP VIT-L/IA [7]] 400M image-texts
9.2
2.1
1.2
3.6
47.7
26.1
Just Ask [102] HowToVQA69N
WebVidYQAAM
X
13.3
5.6
13.5
12.3
53.1
Reserve [110]
VT. Tempprora--11 X
31.0
5.8
FroueBBLLM
WcbVidTOM
26.2
33.7
25.9
41.9
41.9
29.7
FrognBUI (Ours)
WehVidIOM
$1.3
26.8
16.7
38.8
25.9
41.9
58.4
59.7
Table 5: Comparison with the state of the art for zero-shot Vi
Importance of the suffix. Our text input prompts include suffix just to the right of the mask token which consists in a point and an end-of-sentence token for the variant without speech (or a point followed by the speech subtitles for the variant with speech). We found that removing this suffix leads to considerable drop of performance (e.g. the test accuracy on MSVD-QA in the row 3 of Table drops from 33.7% 2.8%). Note that we do not observe such large drop performance when removing the [CLS] token e.g. the accuracy on MSVD-QA drops only from 33.8% to 33.2%. This shows that the bidirectional nature of our framework is key factor for the performance. Intuitively, this suffix forces the model to provide a concise answer. Such a hard constraint cannot be given to unidirectional autoregressive models compared next Section 4.3. We further ablate the importance of the prompt design in the Supplementary Material,
4.3 Comparison with frozen autoregressive models
In this section, we compare our bidirectional framework using language models of various sizes to the larger, autoregressive GPT-based counterparts recently used for zero-shot image question answering [91, 104]. For fair comparison, we adapt autoregressive models video and language inputs similarly as our bidirectional models. In detail, autoregressive variants train a similar visual-to- text projection by using left-to-right language modeling loss [91]. All models in our comparison are trained on Web VidIOM for the same number of epochs. At inference, autoregressive variants use the same template as [91] to which we prepend speech subtitles, greedily decode sequences as [91], and use the same answer vocabulary as bidirectional models. Autoregressive variants select the top answer that maximizes the log-likelihove
when appended to the question prompt. Here also, we use no adapters for all models, such that the architecture of autoregressive models closely follows [91]. This is to avoid biases related to the tuning of the bottleneck reduction hyperparameter in the adapters [28].
We compare autoregressive and bidirectional language models in terms of accuracy and efficiency in Table We observe that our bidirectional framework (rows 4-6) achieves significantly better zero-shot performance-efficee
trade-off compared to its autoregressive counterpart (rows 1-3). For instance, our framework with BERT-Base [17] (row 4) outperforms the autoregressive variant based on GPT-Neo-1.33 [7] (row 1) which uses 12 times more parameters and 8 times more training time. Likewise, our framework with DeBERTa-Ve -2---22-X [25] (row 6) improves over the autoregressive variant based on GPT-J-6B [94] (row 3) that has 7 times more parameters and requires 5 times more training time, showing the efficiency of our bidirectional framework for zero-shot VideoQA.
4.4 Comparison to the state of the art for zero-shot VideoQA
Quantitative comparison. Table 5 presents results of our method in comparison to the state of the art in zero-shot VideoQA settings [101], i.e. when using no manually annotated visual data for training. Our approach outperforms previous methods by significant margin all 8 datasets. In
8

---

Quastion: What is the man
Question: What ftem hanging
Question: What the stting man thiin??
Question: Where is the wommn siting on?
Question: What is the enlor of
hulding at the start of the video?
the wall features trew? GT Answer:
cabirul door the video? GT Answer: Just dresser
Answer: elactric gullar Just Adk: typewriter
GT Arswer: knil smaater
GT Answer camel Just UnfiromenBiMM::
Ask: christmas sock
Ask:
UnfresendiLMM:
soemming
UnfroaendilMt::
Frosenbil (tast-anlyy;
FrouentilLd (tweb-nnlyl:
(suct--nnyy]: wating
FrazenbiLM (tect-only):chiii
FravendiLM Jsact-anlyjt FrozenBiLM lours]:
FrosenBILM (oursje guitar
FrozenBLLM [our]]: quilt
FrosenBil M (ours)t sweater
FrossnBILM (ouns): camull
Figure 3: Zero-Shot VideoQA. Qualitative comparison between Just Ask [102] (row in Table 5), model
(row in Table 5), its unfrezen variant (row Table 1) and its text-only variant (row 2 Table 2). The first two examples are from iVQA [101] and the last three examples are from ActivityNNe-QAA [108].
Trained Fill the blank!
Open-ended
Multiple -hoiip
Method
Params
LSMDC
IVQA MSRVTT-QA
MSVD-QA ActivityNet-QA TGIF-QA QA 2QA TVQA
HCRN[45] [45]
44M
33.4
36.8
57.9
71.4"
HERO [54]
119M
74.1"
73.6*
ClipBERT [48]
114M
37.4
60.3
Just Ask[ [102]
157M
35.4
41.8
47.5
85.3
SiaSamaea [107]
41.6
45.5
39.8
60.2
84.1
MERLOTT [109]
$2.9
Reserve [1 10]
644M
86.1
VIOLET [21]
198M
53.7
43,9
47.9
68.9
All-in-one [93]
110M
46.8
48.3
66.3
UnFregeBiLM (Ours)
890M
$8.9"
37.7
45.0"
53.9
43.2"
66.9
87.5
79.6"
ProgeRil w/o speech (Oum)
30M
58.6
39.7
47.0
54.4
43,2
68.6
81.5
57.5
ProzenBiLM (Ours)
30M
63.5
39.6'
47.0"
54.8"
43.2"
68.6
86.7
82.0"
Table 6: Comparison with the state of the art, and the variant UnFrozenBiLMM which does not freeze the language model weight, on fully-supervised benchmarks. denotes results obtained with speech input.
Supervision
til-in-the-blank LSMDC
Open-ended MSVD-QA Activity TGIF-QA
Multiple-choice How2QA TVQA
iVQA
MSRVTT-QA
0% (zero-shot)
51.5
26.8
16.7
33.8
25.9
41.9
58.4
39.7
2. 56 (few-shot)
56.9
31.1
36.0
46.5
33.2
55.1
71.7
72.5
(few-shat)
59.9
35.3
37.4
61.2
75.8
77.6
4. 100% (ully-supervised)
63,5
39.6
47.0
54.8
43,2
68.6
86,7
82.0
Table 7: Few-shot results, by finctuning FrozenBiLM using small fraction of the downstream training dataset.
particular, FrozenBiLMM outperforms Reserve [110], which is trained on one billion YouTube video clips jointly with vision, language and sound, Just Ask [102], which uses large-scale automatically generated VideoQA data, and a CLIP baseline [71] matching the text concatenating question and answer the middle frame of the video. Note that FrozenBiLM performs competitively even when using no speech input. Finally, we note that BLIP [53] has a different definition of zero-shol where a network finetuned on the maggeVQQA dataset [4] is evaluated directly on VideoQA datasets. Our Supplementary Material presents results where we outperform BLIP [53] in their settings and also includes an analysis of results by question type. In summary. our evaluation shows the excellent performance of our model in the challenging zero-shot setup.
Qualitative results. Figure 3 illustrates qualitative results of zero-shot VideoQA for our FroZenBiLM model and compares them to Just Ask [102), as well as to variants of our approach that do not freeze the language model (UnFrozeniiLMM and use visual modality (text-only), evaluated in Section 4.2. We observe that the unfrozen variant can predict answers that lack text-only commonsense reasoning, egg. in the third example, it is unlikely that sitting is swimming. The text-only variant does have strong language understanding, but makes visually-unrelate contrast, consistently with our quantitative results, our model FrozenBiLM is able correctly answer various questions, showing both strong textual commonsense reasoning and a complex multi-modal understanding. We show additional qualitative results in the Supplementary Material.
predictions. In
4.5 Freezing the BiL.M also beneficial in supervised settings
Fully-supervised VideoQA. We next present an evaluation in supervised setup where we finetune FrozenBiLM on downstream VideoQA task. We emphasize that we also keep our pretrained language model weights frozen all throughout finetuning. As shown in Table 6, approach improves the state

---

of the art on LSMDC-FiB iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA. In particular, FroZenBiLM outperforms strong recent baselines such as All-in-one [93] on 2/3 datasets, VIOLET [21] on 3/4 datasets and MERLOT [109] on 4/5 datasets. Our approach has significantly less trainable parameters compared to the state of the art [21, 93, 109] as we freeze the weights the pretrained language model. We ablate this major difference in Table 6, and find that our FrozenBiL.M with the frozen language model performs better and trains twice faster compared to UnFrozenBiLM where we update the language model during training. This shows that freezing the language model is not only beneficial for zero-shot but also in fully-supervised settings, therefore suggesting that our FrozenBiLM framework also provides a parameter-efficient solution for VideoQA training. Finally, we note that FrozenBiLM performs competitively even without speech input, although speech helps significantly for the performance on LSMDC, How2QA and TVQA.
Few-shot VideoQA. The low number of trainable parameters when training FrozenBiL.M makes particularly well-suited in the low data regime. To verify this, we explore a few-shot VideoQA setting where we finetune our pretrained model using varying fractions of VideoQA training data. From Table 7 we observe significant improvements over zero-shot when using only 1 of training data. Finally, we show in Supplementary Material that freezing the BiLM highly benefits the few-shot performance, consistently with the results in the zero-shot and fully-supervised settings.
5
Conclusion
We have presented FrozenBiLM, a framework that extends frozen bidirectional language models to multi-modal inputs by training additional modules on Web-scraped data, and that tackles zero-shot VideoQA through masked language modeling. We have provided extensive ablation studies and shown the efficiency of our framework compared to its autoregressive variant. FrozenBiL.M improves the state-of of-the-art zero-shot VideoQA on various datasets, performs competitively in fully-supervieed settings and exhibits strong performance in the few-shot VideoQA setting we newly introduce.
Limitations. Promising directions not explored in this work include scaling the size of a bidirectional language model to several billion parameters, and additional training large datasets of You videos with accompanying speech transcripts and/or audio [1 [1]. Also, our model cannot be applied out-of-the-box to complex multi-modal text generation tasks such as video captioning.
Broader Impact. We have showed the superior compute-efficiency of our bidirectional framework compared to autoregressive models for zero-shot VideoQA, and believe it is step towards reducing the environmental impact of such research and its applications [84]. In addition, our models might reflect biases present in vidcos and captions from Shutterstock used to train
model, the text data used to train the language model or the images and captions used to train the visual backbone. is important to keep this in mind when deploying, analysing and building upon these models.
Acknowledgements. This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD01101 1670R2 made by GENCI. The work was funded by a Google gift, the French government under management of Agence Nationale de Recherche as part of the "Investissements d' d'avenir" program, reference 1NR-11--31A---
(PRAIRIE 31A Institute), the Louis Vuitton ENS Chair on Artificial Intelligence, the European Regional Development Fund under project IMPACT (reg. no. feedback. We thank Gaspard Beugnot, Clémence Bouvier and Pierre-Louis Guhur for proofreading.
003/0000468). We thank anonymous reviewers for giving interesting
10

---

References
[1] FrozenBiL.M project webpage. https://antooyagg. https://atttyaggggttbbbbb////
html.
[2] Bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[3] Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. Noise estimation using density estimation for self-supervised multimodal learning. In AAAI, 2021.
[4] Stanislaw Antol, Aishwarya Agrawal. Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C Lawrence Zitmick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015.
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint urXiv:1607.06450, 2016.
[6] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen time: joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
[7] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, 2021.
[8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[9] Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang. Jia Deng, and Rada Mihalcea. LifeQA: A real-life dataset for video question answering. In LREC, 2020.
[10] Aman Chadha, Gurneet Arora, and Navpreet Kaloty. iPerceive: Applying common-sens: reasoning to multi-modal dense video captioning and video question answering. In WACV, 2021.
[11] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. VisualGPT: Data- efficient adaptation of pretrained language models for image captioning. arXiv preprint vrXiv22102.10407 2021.
[12] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In ECCV, 2020.
[13] Seongho Choi, Kyoung- Woon On, Yu-Jung Heo, Abjeong Seo, Youwon Jang, Seungchan Lee, Minsu Lee, and Byoung- Tak Zhang. DramaQA: Character-centered video story understanding with hicrarchical qa. In AAAI, 2021.
[14] Anthony Colas, Seokhwan Kim, Franck Dernoncourt, Siddhesh Gupte, Daisy Zhe Wang, and Doo Soon Kim. TutorialVQA: Question answering dataset for tutorial videos. In LREC, 2020.
[15] Long Hoang Dang, Thao Minh Le, Vuong Le, and Truyen Tran. Object-centric representation learning for video question answering. In IJCNN, 2021.
[16] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE TPAMI. 2021.
[17] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[19] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA-multimon augmentation of generative models through adapter-based finetuning. arXiv preprint 212 2021.

---

[20] Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang. and Heng Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In CVPR, 2019.
[21] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. VIOLET: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.1266 2021.
[22] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale ad adversarial training for vision-and saiio-and-language representation learning. In NeurlPS, 2020.
[23] Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In CVPR, 2018.
[24] Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. KnowlT VQA: Answering knowledge-based questions about videos. In AAAI, 2020.
[25] Pengcheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- enhanced BERT with disentangled attention. In /CIR, 2021.
[26] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean Saan-Baptsste Alayrac, and Aida Nematzadeh. Decoupling the role of data, attention, and losses in multimodal transformers. In TACL, 2021.
[27] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[28] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffiient transfer learning for nlp. In ICML, 2019.
[29] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[30] Deng Huang, Peihao Chen, Runhao Zeng, Qing Du Mingkui Tan, and Chuang Gan. Location- aware graph convolutional networks for video question answering. In AAAI, 2020.
[31] Zhicheng Huang, Zhaoyang Zeng. Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-BERT: Align- ing image pixels with text by deep multi-modal transformers. arXiv preprint 0004 2020.
[32] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. TGIF-QA: Toward spatio-temporal reasoning visual question answering. In CVPR, 2017.
[33] Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering. In AAAI, 2020.
[34] Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In AAAI, 2020.
[35] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. In TACL, 2020.
[36] Hyounghun Kim, Zineng Tang, and Mohit Bansal. Dense-caption matching and frame-selection gating for temporal localization in VideoQA. In ACL, 2020.
[37] Junycong Kim, Minuk Ma. Trung Pham, Kyungsu Kim, and Chang D Yoo. Modality shifting attention network for multi-modal video question answering. In CVPR, 2020.
[38] Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tale Zhang. Deepstory: Video story qa by deep embedded memory networks. In IJCAI, 2017.
12

---

[39] Seonhoon Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak. Self-supervised pre-training and contrastive representation learning for multiple-choice video qa. AAAI, 2021.
[40] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and -ndd--nndd-aan and-language transformer without convolution or region supervision. In ICML. 2021.
[41] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In ACL, 2018.
[42] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations, In ICLR, 2020.
[43] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In CVPR, 2020.
[44] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Neural reasoning, fast and slow, for video question answering. In IJCNN, 2020.
[45] Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for multimodal video question answering. In IJC,, 2021.
[46] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. TVQA: Localized, compositional video question answering. In EMNLP, 2018.
[47] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. TVQA+: Spatio-temporal grounding for video question answering. In ACL, 2020.
[48] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara I. Berg, Mohit Bansal, and Jingjing Liu. Less is more: ClipBERT for video-and-languaar learning via sparse sampling. In CVPR, 2021.
[49] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[50] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: aoroo--nn--lanaage pre-training with entity prompts. arXiv preprint arXiv:2112.09583, 2021.
[51] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou, Unicoder-VIL VL: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
[52] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurlPS, 2021.
[53] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086. 2022.
[54] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for video+Language omni-representaaioon pre-training. EMNLP, 2020.
[55] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang. et al. VALUE: A multi-task benchmark for video-and-lamee and-aaggag understanding evaluation. NeurIPS Track on Datasets and Benchmarks, 2021.
[56] Liunian Harold Li. Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.0355) 2019.
[57] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
13

---

[58] Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond RNNs: Positional self-attention with co-attention for video question answering. In AAAI, 2019.
[59] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks, In ECCV, 2020.
[60] Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresami. VX2TEXT: End-to-end learning of video-based text generation from multimodal inputs. In CVPR, 2021.
[61] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pretraining approach. arXiv preprint
2019.
[62] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: visiolinguistic representations for vision-and aibon-and-language tasks. In NeurIPS, 2019.
Pretraining task task-tgn
[63] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020.
[64] Ziyang Luo, Yadong Xi, Rongsheng Zhang, and Jing Ma. VC-GPT: Visual conditioned GPT for end-to-end generative vision-and aibon-and-lewd und-language pre- mre-trainnng. arXiv preprint arXiv:2201.122 127233 2022.
[65] Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani. PERFECT: Prompt-free and efficient few-shot learning with language models. In ACL, 2022.
[66] Tegan Maharaj. Nicolas Ballas, Anna Rohrbach, Aaron Courville, and Christopher Pal. A dataset and exploration of models for understanding video data through fill-i-tthe-blank question-answering In CVPR, 2017.
[67] Antoine Miech, Dimitri Zhukov, Jean- -aattBattstte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
[68] Ron Mokady, Amir Hertz, and Amit H Bermano. ClipCap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09733 2021.
[69] Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. MarioQA: Answering questions by watching gameplay videos. In CVPR, 2017.
[70] Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Bridge to answer: Structure-aware graph interaction network for video question answering. In CVPR, 2021.
[71] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
[72] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified texi-to-text transformer. JMLR, 2020.
[73] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.0522 2016.
[74] Arka Sadhu, Kan Chen, and Ram Nevatia. Video question answering with phrases via semantic roles. In NAACL, 2021.
[75] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistiBBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 2019.
14

---

[76] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. In EACL, 2021.
[77] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACL, 2021.
[78] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. Look before you speak: Visually contextualized utterances. In CVPR, 2021.
[79] Paul Hongsuck Sco, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end genera- tive pretraining for multimodal video captioning. In CVPR, 2022.
[80] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and siiionand-langee and-language tasks? arXiv preprint arXiv:2107.06383 2021.
[81] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
[82] David R So, Wojeiech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Primer: Searching for efficient transformers for language modeling. arXiv preprint arXiv:2I09.0866 2021.
[83] Xiaomeng Song.
Yucheng Shi, Xin Chen, and Yahong Han. Explore multi-step reasoning in video question answering. In ACM international conference on Multimedia, 2018.
[84] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL, 2019.
[85] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of generic visual- lingu--ing representations. In ICLR, 2019.
[86] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT:
A joint model for video and language representation learning. In ICCV, 2019.
[87] Derek Tam, Rakesh Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. In EMNLP, 2021.
[88] Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
[89] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding stories in movies through guestion-answering. In CVPR, 2016.
[90] Wilson Taylor. "cloze procedure": A new tool for measuring readability. Journalism quarterly, 30(4):415 1953.
[91] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NeurIPS, 2021.
[92] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. In NeurIPS, 2017.
[93] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv preprint arXiv:2203.07330,, 2022.
[94] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A Billion Parameter Autoregressive Language Model, 2021.
[95] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: unified transformer for vision-language representation learning. arXiv preprint 211102211,. 2021.
15

---

[96] Zhenhailong Wang. Manling Li, Ruochen Xu. Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang. Ziyi Yang. Chenguang Zhu, Derek Hoiem, et al. Language models with image descriptors are strong few-shot video-language learners. In NeurlPS, 2022.
[97] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NEXT-QA: Next phase of question-answering to explaining temporal actions. In CVPR, 2021.
[98] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM international conference on Multimedia, 2017.
[99] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021.
[100] Hongyang Xue, Wenqing Chu, Zhou Zhao, and Deng Cai. A better way to attend: Attention with trees for video question answering. IEEE Transactions on Image Processing, 2018.
[101] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning answer questions from millions of narrated videos. In ICCV, 2021.
[102] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Learning to answer visual questions from web videos. IEEE TPAMI, 2022.
[103] Zekun Yang, Noa Gareia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. BERT representations for video question answering. In WACV, 2020.
[104] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of GPT-3 for few-shot knowledge-based VQA. arXiv preprint arXiv:2109.0540 05014, 2021.
[105] Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yucting Zhuang. Video question answering via atrribute-augmentdè attention network learning. In ACM SIGIR, 2017.
[106] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. In AAAI, 2020.
[107] Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji. Lijun Wu, Nong Xiao, and Nan Duan. Learn- ing from inside: Self-driven siamese sampling and reasoning for video question answering. In NeurIPS, 2021.
[108] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet-OU QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
[109] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. MERLOT: Multimodal neural script knowledge models. In NeurIPS, 2021.
[110] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERLOT Reserve: Neural script knowledge through vision and language and sound. In CVPR, 2022.
[111] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. So- cratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0055 2022.
[112] Zueng-Jun Zha, Jiawei Liu, Tianhao Yang, and Yongdong Zhang. Spatiotemporrll-exttue co- attention network for video question answering. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2019.
[113] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for languaee models. arXiv preprint 210:::2000001134,, 2021.
16

---

[114] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. In AAAI, 2020.
[115] Linchao Zhu and Yi Yang. ActBERT: Learning global-local video-text representations. In CVPR, 2020.
[116] Yueting Zhuang, Dejing Xu, Xin Yan, Wenzhuo Cheng, Zhou Zhao, Shiliang Pu, and Jun Xiao. Multichannel attention refinement for video question answering. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2020.
2020.
17

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section for contribution (i), Sections 4.2 and 4.3 for contribution (ii), and Sections 4.4 and 4.5 for contribution (iii).
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Our code, together with instructions needed to download and process the datasets we use, and instructions needed to reproduce the main experimental results, is open-soureed [1]. (b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4. and Supplementary Material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report them in the Supplementary Material, as error bars are in general not reported [101, 107, 109, 110].
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4. and Supplementary Material.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.
(b) Did you mention the license of the assets? [Yes] See Supplementary Material.
(c) Did you include any new assets either in the supplemental material
URL? [Yes]
We provide code and trained models at [1].
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The datasets we use are publicly available and released for non-commercial use only, and this is already specified in the license.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The datasets we use are based on websites such YouTube which strictly remove videos that contain offensive content do not. follow their community guidelines.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A] No crowdsourcing or conducted research with human subjects. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] No crowdsourcing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No crowdsourcing or conducted research with human subjects.
18


Reviews:
[{"review_id": "hkdciq9JU1Q", "reviewer": "Reviewer_bWDb", "summary": "This paper proposed to leverage the power of a frozen bidirectional LM to tackle the zero-shot video question-answering task. Adapters and projection layers are added to the model as the only trainable weights, so it's a very lightweight optimization. Also, the frozen backbones enable us to fine-tune the model without forgetting the knowledge of the pretrained backbones, so it can achieve much better performance compared to fine-tuning the whole model. The proposed model achieves state-of-the-art performance on many zero-shot VideoQA datasets. The experiments in the fully-supervised setting also show that it's possible to apply the same method with much training data and achieve state-of-the-art results.", "questions": "- Is it possible to adapt a pretrained speech module to this framework? As audio is crucial information in videos. Merely adding ASR transcripts to the model can not fully preserve the audio information. If we can incorporate pretrained models like wav2vec, this direction would be much more interesting.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact in the Conclusion section.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written. The flow is easy to follow and the method is clear and simple.\n- The authors carefully design the prompting templates for using the frozen model out-of-the-box.\n- The ablation study is comprehensive, we can clearly see that a frozen LM and adapters and the speech modality are necessary to to achieve good performance.\n- The result shows the effectiveness of this proposed method: it can outperform a fine-tuning baseline or an autoregressive baseline by a large margin.\n- The same method can also be applied to the fully-supervised setting and it can achieve state-of-the-art results when compared with other supervised baselines.\n\nWeaknesses:\n- The idea of utilizing a frozen LM module is not new (e.g. https://arxiv.org/abs/2106.13884 which is a paper using autoregressive LMs to perform image-text few-shot learning). Although this method mainly focuses on VideoQA and BiLM, its novelty is still limited. However, this paper has a good engineering value, as no one has tried to use a frozen BiLM on VideoQA tasks. This paper is the first to do that can achieve surprisingly good results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eSctdZRa-v", "reviewer": "Reviewer_daCv", "summary": "This paper mainly concerns the problem of zero-shot video question answering (VideoQA). It proposes FrozenBiLM, which performs video-text(-speech) pre-training over a frozen language model to enable it with visual understanding capability while preserving its language generalization ability. The resulted model could perform zero-shot VideoQA out of the box. The paper also demonstrates the capacity of FrozenBiLM in terms of few-shot/fully-tuned learning. New SotAs are set on a wide range of mainstream VideoQA benchmarks.", "questions": "Please address the following questions:\n\ni) In line 217, it is evident that a simple averaging over multiple tokens does not preserve the semantic structure of the label/phrase (e.g., \"man riding horse\" vs. \"horse riding man\"). Hence, the resulted model could suffer in the wild when *rare* events/relationships happen (e.g., \"horse riding man\"). It would be great if the authors can shed some lights on possible solutions and perhaps other strategies they have attempted in the experiments to alleviate this issue. One of the related works is [r5].\n\nii) Considering that not all existing methods have speech transcripts as model input and the fact that transcripts are extremely informative esp. on instructional videos, the comparison seems unfair (e.g., in Tab. 6). Please comment on this and also denote methods with speech input in the result tables.\n\n[r5] Salazar et al., Masked Language Model Scoring. ACL 2020.", "limitations": "Appears to be sufficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Despite that adapting a frozen language model for VL tasks is not new (e.g., [88]), nor is adaptor-based transfer learning for Transformers (e.g., [26] and LoRA [r1]), the paper studies this simple yet important baseline on the challenging problem of VideoQA. The overall idea is convincing and well-executed. The experimental results are solid and adequately support the claims from the paper w.r.t. the importance of freezing the language model and adaptors on cross-domain transfer learning. The paper flows well and is easy to follow.\n\nNote that some relevant concurrent work such as Flamingo [r2] is missing in the related work, considering its relevance and the similar model design. Other related work on zero-shot/few-shot VideoQA include [r3] [r4].\n\n[r1] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models. arXiv 2021.\n\n[r2] Alayrac et al., Flamingo: a visual language model for few-shot learning. arXiv 2022.\n\n[r3] Xu et al., Videoclip: Contrastive pre-training for zero-shot video-text understanding. EMNLP 2021.\n\n[r4] Wang et al, Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners. arXiv 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BeuJpAWzhE7", "reviewer": "Reviewer_MVzv", "summary": "The paper addresses zero-shot video question answering by leveraging pre-trained language models and visual encoders. The proposed model encodes videos with CLIP and projects them into language. The language is encoded with a bidirectional language model (e.g. BERT) and trained with a masked language modeling objective. The paper only re-trains the visual projection and adapter layers on a large video-captions dataset, and it shows that freezing the language model and the visual encoder leads to better results than re-training everything. Extensive experiments are conducted and results are reported on 8 videoQA datasets.", "questions": "- The drop in performance when removing the suffix is surprising. Does something similar happen when removing the prefix ([CLS])?\n- How important is the prompt design for the zero-shot performance? Have you tried with different prompt templates?\n- Have you tried using the model for image VQA?", "limitations": "Already discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well-motivated, well-written, and easy to understand. The ideas proposed in the paper (joining pre-trained models together and use them for zero-shot videoQA without retraining) are simple and elegant. Multiple experimental results and ablations on 8 datasets support the hypothesis of the paper.\n\n- The results of the paper are significant to the community: \n  1. Using frozen language models (with no visual information training at all) can lead to better performance than fine-tuning such models on visual-language datasets probably due to catastrophic forgetting.\n  2. Smaller language models like BERT and its variants can outperform larger language models based on GPT.\n  3. Models pre-trained for different tasks can be made to work together by only fine-tunning some adapter layers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept