Mulan: A Multi-Level Alignment Model for Video Question Answering
Authors: Yu Fu
Cong Cao
Yuling Yang
Yuhai Lu
Fangfang Yuan
Dakui Wang
Yanbing Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Video Question Answering
Multi-Level Alignment
Contrastive Learning

Abstract: Video Question Answering (VideoQA) aims to answer questions about the visual content of a video. Current methods mainly focus on improving joint representations of video and text. However, these methods pay little attention to the fine-grained semantic interaction between video and text. In this paper, we propose Mulan: a Multi-Level Alignment Model for Video Question Answering, which establishes alignment between visual and textual modalities at the object-level, frame-level, and video-level. Specifically, for object-level alignment, we propose a mask-guided visual feature encoding method and a visual-guided text description method to learn fine-grained spatial information. For frame-level alignment, we introduce the use of visual features from individual frames, combined with a caption generator, to learn overall spatial information within the scene. For video-level alignment, we propose an expandable ordinal prompt for textual descriptions, combined with visual features, to learn temporal information. Experimental results show that our method outperforms the state-of-the-art methods, even when utilizing the smallest amount of extra visual-language pre-training data and a reduced number of trainable parameters.

Mulan: A Multi-Level Alignment Model for Video Question Answering
Yu Ful.? Cong Cao Yuling Yang! Yuhai Lu",
Fangfang Yuan Dakui Wangl.2 Yanbing Liu
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China School of Cyber Security, University of Chinese Academy of Sciences, Beijing. China *China Academy of Information and Communications Technology, Beijing, China (fuyu1998, caocong, yangyul ing, yuanfangfang wanggakkuui LiuyanbingJ@tie.acc luyuhai@caict.. wa..tttt........................ ac cn
Abstract
(e) Exisliag methods
Video-Level Aligniment
Video Question Answering (VideoQA) aims answer questions about the visual content of a video. Current methods mainly focus on improving joint representations of video and text. However, these methods pay little atten- tion to the fine-grained semantic interaction between video and text. In this paper, we pro- pose Mulan: Multi-Level Alignment Model for Video Question Answering. which estab- lishes alignment between visual and textual modalities at the object-level, frame-level, and video-level. Specifically, object-level align- ment, propose mask-guided visual feature encoding method and visual-guided text de- scription method learn fine- fine-grained spatial information. For frame-level alignment, we in- troduce the use visual features from individ- ual frames, combined with caption generator, to learn overall spatial information within the scene. For video-level alignment, we propose expandable ordinal prompt for textual de- scriptions, combined with visual features, to learn temporal information. Experimental re- sults show that method outperforms the state-of-the-art methods, even when utilizing the smallest amount of extra vinual-language pre-training data and reduced number of train- able parameters. Our code publicly available https://ittub..com/fuuu 1998/Mulan.
woman cooking
Sampled Frazee
Subtiilee
method
Video-Level Alignment
cooking
Sampled
Subtiee
Frame-Level Alignment
cunting..
Sampled
Global Captiaas
Object-Lavel Alignment
Objecti
Object2 dog
Tramel:
Framm2:
spreah
Obpect Masced
Prozze Object Descrittios
Figure 1: The difference between existing methods and method. In contrast to existing methods that solely establish visual-language language alignment at the video level, our method incorporates vingalelanguage alignment at multiple levels.
1 Introduction
Xu et al., 2023) commonly adopt two-step ap- proach to establish a joint representation of video and text. First, an image-language model (He et al., 2016; Radford et al., 2021) utilized to extract visual and textual features from the video frames. Then, alignment between the visual and textual modalities is established based on the extracted vi- sual and textual features. However, existing meth- ods align video and text globally learn semantic correlations, disregarding the fine-grained interac- tion between local salient information the video and important textual descriptions. Furthermore,
Video Question Answering (VideoQA) is a task that focuses on answering questions related to the represen-
visual content of video. It serves tative task that showcases the fusion of visual and linguistic modalities. It demands the ability to com- prehend and integrate data from both modalities to learn complex spatiotemporal information.
Given videos are a continuous sequence of images, the mainstream models (Li et al., 2020, 2022c; Yang et al., 2021, 2022a: Zellers et al., 2021;
"Corresponding authors.

---

the majority of subtitles in the dataset are brief, re- sulting in a lack of detailed textual descriptions for significant video content, which further undermines the effectiveness of semantic alignment. Therefore, relying solely on global alignment is insufficient to address the semantic gap issuc.
27M
17M
12.98
Ours
of PT
SOTA
2.5M
2.5M
IVQA
MSRVIT-QA
MSVD-QA
TOP-1 Acd (%)
To address the above issues, we propose Mu- lan: a Multi-Level Alignment Model for Video Question Answering. Specifically, we establish alignment between visual and textual modalities at the video-level, frame-level, and object-level, as shown in Figure 1. Our method can be divided into three stages:
47.4
48,0
58.2
59.2
Figure 2: Comparison of the size extra pre-training dataset and performance between our method and state- of-the-art art methods. Our method demonstrates superior performance compared to state-of- thrhe-r methods, even when utilizing extra pre-training datasets that least one order of magnitude smaller.
(1) Stage one involves the process of multi-level visual features generation. The image encoding of all sampled frames is regarded as video-level visual features. The image encoding of an individ- ual frame is regarded frame-level visual features. For object-level visual features, have introduced a meask-guided object-level visual feature encoding method.
mal amount of visual-language extra pre-training data (Figure 2), and reduced number of train- able parameters. Moreover, as the size of the pre- training dataset increases, the performance of our method can be further improved.
Related work
(2) Stage two involves the process of multi-level textual description generation. The original descrip- tion of the video is regarded video-Ievel textual description. For frame-level textual descriptions, we have introduced the use of a caption genera- tor. For object-level textual descriptions, we have proposed a vision-guided generative approach.
2.1 Pre-Trained Image-Language Models
In the realm of visual and language fusion method- ologies, exploratory methods, as exemplified by the multi-stream approach(Lei et al., 2018), entail the utilization of a unified encoder to merge various information streams. This method facilitates the utilization of visual and language information at diverse levels of granularity and offers straightfor- ward scalability. However, information of the same granularity merged into distinct streams before aligmment, resulting in a relatively poor semantic consistency among these streams.
(3) Stage three involves the process of multi- level alignment and training. Specifically, we ini- tially convert visual features into inputs for a lan- guage model and design prompts to concurrently establish alignment at the video-level, frame-level, and object-level. Finally, we employ masked lan- guage modeling objective for training the language model, enabling it to acclimatize to visual input and establish multi-level alignment.
In recent years, contrastive learning has been substantiated as efficacious approach for acquir- ing cross-modal joint representations. The image- language pre-trained model, based contrastive learning, aims establish a mapping between global image features and global text features in a shared space (Radford et al., 2021: Li al., 2022c;
Our contributions can be summarised as follows:
(I) We introduce a multi-level visual-language alignment method for video question answering. To the best of our knowledge, this is the first work in the field of video question answering that ex- plores multi-level visual- languge alignment.
Lin et al., 2022; Jia et al., 2021; Wang et al., 2022b). These model considers aligned image-text pairs as positive samples and unaligned pairs as negative samples to optimize visual- vinual-language alignment.
(2) We give a clear and unified spatiotemporal in- formation learning framework. At the object- and the frame-level, the method can learn spatial information, while acquiring temporal information at video-level.
Due to the requirement of a substantial amount of training data for video-language models perform well, recent research has focused on transferring image-language models to video-text tasks (Yang et al., 2022a; Li et al., 2023b; Ych al., 2023). This is because there a strong correlation
(3) The experimental results illustrate that our method surpasses state-of-the the baselines in video question answering tasks, even when using a mini-

---

between images and videos. Sparse sampling offers an effective approach video representation (Li et al.. 2022a; Lei et al., 2021), contrasting with the use of 3D dense features (Feichtenhofer et al., 2019). facilitates the more efficient utilization of pre-trained image-language models within the realm of video processing tasks (Wang et al., 2022e; Fang et al., 2021; Luo et al., 2022).
visual features. Global frame visual features pro- vide a comprehensive representation of the overall spatial information, while local frame visual fea- tures accurately capture specific object details such as position and attributes. It is worth noting that in this process, mask generator is employed instead of an object detector. This choice is motivated by the fact that the output bounding boxes from an object detector are often not specific enough and can also struggle accurately recognize objects in an open-vocabulary setting. Furthermore, due the fact that both local frame visual features and global frame visual features are derived via a uni- fied image encoder, it allows for the coexistence of local frame visual features and global frame visual features within the same semantic space.
Our method continues along this line of method- ology by ingeniously constructing bridge from image-language to video-language through metic- ulous design, thereby facilitating the transfer of knowledge.
2.2 Language Models for Visual-Language Alignment
Some works focus on converting visual information entirely into symbolized textual information (Yang et al., 2022b; Wang et al., 2022e; Lin et al., 2023). Some other works use raw textual descriptions of videos, adopt the approach of freezing the weights of pre-trained language models, and integrate vi- sual information to address tasks that involve both visual and language processing (Alayrac et al., 2022; Eichenberg et al., 2022; Yang et al., 2022a). However, the first category of methods disregards visual information, particularly fine-grained visual details. Conversely, the second category of meth- ods lacks detailed textual descriptions and only achieves alignment at the video level.
Next, for global frame visual features, frame captions are generated using the image-language model BLIP (Li et al., 2022b). From these frame captions, the part-of-speech tagging tool provided by spaCy (Honnibal and Johnson, 2015) is em- ployed to extract noun phrases, thus establishing dynamic vocabulary tailored to the frame. The dynamic vocabulary terms are subsequently amal- gamated with predefined static vocabulary, thereby creating the vocabulary input for the object filter. The object filter is then utilized to extract global and local frame objects from the respective global and local frame visual features. It is worth noting that in this process, due the differences in the information contained in global and local frame vi- sual features, the extracted global and local frame objects may also differ.
In contrast, our method considers multi-level visual information and goes beyond by con- structing semantic representations with multi-level visual-language alignment. In addition, we use lightweight Adapter layers (Houlsby et al., 2019; Hu et al., 2022) and frozen the language model. Our model can be casily applicd to different types of language models, thereby facilitating practical application and deployment.
Finally, the global and local frame visual fea- tures, in conjunction with global and local frame object descriptions and global frame captions, serve as the input to language model equipped with the adapter. Within the information fusion process, alignment between frame-level and object-level has been established. Simultancouslyy temporal infor- mation is integrated by implementing an expand- able ordinal prompt approach, thereby enabling the establishment of video-level alignment.
3 Method
3.1
Overall Framework
Our method can be divided into three stages, as illustrated in Figure 3: (1) multi-level visual fea- ture generation, (2) multi-level textual description generation, (3) multi-level alignment and training. Firstly, for given sampled frame, an image encoder is applied extract global frame visual features. Then, a mask generator utilized to cre- ate masks for the objects within the frame, and the same image encoder is applied to obtain local frame
3.2 Multi-Level Visual Features Generation
To capture multi-level visual information, it is cru- cial understand both the relationships between objects within video frames, which are represented as global frame visual features, and the local se- mantic information of the objects, which are repre- sented as local frame visual features.

---

Description
Loral Frame Visual [Ooject
Obsecti
Chject2 hand orange
Local Franz
Object Masaal
Framell
Object Desertptions
Framee:
finger
Global Frzmm Visual Fostures (Frame- Jorel)
Object
Global Pruze
Image Bacodar
Object Filer
Frame2:
Framel:
kithen,
vegetible, board.
Descrptions
Ofject
(ibject Levell
Object
Captions
Global Frame Visual Features [GLOBAL]
Local Prime Visuil Reabirs
(Frume
Captoon Generstor
Frame1: woman cutting..
woman spirach mary
person cot. cop
Frame2: burch spinach
Mask
Frame3: man cuttingg..
Dynamic voreh
Static voceb
Sempied
Questian:
What does woman out spinach
Pranse
Caption:
Answer:
Fist, woman cutting Second, bunch of spinach ... Third, man cutting
[MASK].
Trgut Conrent [Video-1
Giobal:
First, (SLOBAL] kithenn woman. Second, (GLOSAU) vegetable, cut board. Thind, (GLOBAL) person, fruit.
Local:
Fistt, (LOCAL) woman, coat: [LOCAL] posdle.
[LOCAL] [LOCAL] hand, [LOCAL]
[LOCAL] spinach,
descriptions.
Video
Adager
Langrage Model with Adapler
Language
Output
Truiood
knfle
Answer
Prozest
Alignment Training
Figure 3: Overall framework of our method. (1) Multi-Level Visual Feature Generation. For each sampled frame, we use an image encoder generate glohal frame visual features and utilize the mask generator to guide the generation of local frame visual features. (2) Multi-Level Textual Description Generation. We utilize caption generator to extract frame captions and construct both dynamic and static vocabulary. The object filter then utilized generate frame objects based on these vocabularies. (3) Multi-Level Alignment and Training. We integrate visual and textual information establish object-level and frame-level alignment, incorporating temporal cues to extend the alignment to the video level. Subsequently, we utilize language model equipped with adapters obtain the answers.
Global Frame Visual Features. The video is represented as a sequence of frames f = [ ob- tained through uniform and sparse sampling, where T is the total number of sampled frames. Each frame f: is individually encoded using the image encoder dCLIP to generate global frame visual fea- tures G.
Given the effectiveness of contrastive learning- based image-language models in learning shared representations of images and text in common space, employ CLIP (Radford al., 2021), contrastive learning- based model, as the image encoder. However, due its focus on capturing global information of images, CLIP not well- suited for directly capturing local details.
ad- (1) dress this issue, we leverage the existing unsuper- vised mask generator CutLER (Wang et al., 2023b) (2) to guide the image encoder CLIP in generating local frame visual features.
= =CLIP(I) D.
c (of e TTD.
where D, is the dimension of the visual features. As G. represents the visual features of the origi- nal frames, it contains the overall spatial informa- tion of those frames. Therefore, we consider G to represent frame-level visual information. Dur- ing the experimental process, the image encoder is frozen, meaning that its weights
To obtain local frame visual features, we first utilize the mask generator CutLER dCuLER gen- erate a set of image masks m, for each frame f,. We refer to the collection of mask sets for each frame as m.
updated. Local Frame Visual Features. Our goal is to establish multi-level iisual-langure alignment.
m; = ΦCOiLER (fa)
(3)
m = {m;}
(4)

---

Then, we apply cropping and masking opera- tions to the images and feed them into the image encoder ΦCLIP to obtain the local frame visual fea- tures v²:
learning, to compute the text features r for the final vocabulary V:
r= (OCLIP-ext (V:)
(8)
of = OCLIP (Teropp @ mi)) E M.D.
(5) where the size of the final vocabulary V. Finally, we determine object descriptions for each visual feature by evaluating the cosine sim- ilarity between the visual features Ge or v and text features For the global frame visual features oc we generate a total of M global object de- scriptions for each visual feature, denoted as 1 Similarly, for the local frame visual features of. we generate total of M local object descriptions for each visual feature, denoted as t².
wt tuf E M)x..
(6)
where (-) denotes the operations of cropping and masking, and @ is the Hadamard product oper- ation. We assume that the number of masks for the i-th frame is N₁.
Due to the removal of object-irrelevan regions in o², it focuses solely on the target objects them- selves. Therefore, we consider n represent object-level visual information. During the experi- mental process, the mask generator is also frozen.
Due to the different focuses of global frame vi- sual features S and local frame visual features v², there also differences in the global object de- scriptions g and local object descriptions t', We consider 1 and 1f to represent object- textual information.
3.3 Multi-Level Textual Description Generation
To capture multi-level textual information, it is crucial to utilize both the text that describes the relationships between objects, represented by the global frame captions, and the detailed information about the objects, represented by the descriptions of global and local frame objects.
3.4 Multi-Level Alignment and Training
Multi-Level Alignment Prompting. To achieve Visual-Language alignment, we integrate the ob- tained object-level and frame-level visual features with their corresponding textual descriptions. This integration allows us to align and fuse the visual and textual information at both the object-level and frame-level. Then, we integrate expandable tempo- ral markers to facilitate visual-languse alignment at the video-level.
Global Frame Captions. To establish textual connections among objects in the frame and ob- tain an overall description of the frame's informa- tion, we utilize the image-language model BLIP (Li et al., 2022b) to generate global frame captions == {c:)-. We consider to represent frame-level textual information.
As illustrated in Figure 3, have devised the following prompt:
Global and Local Frame Object Descriptions. Global frame captions alone may capture the fine-gre -ra-grained details of objects. Therefore, con- structing textual descriptions only at the frame level is insufficient. To address this issue, we further utilize global and local frame visual features to construct global and local object descriptions. Firstly, we construct a predefined static vocab- ulary VS. which includes set of candidate ob- ject names and attributes. However, the predefined static vocabulary cannot cover all objects and at- tributes. To address this issue, we utilize spaCy to extract noun phrases from the global frame cap- tions c, creating a dynamic vocabulary D The final vocabulary V is defined as follows:
"Question: <Question? Answer: [MASK].
Caption: <Caption>. Global: <Global Objects>.
Local: <Local Objects>. Subtitles: <additional description>."
For the alignment at the object-level, con- sider "a visual feature and its corresponding object description" the alignment unit. We iteratively combine these units to align the global and local frame visual features (v or 2²²) with their respec- tive object descriptions (tC or t.). To leverage the inductive bias of language proximity, we sort the object descriptions in descending order based on their cosine similarity between text features and visual features. Specifically, for each global frame visual feature "[GLOBAL]" and local frame visual fea- ture "[LOCAL]", we construct the following prompt: "[GLOBAL] <Description 1> ..." "[LOCAL] <escription 1>,
= V DVD
Next, we utilize the text encoder ÓCLIP-ext in the CLIP model, which based on contrastive

---

For the alignment at the frame-level, we con- sider all object-level information grouping by frame and global frame captions "First, <Caption
4 Experiments
4.1 Experimental Setup and Datasets
1>. Second, <Caption 2>. as the alignment unit.
General Setup. To ensure reproducibility, we uti- lize the same fixed seed for all experiments. Each experiment was conducted three times, and the crage results were reported as the final outcome. For all other methods, we report the experimental results as documented in the paper.
For the alignment the video-level, we con- sider the frame-level information and additional descriptions as the alignment unit. At the video- level, temporal information is indeed crucial for video understanding. Therefore, we incorporate expandable ordinal markers into our model, such "First," " "Second," " and so on.
Pre-Trained Models. We utilize the CLIP ViT-L/14 model (Radford et al., 2021) the im- age encoder ΦCLIP with an input image resolu- tion of 336 x 336. We utilize the Cutler (Wang et al., 2023b) model as the mask generator OCuLLER which leverages the Cascade Mask R-CNN (Cai and Vasconcelos, 2021) as the detector for gener- ating masks. We utilize the BLIP model (Li et al., 2022b) with ViT-L (Dosovitskiy al., 2021) the caption generator. We utilize the DeBERTa- -2-- Xargg model (He et al., 2021) as the language model.
Language Model with Adapter. We use the same Adapter layer as the Fr (Yang et al., 2022a) model. Each visual fcature is passed through a linear projection layer and incorporated into the language model as an individual prompt. The global and local visual features (r and o²) are linearly mapped using the projection matrices (PG and Pl)) to obtain the global and local feature prompts (u and )²
Note that in practical applications, the cap- (9) tion generator usually incorporates an image en- coder (Li et al., 2022b, 2023a; Wang al., 2022c), which facilitates the integration of the image en- coder and caption generator to streamline the pro- cess. In our experiments, we intentionally sepa- rated the image encoder and subtitle generator to ensure a fairer comparison with other CLIP-based methods.
wh = (v Ei (1 N
(10)
For autoencoder language models like De- BERTa (He et al., 2021), we employ a frozen MLM classifier head, denoted as m to predict answers from the vocabulary set A, which is constructed from the answers appearing in the training set. Multimodal Training. During training, we only update the parameters of the visual-to-text projec- tion modules and the adapter module. To en- sure consistency with language models, We use the visually-condittoneed Masked Language Mod- eling (MLM) objective, where some tokens {am} are randomly masked, and the model has to pre- dict these masked tokens based on the other tokens and the visual input. Formally, minimize the following loss:
Datasets. We pre-trained our model using the Vid-2M (Bain et al., 2021) dataset. We con- ducted evaluations on the iVQA (Yang et al., 2021), MSRVTT-QA (Xu et al.. 2017), MSVD-QA (Xu et al., 2017), and TGIF-FrameQA (Jang et al., 2017) datasets to assess the performance of our approach. Detailed information about the dataset is provided in the Appendix A.
Furthermore, given that pre-trained models are trained with additional datasets before being used, our method also incorporates these additional datasets. Our method predominantly integrates 400 million image-text pairs from the image encoder CLIP, along with 14 million image-text pairs from BLIP(Li et al., 2022b), and an additional 1.3 mil- lion image-text pairs from CutLER (Wang et al., 2023b). Therefore, we incorporate a total of 415.3 million image-text pairs through all pre-trained models.
C,(x,v)= M Σ log Pp (rr,
(11)
where ã is the corrupted text sequence, y is the sequence of visual input, (ā, 3m is the proba- bility for the (masked) m-th token in ã be F' and M is the number of masks in the sequence z. In detail, we adopt the same configuration as BERT (Devlin et al., 2019) for our method.
Implementation Details. Detailed implementa- tion information is provided in the Appendix B.

---

Trained
Extra
Method
iVQA
MSRVTT-QA
MSVD-QA
TGIf-FrameQA
Params
PT Data
FrozenBiLMM (Yang al., 2022a)
30M
10M
39.6
47.0
54.8
68.6
HiTeA et al., 2022)
297M
45,9
55.3
73.2
mPLUG-2 (Xu al., 2023)
17M
58.1
75.4
Flamingo-3B (Alayrac et al., 2022)
37.7
25.6
42,6
Plamingo-9B (Alayrac et al., 2022)
1.8B
40.7
47.2
Flamingo (Alayrac et al., 20222
27M
45.3
47.4
52,3
LAVENDER (Li al., 2023b)
198M
30M
45.0
56.6
73.5
Just Ask (Yang et al., 2021)
157M
69M
35.4
41.8
47.5
VideoCoCa (Yan et al., 2022)
2.1B
100M
39.0
46.3
56,9
InternViden (Wang et al., 2022d)
1.3B
110M
47.1
55.5
72.2
MERLOT (Zellers et al., 2021)
223M
180M
43.1
69.5
All-in-one ( (Wang et al., 2023a)
110M
283M
46.8
48.3
66.3
GIT (Wang et al., 2022a)
700M
800M
43.2
56.8
72.8
GIT2 (Wang et al., 2022a)
5.1B
12.9B
45.6
58.2
74.9
Mulan
31M
2.5M
47.4
48.2
58.2
72.0
Table I:
Comparison with the state of the fully-supervised benchmarks. Top-1 accuracy is reported. "Extra PT Data" refers to the quantity of extra video-text pairs and image-tex, pairs utilized during the pre-training phase. All methods are sorted ascending order based on the "Extra Data" from smallest largest. Throughout the paper, we utilize bold and underline formatting emphasize the top results.
Trained
Extra
Method
iVQA
MSRVTT-QA
MSVD-QA
TGIF-FrameQA
Params
PT Data
CLIP (Radford et al., 2021)
9.2
2.1
7.2
3.6
HiTeA (Ye et ul., 2022)
297M
5M
8.6
18.2
HiTeA (Ye et al., 2022)
297M
5M
21.7
37.4
FrozenBiLM (Yang et al., 2022a)
30M
10M
26.9
16.7
33.8
41.9
UnFrozenBiLM (Yang et al., 2022a)
890M
10M
21.0
17.6
31.9
30.7
mPLUG-2 (Xu et al., 2023)
17M
43.8
55.3
Flamingo-3B (Alayrac et al., 2022)
1.4B
27M
32.7
11.0
27.5
Flamingo-9B (Alayrac et al. 2022)
1.8B
27M
35.2
13.7
30.2
Flamingo (Alayrac et ul., 2022)
10B
27M
40.7
17.4
35.6
Just Ask (Yang et al., 2021)
157M
69M
13.3
5.6
13.5
Mulan
31M
2.5M
35.7
20.3
38.7
55.6
Table Comparison the state art on zero-shot benchmarks. Top-I accuracy is reported. We gray out the method additionally supervised pre-training VQA v2(Antol al., 2015) dataset.
4.2 Comparison with State-of-the-art
the dependency additional pre- pre-training data.
In this section, we conduct comprehensive eval- uation of our method by comparing it with state- of-the-art methods in both the fully-supervised and zero-shot settings.
Zero-Shot Benchmarks. Table 2 presents the results of our method compared to the state-of- the-art zero-shot methods. The few-shot results of our method can be found in Appendix C. Our method achieves state-of-the-an performance the TGIF-Frame dataset, while obtaining competi- tive results on the iVQA dataset. Similarly, we have utilized the smallest extra pre-training dataset and a reduced number of trained parameters. Similar to a fully-supervised setting, our method demonstrates relative advantage over FroZenBiLM (Yang et al., 2022a) in the zero-shot setting. In zero-shot tasks, the size of the pre-training dataset plays more crucial role compared to fully-supervised tasks. However, even with the smallest extra pre-training dataset, our method still achieves state-of-the-ard or competitive performance. This can be attributed to the effective transfer of the zero-shot capability
Fully-Supervised Benchmarks. Table 1 presents the results of our method compared to the state-of-the-an art sully-supervised methods. Our method achieves state-of-the-art performance on the iVQA, MSRVTT-QA and MSVD-QA datasets, while obtaining competitive results on the TGIF- FrameQA dataset. Note that we have utilized the smallest extra pre-training dataset compared to all method outperforms
methods. In particular, FrozenBiLM (Yang et al., 2022a), which also uti- lizes a language model with Adapter. This can be attributed to the effective transfer of the zero-shot capability of the mask generator, image encoder, and caption generator in our method, which reduces

---

Video Level
Frame Level
Object Level
IVQA
MSRVTT-QA
MSVD-QA
TGIF-FrameQA
26.5
16.3
31.0
25.3
/
35.5
19.6
36.7
/
30.3
18.3
35.3
47.5
35.7
38.7
55.6
Table 3: Ablation studies on frame-level and object-level alignment utilizing video-level alignment.
Video Level
Frame Level
Ohject Level
iVQA
MSRVTT-QA
MSVD-QA
TGIF-FrameQA
X
24.6
18.8
42.7
x
24.5
42.9
x
27.5
11.3
21.2
47.5
35.7
38.7
55.6
Table 4: Ablation studies frame-level and object-level alignment without utilizing video-level alignment.
60
compared to concrete information. As a result, our method achieves suboptimal performance in the fully-supervised setting.
-S-IVDA IVDA
50
MSRVIT-QA
Accuracy (88)
MSVD-QA
TGIF FramerAA
For the iVQA dataset, the zero-shot setting, our method demonstrates comparable performance to Flamingo-9B, which has 1.8 billion trained pa- rameters. In practice, it is commonly observed that larger trained parameter sizes tend to lead to improved generalization performance. Therefore, although method achieved suboptimal perfor- mance compared to Flamingo, which has 10 bil- lion trained parameters, it is still significant ac- complishment considering that our trained parame- ters constitute only 0.3% of Flamingo and 1.7% of Flamingo-9B.
30
20
10
250
2.5k
25k
250k
2.5M
Size Pre-Training Dataset
Figure 4: The influence the size of the pre-training dataset on performance.
of the mask generator, image encoder, and cap- tion generator our method, which reduces the dependency on additional pre-training data.
4.3 The Influence of Multi-Level Alignment
The influence of the size of the pre-training dataset is shown in Figure 4, and detailed data can be found in Appendix C.2. For all datasets, our method exhibits enhanced performance the size of the pre- mre-training dataset expands. This indicates that our method still has significant potential for improvement.
In this section, we conduct an ablation study to an- alyze the influence of multi-level alignment and explore the complementary and substitutive ef- fects among the three levels of alignment. All experiments were conducted with the identical pre- training configuration, as elucidated in appendix B. To minimize the influence of fine-tuning and con- serve computational resources, all ablation exper- iments will be conducted and reported under the zero-shot setting.
Dataset-Specifie Analysis. For the TGIF- FrameQA dataset, 51% of the questions focus on concrete object-related information such as color and quantity. while 49% of the questions address abstract information, such as actions. Our method performs well in capturing concrete object-related information in the zero-shot setting, resulting in strong performance in that scenario. However, in the fully-supervised setting, our method faces challenges in further improving its capability to extract concrete information, and it exhibits rela- tively low ability in capturing abstract information
The Complementary Effect. Table presents the results of the ablation studies on frame-level and object-level alignment, with the utilization video-level alignment. This experiment aims to un- cover the complementary effects of frame-level and object-level alignment on video-level alignment. For the MSRVTT-QA and MSVD-QA datasets, where the primary question types involve global de- scriptions of the videos, and for the iVQA dataset,

---

which focuses on scene and coarse-grained object information, frame-level alignment plays a dom- inant role as encompasses a significant major- ity of the spatiotemporal information relevant to the questions. In contrast, for the FGIIF-rammeQA dataset, which primarily focuses on object infor- mation, object-level alignment assumes the leading role as it helps capture fine-grained spatial informa- tion of the objects.
Limitations
In this work, we analyze the limitations follows: (1) Influence of maximum sequence length: Our method is influenced by the maximum sequence length of the language model. Although, as dis- cussed in Appendix E, there is a diminishing marginal utility of increasing the number of sam- pled frames, and expanding sequence lengths is a crucial direction in the development of large lan- guage models. However, reducing the temporal and spatial redundant information between differ- ent levels can still be beneficial.
Overall, the comparison between row 4 and the other rows in Table supports the complementary effect of object-level and frame-level alignment, and further confirms that utilizing three levels of alignment simultaneously is the optimal method.
(2) Abstraction ability: Our method provides less assistance in answering questions that focus on abstract information, while it offers greater assis- tance in answering questions that focus on concrete information. To enhance the abstraction ability of the model, we believe that deeper interactions be- tween different levels and modalities are beneficial.
The Alternative Effect. Table 4 presents the results of the ablation studies frame-level and object-level alignment, without the utilization of video-level alignment. This experiment aims to uncover the alternative effects of frame-level and object-level alignment on vidco-level alignment.
Overall, the comparison between row 3 and rows I to 2 in Table 4 provides evidence that the com- bined effect of frame-level and object-level align- ment is superior to their individual effects. Addi- tionally, the comparison between row 4 and rows I confirms the importance of video-level align- ment, suggesting that object-level and frame-level alignment have less substitutive effects on vidco- level alignment.
Ethics Statement
This work does not involve any direct ethical con- cerns. We are dedicated to advancing the field of video question answering. All experiments were conducted on open datasets, and the findings and conclusions of this paper are accurately and objec- tively reported.
Experimental Summary. Considering all the conclusions, it can be inferred that the three levels of alignment exhibit complementary effects. Addi- tionally, we investigate the influence of misalign- ment between the visual and textual modalities in Appendix D. For the influence of temporal infor- mation, please refer to Appendix E. Moreover, we conducted experiments to examine the influence of accumulated errors in the Appendix F.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot Binkkowsi,, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Si- monyan. 2022. Flamingo: a visual language model for few-shot learning. In Advances in Neural Infor- mation Processing Systems, volume 35, pages 23716- 23736. Curran Associates, Inc.
5 Conclusions
In this paper, we introduce Mulan: A Multi-Level Alignment Model for Video Question Answering, which enhances temporal and spatial information learning through multi-level aligmment. Addition- ally, we conduct experiments on public VideoQA datasets and achieve ofate-ofthhe-ar results. Fi- nally, we provide ablation studies to demonstrate the complementary effects of multi-level alignment and the effectiveness of our method.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question swering. In International Conference on Computer Vision (ICCV).
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zis- serman. 2021. Frozen in Time: A joint video and image encoder for end-to-end retrieval. In Proceed- ings of the IEEE/CVF International Conference on Compuder Vision (ICCV), pages 1728-1738.

---

Zhaowei Cai and Numo Vasconcelos. 2021. Cascade R-CNN: High quality object detection and instance segmentation. IEEF Transactions on Pattern Analy- sis and Machine Intelligence, 43(5):1483-144
of Machine Learning Research, pages 2790-2799. PMLR.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yunseok Jang, Yale Song. Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. TGIF-QA: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and Dision-languee representation learning with noisy text supervision. Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4904-4916. PMLR.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015. San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Constantin Fichenberg, Sidney Black, Samuel Wein- bach, Letitia Parcalabescu, and Anette Frank. 2022. MAGMA - multimodal augmentation of generative models through adapter-based finetuning. In Find- ings the Association for Computational Linguistics: EMNLP 2022, pages 1416-2428, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci. Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020. The Open Images Dataset V4. International Journal of Com- puter Vision, 128(7):1956-1981
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021. Clip2video: Mastering video-text retrieval via image clip. arXiv preprini arXiv:2106. 1097.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg. Mohit Bansal, and Jingjing Liu. 2021. Less Is More: Clipbert for video-and-langue language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1331-7341.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming Hc. 2019, Slowfast networks for video recognition. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV).
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
Jie Lei. Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized, compositional video ques- tion answering. In Proceedings of the 2018 Con- ference on Empiricul Methods in Natural Language Processing, pages 1369-1379, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Dechndig-eehhnned bert with disentangled attention. In International Conference on Learning Representations.
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C.H. Hoi. 2022a. Align and Prompt: Video-and-language pre-training with entity prompts. Proceedings of the IEFF/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 4953-4963.
Matthew Honnibal and Mark Johnson. 2015. An im- proved non-monotonic transition system for depen- dency parsing. In Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Processing, pages 1373-1378, Lisbon, Portugal. As- sociation for Computational Linguistics.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. BLIPP-2 Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint 27X
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. BLIP: Bootstrapping language-image pre- training for unified vision-language understanding

---

and generation. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1888--19900. PMLR.
Jinpeng Wang, Yixiao Ge, Guanyu Cai. Rui Yan. Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022b. Object-aware video- language pre-training for retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3313-3322.
Linjie Li, Yen-Chun Chen. Yu Cheng. Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. HERO: Hierarchical en- coder for Video+Language ommi-representation pre- training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2065, Online, Association for Computational Linguistics.
Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2023a. All in One: Exploring unified video-language pre-training, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6598-6608.
Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. 2023b. LAVENDER: Unifying video-language understand- ing as masked language modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23119-23129.
Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. 2022c. InternIm- age: Exploring large-scale vision foundation mod- els with deformable convolutions. arXiv preprint arXiv:2211.05773
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022c. CLiP-Event: Connecting text and images with event structures. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Puttern Recognition (CVPR), pages 16420-16429.
Xudong Wang. Rohit Girdhar, Stella X Yu, and Ishan Misra. 2023b. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3 3124-3134.
Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. 2022. Learning to recognize procedural activities with distant supervision. IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 13853-13863.
Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao. Hongjie Zhang, Jilan Xu. Yi Liu. Zun Wang, Sen Xing, Guo Chen, Junting Pan, Ji- ashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. 2022d. Intern Video: General vidco foundation mod- els via generative and discriminative learning. arXiv preprint ar arXiv:2212.03799.
Proceedings of the
Xudong Lin, Simran Tiwari, Shiyuan Huang. Manling Li, Mike Zheng Shou, Heng Ji, and Shih-Fu Chang. 2023. Towards fast adaptation of pretrained con- trastive models for multi- multi-channel video- addo-llnguaaee re- trieval. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14846-1455
Zhenhailong Wang. Manling Li, Ruochen Xu, Lu- owei Zhou, Jie Lei. Xudong Lin. Shuohang Wang. Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2022c. Language models with image descriptors are strong few-shot video-language learners. In Advances in Neural In- formation Processing Systems.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. CLIP4Clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293- 304.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined atten- tion over appearance and motion. In Proceedings of the 25th ACM International Conference on Multime- dia, MM '17, page 1645-1653, New York, NY, USA. Association for Computing Machinery.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. 2023. 2: A modularized multi- modal foundation model across text, image and video. arXiv preprint 202:23002.00422
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Li- juan Wang. 2022a. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang. Soham Ghosh, Yonghui Wu, and Jiahui Yu. 2022. Video-Texx Modeling with Zero-Shot Trans- fer from Contrastive Captioners. arXiv preprint 2rXiv:2212.04979.

---

Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. 2021. Just Ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 1686-1697.
for training/valicaiionnrssingg Similar MSVD- QA, the QA pairs in MSRVTT-QA are also auto- matically generated from video descriptions.
TGIF-FrameQA (Jang et al., 2017) is an open- end VideoQA benchmark based on GIF videos. In the TGIF-FrameQA dataset, the majority of videos have short durations, typically less than 5 seconds. These videos are primarily classified into four cat- egories: objects, quantities, colors, and positions. consists of 46k gifs and 53k QA pairs, split into 39k/13k for training/testing.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022a. Zero-shot video ques- tion answering via frozen bidirectional language mod- els. In Advances in Neural Information Processing Systems.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022b. An empirical study of GPT-3 for few-shot knowledge- based VQA. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3):3011-3089
B Implementation Details
Hy Hyperparameter Settings. For all experiments. we employ uniform sampling to select 8 frames (T 8) from each video. For pre-training ex- periments, we solely utilize video-level textual de- scriptions to avoid the model gathering information from other frame-level and object-level textual de- scriptions, which could potentially impact perfor- mance. For evaluation and fine-tuning experiments, we impose certain constraints to accommodate the token limitations of the language model. Specif- ically, we limit the generation of maximum of 8 objects per frame. Additionally, the maximum number of global frame object descriptions per frame is set and the maximum number of local frame object descriptions per object is set to 2. For pre- pre-training experiments, we use a sequence length of 128, and for evaluation and fine-tuning experi- ments, we use sequence length of 512. The visual feature dimension of the pre-trained image encoder ViT-L/14 (Dosovitskiy et al., 2021) is 768, while the hidden dimension of the pre-trained language model DeBERTa-V2-XL.n (He al., 2021) is 1536.
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu,
Qi Qian, Zhang, and Fei Huang. 2022. HiTeA: Hierarchical temporal-aware video-language pre- training, arXiv preprint arXiv:2212. 14546.
Chun-Hsiao Yeh, Bryan Russell, Josef Sivic, Fabian Caba Heilbron, and Simon Jenni. 2023. Meta-persomalizingg vision-language models to find named instances in video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 19123-19132.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. MERLOT: Multimodal neural script knowl- edge models. In Advances in Neural Information Processing Systems, volume 34, pages 23634-23651. Curran Associates, Inc.
A Datasets
Vid-22 (Bain et al., 2021) consists of 2.5 mil- lion videos and their corresponding text pairs, col- lected from the Shutterstock website. The dataset does not include audio, and the video captions are obtained from pre-existing alternative textual de- scriptions.
Training, We conducted all experiments on 8 NVIDIA GeForce RTX 3090 GPUs. The pre- training on the WebVid-2M (Bain et al., 2021) dataset took approximately 10 hours, with total of 2 epochs. For the pre-training experiments, we the learning rate to 3 x 10-3 and trained for 2 epochs with a batch size of 16. For the fine-tuning experiments, we set the learning rate to x 10-5 and trained for 10 epochs with a batch size of 4. For all training experiments, we employ the Adam opti- mizer (Kingma and Ba, 2015) with B = (0.9. 0.95) and no weight decay.
iVQA (Yang et al., 2021) is an open-end VideoQA benchmark that focuses on objects, scenes, and people in instructional videos. It con- sists of 10k video clips and 10k QA pairs, split into 6k/2k/2k for training/aliidaaionnettnngg
MSVD-QA (Xu et al., 2017) is an open-end VideoQA benchmark that consists of 2k video clips and 51k QA pairs, split into 32k/6k/133 3k for train- ing/validationntesting. The QA pairs in MSVD-QA are automatically generated from video descrip- tions.
MSRYTT-QA (Xu et al., 2017) is an open-end VidcoQA benchmark that consists of 10k video clips and 243k QA pairs, split into 158k/12k/73k
Static Vocabulary Setting. We constructed the static vocabulary based on the class names from the OpenImage v7 dataset (Kuznetsova et al., 2020),

---

Supervision
iVQA
MSRVTT-QA
MSVD-QA
TGIF-FrameQA
0% (zero-shol)
35.7
20.3
38.7
55.6
(few-shot)
41.3
37.6
48.7
63.7
10% (few-shot)
42,9
52.3
67.7
100% (fully-supervised)
47.4
58.2
72.0
Table 5: Zero-shot and few-shot Results.
Extra PT Data
iVQA
MSRVTT-QA
MSVD-QA
TGIF-PrameQA
250
7.2
5.5
18.7
26.7
2,5k
20.4
15.8
25k
28.1
16.2
31.0
250k
31.8
18.1
44.1
2.5M
35.7
20.3
38.7
Table 6: The influence of the pre-training dataset size. Zero-shot results are reported.
which consists of 21k noun phrases. For these noun phrases, we calculate the cosine similarity between cach pair, and if the cosine similarity between two words exceeds 0.95, remove the word with the smaller frequency. In addition, manually re- moved less informative words from the vocabulary, such as "video," "picture." "photo," and so on.
D The Influence of Misalignment
To examine the influence of modality misalign- ment, we conducted experiments that involved the separation of visual and textual modalities. The ex- perimental results are presented in Table 7. In this experiment, we conducted pre-training using all modalities and evaluated the performance by sepa- rately considering the visual and textual modalities in the zero-shot setting.
Results in Low-Data Regime
From the perspective of the individual effects of the visual and textual modalities, the textual modality plays more prominent role compared the visual modality. This is primarily because we leverage the language model for multimodal fusion. In terms of question types, the textual modality demonstrates significant assistance, especially for questions of the "number". "where", and "when" types. These question types often require a higher level of abstraction, and the textual modality pro- vides more abstract representation compared the visual modality. Conversely, the visual modal- ity proves to be particularly advantageous for "who" type questions, as this type of question requires the model to distinguish between visual entities present in the scene. The visual modality provides more detailed information about different visual entities, enabling the model to better address "who"" type questions. Overall, the experimental results across all datasets consistently demonstrate that the synergistic alignment between the visual and textual modalities can effectively improve the per- formance.
To examine the influence of the low-data regime on the performance of our method, we conducted experiments during both the pre-training and fine- tuning stages.
C.1 Few-Shot Setting
We employed few-shot learning as an experimental sctup to evaluate the performance our method in the low-data regime during the fine-tuning stage, The experimental results under the few-shot set- ting are presented in Table 5. The experimental results reveal substantial performance improve- ment when utilizing only 1% and 10% of the train- ing data compared to the zero-shot setting.
C.2 Size of the Pre- Fre-Training Dataset
We manipulated the size of the pre-training dataset and performed zero-shot testing to evaluate the per- formance of our method in the low-data regime during the fine-tuning stage. The detailed experi- mental results presented in Table 6. The exper- imental results confirm the substantial impact of the pre-training dataset size on the model's perfor- mance. The performance of our method improves as the size of the pre-training dataset increases.
E
The Influence of Temporal Information
The temporal nature of videos is the fundamental distinction between videos and images. this sec-

---

Modality
Benchmark
Acc
What
Number
Color
Where
Who
When
Visual
Textual
1.I
28.3
1.2
iVQA
1.3
/
35.7
/
11.6
20.9
12.9
9.4
35.4
2,2
MSRVTT-QA
X
19.4
13.1
65.1
11.9
12.6
31.8
2,3
V
20.3
13.1
69.6
12.6
11.0
34.6
5.2
3,1
34.8
25.7
29.5
50.0
46.4
51.7
1.7
3,2
MSVD-QA
X
37.1
30.2
62.6
37.5
50.0
47.6
10.3
3.3
V
38.7
30.4
68.8
56.3
53.6
51.5
6.9
4.1
V
X
31.9
42.3
4.5
41.4
22.4
4.2
TGGI-FrrameQ
X
53.6
46.0
64.2
64.1
23.8
4.3
55.6
46.6
69.8
65.5
23.8
Table The influence of misalignment between visual and textual modalities. "Ace" refers to the overall Top-1 accuracy of the dataset.
Sampling Frame Numbers
iVQA
MSRVTT-QA
MSVD-QA
TGIF-FrumeQA
26.2
16.0
34.2
48.4
2
31.3
19.4
37.4
51.4
33.6
19.8
37.6
53.4
8
35.7
20.3
38.7
55.6
Table 8: The infulence of the number of sampling frame numbers.
Benchmark
Temporal Prompt
Acc
What
Number
Color
Where
Who
When
1.1
X
34.8
iVQA
1.2
/
35.7
2.1
X
19.3
12,9
59.4
12.3
10.2
32.7
4.5
MSRVTT-QA
2.2
/
20.3
13.1
69.6
12.6
11.0
34.6
5.2
3.1
X
37.8
29.9
61.3
62.5
46.4
50.2
6.9
MSVD-QA
3.2
V
38.7
30.4
68.8
56.3
53.6
51.5
6.9
4.1
X
55.0
46.6
69.0
64.4
23.0
TGIF-FramcQA
4.2
55.6
46.6
69.8
65.5
23.8
Table 9: Ablation studies on the expandable ordinal prompt approach.
tion, we examine the influence temporal informa- tion in videos. Specifically, we will conduct exper- iments to investigate the influence of different sam- pling frame numbers and our proposed expandable ordinal prompt approach. When two frames sampled, method shows notable improvement in performance becomes capable of capturing the temporal information present in the video. As the number of sampled frames increases, the perfor- mance of our method continues to improve, albeit with diminishing returns.
number sampled frames increases, the perfor- mance our method continues to improve, albeit with diminishing returns. In conclusion, we have confirmed the importance of the temporal nature of videos.
E.2
The Influence of the Expandable Ordinal Prompt Approach
Table 9 presents the influence of the expandable ordinal prompt approach. The experimental results demonstrate that the expandable ordinal prompt approach enhances the performance of nearly all question types across all datasets when compared not utilizing the temporal prompt method. The experimental results confirm that our proposed ex- pandable ordinal prompt approach facilitates the language model's acquisition of temporal informa- tion in videos.
E.1 The Influence of Sampling Frame Numbers
Table 8 presents the influence of sampling frame numbers. When only single frame sampled, the video question answering system degrades to an image question answering system, resulting in the poorest performance due to the inability to cap- ture the temporal information the video. As the

---

Ratio of Random Noise
0%
25%
50%
75%
100%
Trame-Level Visual Features
38.7
36.0
33.8
30.2
Object-Lovve Visual Features
38.7
38.2
38.1
38.0
37.4
Frame-Level Textual Description
38.7
38.0
37.2
35.4
Object-Level Textual Description
38.7
38.6
38.1
38.0
37.6
Table 10: Top-1 accuracy at different levels of random noise ratio in the MSVD-QA dataset.
Ratio of Random Noise 0%
25%
50%
75%
100%
Frame-Level Visual Features
55.6
49.5
45.4
Object-Level Visual Features
55.6
54.7
51.2
48.4
Frame-Level Textual Description
55.6
55.5
54.5
52.8
48.5
Object-Level Textual Description
55.6
54.9
53.7
53.5
51.5
Table 11: Top-1 accuracy at different levels of random noise ratio in the VGGIF-FrameQA dataset.
F The Influence of Accumulated Errors
the cascading errors arising from object-level in- formation in comparison to the MSVD-QA dataset. This differentiation stems from the fact that the TGIF-FrameQA dataset is designed focus intricate details, whereas the MSVD-QA dataset emphasizes the holistic video information.
From structural standpoint, our method conforms to pipeline structure. At an equivalent scale, a pipeline structure, when compared to an end-to-end structure, offers increased flexibility. However, the pipeline structure method tends to result in higher incidence of cascading errors.
To examine the influence of cascading errors, We designed experiments in which we introduced sim- ulated errors by randomly substituting language information and adding random noise to visual information. Specifically, concerning the textual information of sample, we randomly substitute sentences or words with sentences or words from another sample. Regarding the visual information sample, we randomly added Gaussian noise to the pixel images of frames objects.
Table presents the results on the MSVD-QA dataset, which emphasizes the overall vidco infor- mation. Table presents the results on the TGIF- FramoQA dataset, which focuses on fine-grained details. Overall, the cascading errors generated by frame-level features surpass those of object-level features due the dependence of object-level fea- tures on frame-level features. Similarly, the cas- cading errors urising from the visual modality out- pace those from the textual modality, primarily attributed to the demanding visual capabilities re- quired in video question answering tasks and the dependence of text description generation on visual features. Furthermore, the ratio of random noise increases, the performance undergoes acceler- ated decline rather than linear descent. When the random noise ratio reaches 75%. our method still manages to maintain relatively comparable performance. From a dataset perspective, the TGIF- FrameQA dataset exhibits heightened sensitivity to


Reviews:
[{"review_id": "mj4oDGR1xv", "reviewer": "Reviewer_yuTo", "summary": "", "questions": "The method is relatively complex, and I have the following questions:\n1)The model is relatively complex, and under the same settings, is it comparable in terms of running time with previous methods？\n2) I think this is a pipeline structure, whether the accumulation of errors can be evaluated, and which part of it most affects the results\n3) Will the method in 3.3 lead to excessive dependence of object level features on frame level features?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our appreciation to the reviewer for the valuable experimental suggestions as well as the recognition of the novelty of our framework.\n\n**Q1: The innovation of the article is mainly reflected in ‘first work in the field of video question answering that explores multi-level visual-language alignment’, but the specific method innovation is average, or it is the previous work.**\n\nAs highlighted by the reviewer, one of our major contributions is the novelty of the framework. Another important novelty of our method primarily lies in the proposed mask-guided visual feature encoding method for object-level feature extraction, as opposed to the direct use of object detectors. The motivation behind this design is to enable both frame-level and object-level features to be processed using a singular image encoder, thereby coexisting within the same semantic space. Moreover, this design encompasses an open vocabulary capability that is free from reliance on predefined object detection labels. It also enhances precision in object recognition, particularly for irregular objects.\n\n**Q2: The article is well organized overall, but there are unclear parts, such as the third paragraph of the introduction.**\n\nWe have already revised the third paragraph of the introduction, which will be reflected in the revised version. Specifically, we relocated certain minor detailed descriptions from the third paragraph of the introduction to Section 3.1 and provided additional explanations. Furthermore, the three authors of this paper have collectively reviewed the entire paper and made corrections, encompassing sections beyond the third paragraph of the introduction.\n\n**Q3: The model is relatively complex, and under the same settings, is it comparable in terms of running time with previous methods?**\n\nIn our experimental environment and settings, the average inference time for a single sample in the MSRVTT-QA dataset is 780 ms. For comparison, the average inference time for FrozenBiLM, which has fewer parameters, is 417 ms. For larger-scale models, it is typically observed that longer inference times are incurred.\n\n**Q4: Our method is a pipeline structure, whether the accumulation of errors can be evaluated, and which part of it most affects the results.**\n\nWe concur that our method adheres to a pipeline structure. At an equivalent scale, a pipeline structure, in comparison to an end-to-end structure, tends to incur more cascading errors. However, the pipeline structure approach affords greater flexibility. \n\nSimultaneously, we express our gratitude for the reviewer's supplementary suggestions regarding auxiliary experiments. We designed experiments in which we introduced simulated errors by randomly adding noise to visual information and substituting language information. This allowed us to conduct a cascade error analysis. The experimental outcomes elucidate that the cascading errors engendered by frame-level features exceed those originating from object-level features. Similarly, the cascading errors arising from the visual modality outpace those from the textual modality. Among them, the most pronounced cascading errors are induced by the frame-level visual features. We will integrate the aforementioned experiments and furnish detailed experimental data, along with more comprehensive experimental setups and analyses, in the appendix of the revised version.\n\n**Q5: Will the method in 3.3 lead to excessive dependence of object level features on frame level features?**\n\nSection 3.3 delineates the process of text description generation. The dependency process in text description generation can be described as follows: object-level text descriptions depending on both the dynamic and static vocabularies, with the dynamic vocabulary depending on frame-level text descriptions. Hence, we posit that object-level text descriptions are dependent on frame-level text descriptions. However, due to the presence of a predefined static vocabulary, the model retains a certain degree of text generation capability even when the quality of the dynamic vocabulary is subpar. Consequently, this dependency is not excessive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper propose a method Mulan: a Multi-Level Alignment Model for Video Question Answering, which establishes alignment between visual and textual modalities at the object-level, frame-level, and video-level.  The experiment is sufficient and experimental results illustrate that this method surpasses state-of-the-art baselines. \nThe framework proposed in this paper is novel, which Incorporates rich features to extract information in video.", "reasons_to_reject": "1.The article is well organized overall, but there are unclear parts, such as the third paragraph of the introduction.\n2.The innovation of the article is mainly reflected in ‘first work in the field of video question answering that explores multi-level visual-language alignment’, but the specific method innovation is average, or it is the previous work.\n", "questions_to_authors": "The method is relatively complex, and I have the following questions:\n1)The model is relatively complex, and under the same settings, is it comparable in terms of running time with previous methods？\n2) I think this is a pipeline structure, whether the accumulation of errors can be evaluated, and which part of it most affects the results\n3) Will the method in 3.3 lead to excessive dependence of object level features on frame level features?", "paper_topic_and_main_contributions": "This paper is about Video Question Answering. The author introduces a multi-level visual-language alignment method for video question answering, which provides a novel framework to learn spatial information and contains object-level, frame-level and video-level character.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RJ2zkcIign", "reviewer": "Reviewer_YRUS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to the reviewer for acknowledging the merit of our proposed idea and recognizing the promising experimental results.\n\n**Q1: The loss function.**\n\nAs mentioned in lines 383 to 389, we employ a masked language modeling objective as our loss function, following the setup established in BERT. The mathematical formulation of the masked language modeling will be provided in the revised version.\n\n**Q2: The update process of the visual-to-text projection module $P$.**\n\nAs mentioned in lines 369 to 375, the projection matrices $P$ ($P^G$ and $P^L$) correspond to a linear layer, and lines 373 to 375 describe their update process.\n\n**Q3: The update process of the adapter module.**\n\nAs mentioned in lines 364 to 365, we use the same Adapter layer as in the FrozenBiLM. In specific terms, for the hidden state $z$, the update process of $Adapter(z)$ is as follows:\n\n$Adapter(z) = z + W^{up}\\psi(W^{down}z)$\n\nIn this process, $W^{down} \\in \\mathbb{R}^{D\\times D_h}$, $W^{up} \\in \\mathbb{R}^{D_h\\times D}$, $D$ is the hidden dimension of the transformer, $D_h$ is the bottleneck dimension, and $\\psi$ is a ReLU activation function.\n$D_h$ is typically set to be smaller than $D$ such that the adapters are lightweight.\n\n**Q4: In the zero-shot setting, how to conduct the video-level alignment?**\n\nAs mentioned in Figure 1, during the training phase, for a video-text sample, video-level alignment refers to aligning all sampled frames with the provided original textual description within the sample. However, during the evaluation phase, where the model receives a question and a video segment as input, video-level alignment is not applicable. A comprehensive explanation of video-level alignment will be incorporated in the revised version.\n\n**Q5: If multiple videos are relevant to the same text description or multiple text descriptions are relevant to the same video, how to align videos and text descriptions?**\n\nIn video-language tasks, a standard sample generally involves a single video paired with a corresponding textual description. Hence, our model's input consists of an individual video alongside its associated text. If there are multiple text descriptions relevant to the same video, each description is treated as an independent sample. This principle is similarly applied in cases where multiple videos are related to the same text description. This processing approach is widely employed in the field of video question answering.\n\nConsidering all samples within the dataset, we concur with the scenarios highlighted by the reviewers. Instances where multiple text descriptions are relevant to the same video augment the diversity of video-text descriptions. This increase in textual descriptions assists in alleviating the challenge of information imbalance between modalities, thus contributing to enhanced performance. In cases where multiple videos are related to the same text description, Due to video content varies, textual descriptions at the frame and object levels also differ. This disparity enables our method to effectively differentiate between multiple videos sharing the same textual description.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is interesting.\nResults shown in this paper are promising.", "reasons_to_reject": "My concerns are as follows:\n1) why there is no loss function in this paper? \n2) in page 12, how to use the learning rate, epoch, batch size, Adam optimizer without loss function?\n3) in line 382, authors state that they update the parameters of the visual-to-text projection modules P and the adapter module. Could you provide more details abou how to update the parameters?\n4) in the zero-shot setting, how to conduct the video-level alignment? If multiple videos are relevant to the same text description or multiple text descriptions are relevant to the same video, how to align videos and text descriptions?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the Video Question Answering task. Different from previous methods that target to improve joint representations of video and text, this paper establishs alignment between visual and textual modalities at multiple visual levels.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iDf1mvBYrG", "reviewer": "Reviewer_1gR7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of the ideas, motivation, and experimental outcomes presented in our paper.\n\n**Q1: The authors only train the adapter's parameters, leaving the other model parameters untrained. It might be worth exploring why this decision was made and whether training all parameters could improve performance.**\n\nUtilizing only the adapter layer instead of full fine-tuning represents a parameter-efficient fine-tuning (PEFT) approach. The main purpose of Parameter-efficient Fine-tuning (PEFT) is to achieve a fine-tuning effect similar to that of tuning all parameters by adjusting only a small subset of parameters, aiming to reduce computational and storage costs. Experiments in [1] indicate that adjusting the adapter layer yields performance comparable to full fine-tuning. Moreover, parameter-efficient fine-tuning methods exemplified by Adapters are now widely employed.\n\n[1] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe,\nAndrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning\nfor nlp. In ICML, 2019.\n\n**Q2: Although the authors claim that their model can achieve better performance with less training data, it is worth noting that they only train the adapter's parameters. In contrast, more complex layers are trained in baselines like Flamingo and mPLUG-2.**\n\nWe concur with the reviewers' observations that, compared to other baselines, we exclusively fine-tuned the parameters of the adapter layer. The data concerning training parameters provides evidence for the potential of employing parameter-efficient fine-tuning techniques for adapting language models to multimodal fusion. Our method shows promise, as it can achieve state-of-the-art results even with a smaller dataset when compared to other baselines.\n\n**Q3: Despite using strong pre-trained models like DeBERTa, our model does not significantly outperform strong baselines such as mPLUG-2.**\n\nAll the pre-trained models we employed lack the ability to address video-language tasks in isolation; for instance, DeBERTa operates solely as a monomodal language model. In the zero-shot setting, our method achieves an average accuracy that is 7.8% higher across all datasets compared to FrozenBiLM, despite both methods incorporating DeBERTa. This comparison with FrozenBiLM shows that in video question answering tasks, the capacity to understand visual-language interactions holds greater importance than linguistic comprehension alone.\n\nIn our model, we have merely incorporated an additional 2.5 million video-language samples for video-language alignment. Compared to other strong baselines, we employed a minimal amount of video-language datasets. Nevertheless, our model attains state-of-the-art performance. Considering the dataset size, this outcome is profoundly encouraging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I find the idea of utilizing different levels of video information to improve video question answering performance quite intriguing. This approach could enhance the model's understanding of spatial and temporal relations between objects and frames, as evidenced by promising experimental results. ", "reasons_to_reject": "I have some reservations about the suitability of this work for an EMNLP paper due to the following reasons:\n\n(1) The authors only train the adapter's parameters, leaving the other model parameters untrained. It might be worth exploring why this decision was made and whether training all parameters could improve performance.\n(2) Despite using strong pre-trained models like DeBERTa, the authors' model does not significantly outperform strong baselines such as mPLUG-2.\n(3) Although the authors claim that their model can achieve better performance with less training data, it is worth noting that they only train the adapter's parameters. In contrast, more complex layers are trained in baselines like Flamingo and mPLUG-2. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method to establish alignment between visual and textual modalities at different levels - object, frame, and video. At the object-level, the proposed method uses a mask-guided visual feature encoding method and a visual-guided text description method to learn fine-grained spatial information. At the frame-level, it combines visual features from individual frames with a caption generator to learn overall spatial information within the scene. At the video-level, an expandable ordinal prompt for textual descriptions is proposed, combined with visual features, to learn temporal information. Experimental results show that this method outperforms state-of-the-art methods, even with minimal visual language pre-training data and fewer trainable parameters. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "68DWhKJGAM", "reviewer": "Reviewer_eqr8", "summary": "", "questions": "- How is the static vocabulary V^s predefined? What is a set of candidate objects?\n- Why should we distinguish static and dynamic vocabulary? What’s the functional difference between the two other than their naming?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of our experimental design and the positive experimental results.\n\n**Q1: The distinctions between our method and the method presented in [1].**\n\nReviewers YRUS and 1gR7 found our method to be interesting. Particularly, reviewer yuTo explicitly acknowledged the novelty of our framework. The method presented in [1] is a multi-stream method. Despite both our proposed multi-level alignment method and this method utilizing multi-granularity features, there are significant differences in both motivation and specific methodologies. These distinctions are elaborated as follows:\n\n1. Motivation: The method presented in [1] involves an initial encoding of various streams (such as regional visual features, ImageNet features, etc.) using LSTM, followed by the fusion of these streams. Consequently, the authors have described their methodology as a \"multi-stream\" method. Although features of varying granularity are extracted, features of the same granularity are merged into distinct streams before alignment. Due to the absence of alignment concept in multi-stream methods, the multi-stream approach significantly differs from the multi-level alignment concept in our method.\n\n2. Specific Methodologies: The method presented in [1] employs an object detector to acquire region-based visual features and text information, whereas we propose a mask-guided visual feature encoding method to extract object-level visual features and text information. The motivation behind this design is to enable both frame-level and object-level features to be processed using a singular image encoder, thereby coexisting within the same semantic space. Moreover, this design encompasses an open vocabulary capability that is free from reliance on predefined object detection labels. It also enhances precision in object recognition, particularly for irregular objects.\n\nIn the revised version, we will incorporate a comparison with multi-stream methods in the related work section.\n\n[1] TVQA: Localized, Compositional Video Question Answering. Lei et al, 2018.\n\n**Q2: Performance comparison with mPLUG-2 and the HiTeA variant in the zero-shot setting.**\n\nBoth mPLUG-2 and HiTeA are indeed impressive works. Our perspective on the use of extra VQA dataset is as follows:\n\n1. The intrinsic correlation between visual question answering tasks and video question answering highlights the significant benefit of incorporating extra VQA dataset. For instance, when comparing the HiTeA variant (with the VQA dataset) to HiTeA (without the VQA dataset), we observe substantial absolute performance improvements of 13.1% (from 8.6% to 21.7%) and 19.2% (from 18.2% to 37.4%) on the MSRVTT-QA and MSVD-QA datasets, respectively.\n\n2. Following the reviewers' criteria, it can be stated that we make use of the MSCOCO and similar datasets through the integration of the Cascade Mask R-CNN and BLIP. However, it's worth noting that mPLUG-2 and HiTeA have also leveraged these datasets comprehensively.\n\nHowever, we concur that mPLUG-2 and the HiTeA variant represent impressive zero-shot video question answering methods. But a more comprehensive comparative analysis is essential. Hence, we have incorporated the results of these two methods into Table 1 of the revised version and provided elucidations in section 4.2. Moreover, we have supplemented a detailed analysis of dataset utilization in the appendix.\n\n**Q3: How is the static vocabulary V^s predefined? What is a set of candidate objects?**\n\nAs mentioned in lines 908 to 917, we constructed the static vocabulary based on the class names from the OpenImage v7 dataset, which consists of 21k noun phrases. For these noun phrases, we calculate the cosine similarity between each pair, and if the cosine similarity between two phrases exceeds 0.95, we remove the phrase with the smaller frequency. In addition, we manually removed less informative phrases from the vocabulary, such as \"video\", \"picture\", \"photo\", and so on. \"Candidate objects\" refers to noun phrases within the static vocabulary.\n\n**Q4: Why should we distinguish static and dynamic vocabulary? What’s the functional difference between the two other than their naming?**\n\nAs mentioned in lines 294 to 296, the predefined static vocabulary cannot cover all objects and attributes. In other words, employing a static vocabulary constitutes a closed vocabulary configuration, rendering it challenging to generate textual descriptions for uncommon objects. As a remedy, we extract noun phrases from global frame captions to form a dynamic vocabulary. Overall, in terms of functionality, the dynamic vocabulary serves to expand the generation of object descriptions into an open vocabulary, while the static vocabulary functions as a foundational resource when the quality of dynamic lexicon generation is compromised.\n\n**Q5:In line 226, $v^G$ shoule be $\\phi_\\text{CLIP}$?**\n\nWe express our gratitude for the reviewer's examination. The phrasing at line 226 lacked clarity. We have rectified this in the revised version and conducted similar checks throughout the entire paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed approach shows promising performance while using fewer parameters.\n- Extensive experiments demonstrate the effectiveness of the proposed approach.", "reasons_to_reject": "- Multi-level alignment doesn’t seem to be quite novel and previous works also have tried to use multi-level visual features (for example, [1]), not exactly same approach as the proposed one though. Therefore, a more extensive literature search would be needed to compare this work to previous studies. Employing similar concepts would be fine but the paper should show the difference to the prior works.\n\n- There seems to be missing comparisons in Table 1. Although mPLUG-2 and the HiTeA variant with the best performance appear to perform better in the zero-shot setting, they are not included in the table. If its exclusion is due to its use of the VQA dataset, then a similar argument could be made about the proposed approach, as it also uses the MSCOCO, Visual Genome, Conceptual 12M, etc. dataset through its employment of the Cascade Mask R-CNN and BLIP. Hence, more thoughtfully designed experiments could have ensured a fairer comparison\n\n\n\n\n\n\n[1] TVQA: Localized, Compositional Video Question Answering. Lei et al, 2018.", "questions_to_authors": "- How is the static vocabulary V^s predefined? What is a set of candidate objects?\n- Why should we distinguish static and dynamic vocabulary? What’s the functional difference between the two other than their naming?", "paper_topic_and_main_contributions": "This paper proposes multi-level alignment approach for VideoQA. It employs object mask generator and image captioning models to extract different levels of features. Extensive experiments are conducted to show the effectiveness of the proposed method.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line226: $v^{G}$ should be $\\phi_{CLIP}$ ?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript introduces a new approach to video QA by aligning visual and textual at different levels to accomplish the task. Reviewers and authors engaged in a productive discussion around strength and weaknesses of the approach and some details of the claims. The idea of aligning textual and visual information is novel and most important aspect of the manuscript. Some of the baselines highlighted by the reviewers were not in the original submission and but authors have produced comparison and used the arguments in responses to the reviewers. While this helped with the discussion, a more comprehensive literature survey would have been better in the submission. One of the main advantages of the proposed approach is the claim of using small pre-training data compared with the state of the art to achieve the same performance. However, this claim needs to be adjusted to refer only to the extra pre-training data. Technically, encoders are pre-trained with additional data before being used.
In summary authors have identified a better approach to leverage the information in a video to answer question. However while we see some improvements over state of the art, this is not consistent potentially highlighting the need for further experimentation to improve the approach.
Decision: Accept-Findings