Large Language Models are Temporal and Causal Reasoners for Video Question Answering
Authors: Dohwan Ko
Ji Soo Lee
Woo-Young Kang
Byungseok Roh
Hyunwoo J. Kim
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
temporal and causal reasoning
TL;DR: We investigate that pretrained LLMs' knowledge is a strong prior for temporal and causal reasoning on challenging VideoQA and propose a novel framework, Flipped-VQA, to efficiently fine-tune LLMs on VideoQA by leveraging such LLMs' prior knowledge.

Abstract: Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks.
We observe that the LLMs provide effective priors in exploiting $\textit{linguistic shortcuts}$ for temporal and causal reasoning in Video Question Answering (VideoQA).
However, such priors often cause suboptimal results on VideoQA by leading the model to over-rely on questions, $\textit{i.e.}$, $\textit{linguistic bias}$, while ignoring visual content.
This is also known as 'ungrounded guesses' or 'hallucinations'.
To address this problem while leveraging LLMs' prior on VideoQA, we propose a novel framework, Flipped-VQA, encouraging the model to predict all the combinations of $\langle$V, Q, A$\rangle$ triplet by flipping the source pair and the target label to understand their complex relationships, $\textit{i.e.}$, predict A, Q, and V given a VQ, VA, and QA pairs, respectively.
In this paper, we develop LLaMA-VQA by applying Flipped-VQA to LLaMA, and it outperforms both LLMs-based and non-LLMs-based models on five challenging VideoQA benchmarks.
Furthermore, our Flipped-VQA is a general framework that is applicable to various LLMs (OPT and GPT-J) and consistently improves their performances.
We empirically demonstrate that Flipped-VQA not only enhances the exploitation of linguistic shortcuts but also mitigates the linguistic bias, which causes incorrect answers over-relying on the question.
Code is available at https://github.com/mlvlab/Flipped-VQA.

Large Language Models are Temporal and Causal Reasoners for Video Question Answering
Dohwan Ko Ji Soo Lee' Wooyoung Kang²
Byungseok Roh² Hyunwoo J. Kim
'Department of Computer Science and Engineering, Korea University Kakao Brain
(ikodoh, simplewhites, te9, (edwin.kang, peter... p www.)kkaaabbbbiinccoom
Abstract
Large Language Models (LLMS)) have shown remarkable performances on wide range of natural language understanding and genera- tion tasks. We observe that the LLMs pro- vide effective priors in exploiting linguistic shortcuts for temporal and causal reasoning in Video Question Answering (VideoQA). How- ever, such priors often cause suboptimal results on VideoQA by leading the model to over- rely on questions, i.e., linguistic bias, while ignoring visual content. This is also known as "ungrounded guesses' "hallucinations'. To address this problem while leveraging I.L.s* prior on VideoQA, we propose novel frame- work, Flipped- VQA, encouraging the model to predict all the combinations of (V, Q. A) triplet by flipping the source pair and the target label to understand their complex relationships, i.e. predict A, Q. and given a VQ. VA, and QA pairs, respectively. In this paper, we develop LLaMA-VQA by applying Flipped- VQA to LLaMA, and outperforms both L.Ms- and non-LLMssbaase models five challeng- ing VidcoQA benchmarks. Furthermore, our Flipped-VO VQA general framework that is applicable to various I.LMs (OPT and GPT-J) and consistently improves their performances. We empirically demonstrate that Flipped- VQA not only enhances the exploitation of linguistic shortcuts but also mitigates the linguistic bias, which causes incorrect answers over-relying on the question. Code is available at https: //gi thub. http://llaab//ippedddeee
Answer
L.MG
Accuracy
Quesiia::
Why the wlate hold fightyy the boy in wahe?
Chokow:
①
look buy keeps moving
VGA 125M
① poenny for comara
03.
① prevent falling
Caural
Temppeal
Descr ptte
Theal
boy
Question type
(a)
(h)
Figure LLMs' temporal and causal reasoning abil- ity. (a) An example of causal question that LLMs correctly answer without visual content. (b) Compari- son of LLaMA 33B (QA) and OPT 125M (VQA).
achieved human-level performance wide range of challenging tasks like professional & academic QA (Hendrycks al., 2021), I), science QA (Clark et al., 2018) 8), mathematics QA (Cobbe et al., 2021), code generation (Chen et al., 2021), and common- sense reasoning (Zellers et al., 2019; Sakaguchi et al., 2021) since they are pretrained with large- scale corpora (e.g., CommonCrawl. Bookcorpus, and Wikipedia) which entail massive human knowl- edge. With such pretraining data, usually compris- ing series of contexts, LLMs are trained to predict the next token given the preceding tokens. There- fore, LLMs learn to predict the next context given a series of contexts during pretraining, so they im- plicitly learn temporal and causal reasoning ability.
1 Introduction
To assess LLMs' temporal and causal reason- ing ability, we explore a popular multi-modal understanding task, Video Question Answering (VideoQA), which requires the model to predict the correct answer (A) given a video (V) and question (Q) pair. Recent challenging VideoQA benchmarks demand the model answer the question which asks temporal and causal relationships, e,g., the next event of video or the reason why scene happens. We observe that LLMs effectively handle
Large Language Models (LLMs) have exhibited an impressive ability free-form generation tasks and multi-choice question-answe answering tasks in natu- ral language processing (Chung et al., 2023; Jiang et al., 2023; Fei et al., 2023; Cai et al., 2022: Chen et al., 2023; Saha et al., 2022). These LLMs have
"Equal contribution. Cooresponniig author.

---

such challenging VideoQA benchmarks by lever- aging their strong prior knowledge of temporal and causal reasoning learned from the pretraining phase. For example, in Fig. la, LLMs correctly answer causal questions solely based on the text question and options without referring to the visual content by exploiting linguistic shoricut. Also, Fig. lb shows that language-only QA model equipped with larger language model, e-g., LLaMA 33B, denoted by QA 33B outperforms a VideoQA model with OPT 125M trained with full (V, Q A) on causal and temporal questions by a large margin of 13%. Although LLMs' prior knowledge is effective for addressing complex temporal and causal ques- tions, this sometimes leads to suboptimal answers when the model overly depends on inaccurate lin- guistic prior, i.e., linguistic bias, while ignoring the visual content. This is known as the 'hallucina- tion problem' of visual question-ansee answering models equipped with LLMs (Alayrac et al., 2022). Al- though in the literature linguistic shortcut and lin- guistic bias are interchangeably used, in this paper, we use the former specifically when the linguistic prior is correct and the latter otherwise.
incorrect answer over-relying on the question. To sum up, our contributions are as follows:
We investigate that pretrained LLMs' knowl- edge is strong prior for temporal and causal reasoning on challenging VideoQA
We propose novel framework, Flipped- Flipped-VQA, efficiently fine-tune LLMs on VideoQA by reasoning and understanding the complex re- lationships of (V, Q, A) triplet, using LLMS'' prior knowledge of temporal and causal rea- soning. Flipped-VOU VQA requires LLMs to per- form three tasks: VQ + A, VA + Q. and QA - V, and we combine these objectives LLMs' language generation objective.
LLaMA-VQA trained by Flipped- VQA, out- performs the baselines on five challenging VideoQA benchmark datasets. Also, our ex- periments demonstrate that Flipped-VQA is generally applicable various decoder-only LLMs and consistently improves their perfor- mances.
Our extensive analyses show that Flipped- VQA is effective in exploiting linguistic short- cuts to answer the question based on LLMs' prior knowledge and alleviating the linguis- tic bias by increasing the utilization of visual contents.
Here, we propose a novel learning framework, Flipped-VQA, predicting all the combinations of (V, Q, A) triplet by flipping the source pair and the target label, i.e., VQ A (main task), VA Q. and QA - V (auxiliary tasks). In other words, to understand complex relationships be- tween the video, question, and answer, LLMs are asked to additionally predict the question given a video-answer pair and the vidco given a question- answer pair by leveraging their knowledge of tem- poral and causal reasoning. In our experiments, we develop LLAMA-VQA by applying VQA to LLaMA (Touvron et al., 2023) and it outper- forms other baselines on five challenging VideoQA benchmark datasets: NExT-QA (Xiao al., 2021), STAR (Wu et al., 2021), DramaQA (Choi al., 2021), VLEP (Lei et al., 2020), and TVQA (Lei et al., 2018) with a small number of learnable parameters (only 0.06% of total model parame- ters). Furthermore, Flipped- VQA improves the performance of GPT-J (Wang and Komatsuzaki, 2021) and OPT (Zhang et al., 2022), implying that our framework is generally applicable other decoder-only LLMs. We empirically demonstrate that Flipped- VQA encourages LLMs exploit lin- guistic shortcuts by leveraging their prior knowl- edge and also mitigates linguistic bias which causes
2 Related works
LLMs for temporal and causal reasoning. Ex- posed to a wide range of corpora during pretrain- ing, LLMs perform diverse reasoning tasks (Liu et al., 2023; Wang et al., 2023e; Ozturkler et al.. 2023; Ho et al., 2022; Li et al., 2022; Wang et al., 2023c; Kıcıman et al., 2023). Particularly, a line of works (Zhang al., 2023a; Li et al., 2023e; Ka- malloo et al., 2023; Wang et al., 2023b; Tan et al., 2023) focuses on the temporal and causal reason- ing skills of LLMs. For temporal reasoning. Zhang and Choi (2021) assesses LLMS by asking open- domain time-sensitive questions in both closed and open settings. Similarly, Hobbhahn et al. (2022) investigates the ability through querying subject and relation pairs of different time periods. Further- more, some works have also explored the causal capabilities by evaluating whether LLMs can under- stand the causal implications given in the sentence. Long et al. (2023) uses simple causal graphs and de- termines if LLMs can understand the relationship

---

Lvga
vvgg
Question: What does the boy blue do after watching girl
middle?
towards
Ser
to
help
carry
[FOS]]
A
Large Language Models (LLMs)
N
Large Langunge Modelt (LIMM)
Laye
Token Embedder
Token Embedder
Vianal Enoosler
Video
KING
ONN.
Eqav
Lincer Layer
A
Visual
*
Tokco Limbedder
*
Tokm Umbodder
Question:
Large Language Models (LLMs)
What does the boy in blue after watching the girl in rod while the middle?
walk towards her to help carry her [EOS]
Frozen
Linear Lager
Trained
Backpropagated
Question
Answer
Video
Figure 2: Illustration of LLMs with Flipped- VQA. Flipped-VOL VQA consists of three objectives: L Lrags and Lape- Cvop is common objective, which predicts the answer given a video-question pair, for VideoQA, Likewise, Lvaq and Lyar are objectives question and video prediction by leveraging LLMs knowledge. other words, for each objective, VQ. VA. and QA pair used as prefix tokens predict A. Q, and V, respectively, Trainable parameters interleaved in LLMs stand for adapter tokens as in LLaMA- Adapter. Our framework employs only relatively small number trainable parameters LLMs, e.g., 4.5M trainable parameters among the total parameters of LLaMA 7B (0.06%).
between nodes. this work, we further examine the temporal and causal reasoning skills of LLMs expanded to the multi-modal setting of challenging VideoQA.
tionships present in the video.
Video Question Answering (VideoQA). VideoQA aims to answer natural language questions given video that requires multi-modal understanding and reasoning skills different semantic levels. Previous VideoQA benchmarks (Xu et al., 2017; Jang al., 2017) target short videos and ask ques- tions based on visual facts such location and objects/attributes. In contrast, more recent bench- marks (Xiao et al., 2021; Lei et al., 2020, 2018; Wu et al., 2021; Choi et al., 2021) tend to tackle temporal and causal questions referencing a longer video. Specifically, NEXT-QA (Xiao et al., 2021) requires uncovering the cause/intention of cer- tain event (e.g., Why did ...) reasoning about subsequent actions the video (e.g., What/How do/react after ...). In this work, we address these challenging VideoQA benchmarks through LLMs' temporal and causal reasoning abilitics.
LLMs for multi-modal understanding. Vari- lines of work attempt incorporate different modalities into LLMs to leverage the models' gen- cration power and knowledge in performing multi- modal downstream tasks. There exist various tech- niques (Hu et al., 2021; Jia et al., 2022; Ju et al., 2022) in the literature to bridge the gap between dif- ferent modalities. For instance, Flamingo (Alayrac al., 2022) ingests visual content into the frozen Chinchilla (Hoffmann et al., 2022) through Per- ceiver Resampler. LLaMA- -.LMM--Apppter (Zhang al., 2023c) fine-tunes LLaMA by applying linear pro- jection along with the adaption of prompts to in- corporate the visual information. Recently, there have been several approaches to develop a LLMs- based video chat model trained on massive multi- modal instruction tuning dataset that enables com- prehensive understanding across different modali- ties (Zhang al., 2023b; Li al., 2023b.d). Specif- ically, VideoChat (Zhang al., 2023b) proposes video-centric instruction dataset that primarily em- phasizes spatio-temport reasoning and causal rela-
Method
We present Flipped-VQA. simple yet effec- tive framework for Video Question Answering (VideoQA) that leverages the LLMs' prior knowl-
Further details with examples of NEXT-QA are provided
in Sec. D.

---

edge of temporal and causal reasoning. In addi- tion the target task of predicting an answer A given video and a question Q (i... VQ + A), our framework flips the role of inputs and outputs, requiring the model to predict given QA and Q given VA. We apply our framework to LLaMA (Touvron et al., 2023) and develop LLaMA-VQA but it is applicable to any decoder- only LLMs. In this section, we first describe the overall architecture of LLaMA-VOL VQQA and then in- troduce our objective. The overall architecture is illustrated Fig. 2.
[SOS] Video: <i>> <1>> <UN,>
Question: <question>
Choices:
(A) <option 1>
(B) <option 2>
(C) <option 3>
(D) <option 4>
(B) <option 5>
Answer: The answer is <answer> [EOS]
Table I: Input Prompt of I.I. LLAMAAVQA.
3.1 LLaMA-VQA
LLaMA, and the output feature is calculated as:
LlaMA-VQA is built on LLaMA with a few addi- tional learnable parameters. First, LLaMA-VQA adopts learnable linear layer to project the vi- sual embeddings, extracted from the frozen visual encoder CLIP ViT/L14 (Radford et al., 2021), to LLaMA's text token embedding space, see Fig. Specifically, given a raw video Tv, a sequence of visual tokens is calculated as v = [#,,.... UN.] = f ((CLIP(Z.)) E IR ND. where N₂ is the number of video frames and D is feature dimension. Sec- ond, as in LLaMA-Adapter (Zhang et al., 2023c), we additionally adopt several trainable adapter PN,]] which are prepended the key and value of each self-attention layer, where Np is the number of adapter tokens. Fur- ther descriptions of LLaMA- -ddattt are provided in Sec. C. So the number of trainable parameters of LLaMA-VQA 7B is 4.5M, only 0.06% of to- tal parameters of LLaMA 7B. With such few trainable parameters, LLaMA-VQA effectively pre- serves LLMs' prior knowledge and leverages it in exploiting linguistic shortcuts for VideoQA.
[h" h h2 h*] LLaMA((v,
where h" is sequence of output features, i.e., h" [A]. Mi....,,KK 1, and h9, h are similarly de- fined.
3.2 Flipped-
To utilize LLMs' temporal and causal reasoning abilities, we here present Flipped- -VQ, consisting of three objectives, for reasoning the complex re- lationship between video, question, and answer of VideoQA.
tokens p
VQ A. Predicting answer given video- question pair is the primary task of VideoQA. Its objective function formulated as:
Luga - log P(al v. q)
N₂-1 Σ
(2)
log P(a P(a4slv.q.as))
where and q are given prefix to generate the swer a. Note that P(a1 v, q. ago) :- P(ailv, q).
The question q [91...... 4N2 DD and answer = ( a... E No tokens are tracted from raw question 2ig and answer xa texts by a tokenizer, where N, and N₂ are the numbers question and answer tokens respectively. The input prompt with visual tokens for LLaMA-VQA is provided Tab. 1, where v and q serve as prefix tokens, see Sec. B for further prompt details. For simplicity, we omit the tokens for the prompt tem- plate (e.g., "Video:" and "Question:") and only consider content tokens (e.g.. '<v,>' and "<ques- tion>') in our equations. Note that q E N rep- resents only the question tokens and choice tokens are omitted in following notations. Then, token sequences v. and a are concatenated and fed to
Then, the probability in Eq. (2) is calculated as:
P(at+ilv, = Softmax(Linear(tt)) (3)
At the inference phase, the model predicts the an- swer as:
à arg max P(alv. q).
(4)
acAA
where A is set of candidate answers. i.e., choices. We now flip the role of inputs and outputs and define two auxiliary tasks: question generation and video prediction.
VA + Q. Similar to Lvgas we also encourage the model to generate the question from the video and

---

answer as:
p()lv,q) P(g'v, Pall)) Hence, learn- ing likelihood P(q'v, a) via the question gener- ation given a video and an answer benefits the pri- mary task of predicting the answer given a video and a question, which is the posterior probability P(alv, q). Similarly, the same argument holds for video prediction: P(alv, q) P(v q. a). These relationships explain why training VQA model with flipped tasks boosts the performance of the target task. More detailed discussion is provided in Sec. E.
vay = = -log a)
-
(5)
Σ
log (1111)v,,,gs))
where P(q++1|v a, q Softmax (Linear (At)). By Eq. (5), LLaMA-VQA has to generate the ques- tion which derives the answer from the video, lever- aging its prior knowledge of temporal and causal reasoning.
QA + V. Another flipped task is video prediction given question and an answer. It is formulated as:
Experiments
We verify the effectiveness of our framework leverage the powerful prior knowledge induced by an LLM. For a thorough analysis, our framework is applied to various LLMs: LLaMA (7B, 13B, and 33B), OPT (125M ~ 6.7B), GPT-J (6B). We conduct experiments and analyses to answer the following research questions:
Laur log P(vlq. a)
N,
= Σ
log P(rrtllq,,,ss
In contrast the text generation loss in Eq. (2) and Eq. (5), which selects token among the fixed vocabulary set (discrete space), it is too challeng- ing to generate a video. So we instead adopt In- foNCE (Oord et al., 2018) to maximize the mu- tual information between the input frame feature 1++1 IR R and the output feature (hi IR (R of LLaMA-VQA. Then, the likelihood in Eq. (6) is calculated as:
Q1. Do LLMs possess the knowledge of temporal and causal reasoning? Q2. Is Flipped- VQA effective for dealing with challenging VideoQA? Q3. How does Flipped- VQA alleviate linguistic bias?
Datasets. We experiment on five multiple-choice VideoQA benchmark datasets (NEXT-QA, STAR, DramaQA, VLEP, and TVQA) which require chal- lenging temporal and causal reasoning abilities. Further experimental settings and implementation details are provided in See. and Sec. B.
exp(Ue11 11)) Ny 1=1 exp(u; h2))
P( (q. a, Uct)
(7)
where hi is the token representation right before the start of visual tokens. This encourages the model to predict the order of video frames given preceding frames, i.e., next frame prediction, by an- alyzing the question and answer with LLMs' prior knowledge. This formulation enables video predic- tion via a unified text- eext-generation-aee with minimum modification.
4.1 Temporal and causal reasoning of LLMs
We investigate LLMs' strong prior of temporal and causal reasoning to answer Q1 by comparing our framework with both bLMsd and non-LLMs- based models for VideoQA.
QA model
Comparison of various sizes of LLMs. We first conduct the experiment on various LLMs sizes verify the effectiveness of LLMs' temporal and causal reasoning ability on VideoQA in Fig. 3. Note that Flipped-VQA is not applied in this experi- ment to show that LLMs already possess strong rea- soning ability themselves. In Fig. 3a, we evaluate various sizes of LLMs trained with entire (V. Q, A) triplets and the result shows that the performances on causal and temporal questions are dramatically improved as the model size increases. On the other hand, the performance gain of descriptive questions is relatively smaller than causal and temporal ques-
We combine all three objectives, which are LLMs' language generation losses and its variant. Finally, we train I.LaMA-VQA with the following loss:
CFlipped-VQA = Lvga + Lvag + Lqav-
We accumulate gradients of three different objec- tives and then update the learnable parameters. Remarks. We observe that the objectives of the primary task and auxiliary tasks can be inter- preted as learning posterior and likelihood, respec- tively. For instance, by the Bayes rule, we have

---

Models
Languagee Model
trainable paramns
NEXT-QA
STAR
DramaQQ
VLEP
TVQA
Seq.
Tal.
Tor.
HGA (Jing
2020)
50.4
FrosmBiLM (Yang ot al,, 2022)
DeBERTa
30M
82.0
MERLOT (Zellers
2021)
RoBERTa
HCRN (L. et al., 2021)
ISTM
44M
45.9
49.3
53.7
48.2
SPCRI (Kim et
BERT
81.0
76.2
VGT (Xino er al., 2022)
BERT
53.4
56.4
69.5
56.9
(Wang 2023a)
110M
50.8
47.8
Vidl. (Cheng al. 2023)
BERT
25M
79.0
ATP (Buch
CLIP
MIST (Gao al., 2023)
54.6
36.6
06.9
57.2
55.6
54.2
54.2
44.5
53.9
HileA (Ne
BERT
62.4
58.3
754
IntemVidon (Wing et al., 2023)
C C.JP text enooder
1,38
62.5
58.5
75.8
63.2
62.7
65.6
54.9
51.9
58.7
63.9
57.2
LLaMA-VQA (Ours)
75.8
57.2
Table 2:
Comparison on five challenging VideoQA benchmarks with non-LLMs-baseed models. NExT- QA involves causal, temporal, and descriptive question types. STAR contains four question types: interaction, sequence, prediction, and feasibility. Total accuracy is highlighted in grey.
Temperei Dencriptioe
Models
LLMSs
# total params
truinable parsin
Can.
Tem.
NExT-QQ
Des.
Tex.
Accurscy (%)
BLIP-2 (L. al., 2023a) SeVil A (Muerall 2023g
FlaaTS
12.18
188M
65.2
FlanTS
12.18
168
74.3
60.4
$1.3
73.8
Accursa
LLaMA
70
4.5M
72.7
69.2
75.8
72.0
HLMA--Q4
11aMA
138
8M
75.3
71.7
75.9
74.2
Inowl Decriptie
1L1MA
338
USM
Table 3: Comparison with LLM-based models.
parameters LLMS
of parameters LLMS
(a) VQA
(b) QA
Figure 3: Performances of LLMS on three question
Comparison with non-LL models. In Tab. 2, we then show the results of LLaMA-VQA in comparison non-LLMs-bbseed models five challenging VideoQA benchmark datasets. Our LLaMA-VQA outperforms all the baselines across various datasets by a significant margin, especially on causal and temporal questions. For example, in NEXT-QA, the performance gain in descrip- tive questions type is marginal compared to In- tern Video, but it yields more than 10% improve- ments in both temporal and causal questions. Also, in STAR, LLaMA-VQA surpasses MIST all question types resulting an 11.5% increase in total accuracy. Particularly, the performance on sequence questions, which ask for temporal rea- soning about consecutive actions, is improved by large margin of 13.7%. These results highlight that LLMs-based LLaMA-VQA achieves remark- able capability, especially temporal and causal reasoning questions compared to non-LLMs- Mas-Leds- models, by mainly leveraging its pretrained prior (introducing only 4.5M learnable parameters).
types of NEXT-QA. Performances of various sizes of OPT (125M ~ 6.7B) and LLaMA (7B ~ 33B) are re- ported. A VideoQA approach with a larger language model achieves better performance in both VQA and QA settings. Surprisingly, the QA approach with LLaMA (33B) outperforms VQA models with OPT (125M ~ 6.7B) in temporal and causal reasoning.
tions. The performance gap between descriptive and causal questions has decreased from 17.2% on 125M to 1.1% 1 on 33B.
Also, to verify the LLMs' prior knowledge of temporal and causal reasoning, we evaluate L.MSs trained with only (Q. A) pairs by forcing the model solely rely the question. In Fig. 3b, with only linguistic information (i.e., question), the perfor- mance of causal questions is lower than descriptive questions on 125M, but significantly improves as the model size increases and outperforms the de- scriptive question accuracy on 33B by a margin of 6.6%. Without visual content, this model already outperforms MIST (Gao et al., 2023) in Tab. 2, non-LLMs-basse model, in terms of causal and temporal question types. These results suggest that larger LLMs possess more powerful prior of causal and temporal reasoning obtained during pretrain- ing, and such prior plays a significant role in ex- ploiting linguistic shorteuts for complex VideoQA.
Comparison with LLMssbased models. We also explore larger LLaMA- VQA (~ 33B) and compare them with LLMs-based models NEXT-QA in Tab. 3. Our LLaMA-VQA 33B outperforms BLIP- and ViL in terms of the total accuracy only with 9.2M trainable parameters. Specifically, the performance gain on causal and temporal questions

---

1.Ms
Sizes
Kpochs
Objecdives
NEAT-QA
STAR
DramutJA
Eup
4ym
57.0
51.7
73.0
57.2
56.6
73.2
62.3
63.3
78.7
Choices:
Choéces:
push himself
Look lady
62.8
59.3
80.7
steer wieel
continue cycling
GPT4
signal
walk toward the red chair
64.2
60.1
81.1
paddle hold to long roge
point at
script
67.1
63.7
82.7
touch her own face
<<<
LLaMA 7B
68.7
71.2
60.9
6.1
82.6
63.3
① huld long rope
Answer:
① Look at the lady in red
72.0
Original
Original Question:
Table 4: Comparison of various LLMs and objectives. Total accuracy is reported.
How did the man in red pants keep contact speeding boal?
What did the girl do after she stogped hitting the hoand front her?
Generated Question:
Generated Question:
is 2% and 3.2% compared to SeViLA. On the other hand, the accuracy of LLaMA-VQA on descriptive questions is lower than baselines since they were further pretrained with large-scale image-caption pair datasets, which boosts the descriptiveness ca- pability of visual content. Finally, as the model size of LLaMA- VVQA increases (7B - 33B), the performance on causal and temporal questions is increased by 3.5% and 3.4%, respectively implying that larger LLMs have more powerful temporal and causal reasoning capabilities.
balance?
What did pirl do down the middle the viden?
(a) Causal question
(b) Temporal question
Figure 4: Examples of question generation.
task, Flipped-VQA accumulates gradients from three different objectives. So we conduct an ad- ditional experiment by increasing the gradient ac- cumulation steps three times more than the base- line, i.e., additionally train Lvga for 15 epochs while the others are trained for epochs. In Tab, 4, the performance of Cvp with 15 epochs on par with the one with 5 epochs across various LLMs and datasets. This suggests that the performance gain of Flipped- -VA creased gradient accumulation training sched- ulc, but comes from the capability of LLMs' prior knowledge exploited by Lvay and L
4.2 Flipped-VQA on challenging VideoQA
We here discuss Q2 by amalyzing the effectiveness of Flipped-VQA challenging VideoQA.
Ablation studies of Hiipped-QQA Tab. 4 shows the ablation studies of VQA on various LLMs (OPT, GPT-J, and ILLMA).. Compared to the baseline LLMs with Lyqa, introducing ques- tion generation objective Vvyy improves the perfor- mances by 3.7%, 1.6%, and 2.5% NExT-QA on OPT, GPT-J, and LLaMA, respectively. This result demonstrates that generating intricate questions of NExT-QA given a video-answer pair encourages LLMs to leverage their temporal and causal reason- ing knowledge to predict the answer. In addition, further improvement is observed by adapting video predicting objective Lyav that helps understand the order of visual contents based the question and answer. The accuracy GPT-J increased by a margin of 2.9%, 3.6%, and 1. 1.6% respectively on NEXT-QA, STAR, and DramaQA. Overall, each component of Flipped- VQA improves the perfor- mance across various LLMs, implying that Flipped- VQA is effective training objective for LLMs to deal with challenging VideoQA by leveraging their pretrained knowledge.
VQA does not come from the in-
Qualitative results of generated questions. Fig. 4 illustrates examples of questions generated by LLaMA-VQA conditioned on the video and an- swer with the objective of Lvaq, in the NEXT-QA validation set. We observe that the majority of (V. plau-
Q, A) triplets with generated questions sible enough to answer, while the generated ques- tions depict different aspects from the video than the originals. For instance of the causal question in Fig. 4a, LLaMA-VQA combines the visual con- tent, a man waterskiing, with the answer "hold on to long rope" and expresses as the man is "main- tain balance". Note that the idea of "keep[ing] contact" in the original question aligns with the idea of "maintain[ing] his balance", there no difficulty for the model answer the generated question based on the given video. Hence it shows how LLMs are using their pretrained knowledge to generate the question appropriate to the given
Unlike solely using Luga to perform the main

---

Explosting linguistic shoneut
Miligating linguistic hiss
P (Fuvo Y Eng x)
(Kupes
Riv xX)
x
*
Ratio
62.78 87.3% 50.7%
53,8%
Question:
Question:
Why are there lines everywhere on ground?
What did the black after
Table 5: Ratio of the number of samples. Yaiq de-
the buby picked up the towel?
notes the prediction of the model trained with only (Q, A). Y y stands for the prediction of LLaMA-VQA either trained with or without Flipped-VOO VQA. Y is ground truth.
Cholees:
Cholees:
① allow the girl to play
① continue feeding
O man draws ground
louch buby's ficel
② dogs are running
① stares at the camera.
② plane take off icids playing
bahy's 's + keeps it back
YAiQ
YAQ Q
video and answer.
Answer: msn drows on ground
Answer: O rub the baby's face
More interestingly, LLaMA-VQA is capable of understanding intricate interactions present in the video. For the temporal question in Fig. 4b, unlike the original question that asks for the action after the girl "stop[s] hitting the board", the generated question asks for the action after "ioo down". This reveals that LLaMA-VOR VQQA understands the in- teraction between objects and the sequential events in the video, and thus it can generate temporal ques- tions adequate to the answer. These results suggest that ILaMA-VE -VQA successfully understands the complex relationship between the vidco, question, and answer with Flipped- by leveraging its prior knowledge of temporal and causal reasoning.
YANE Flipped-VOQ.
Faiv.q wis Ritpped-VQA
Answer: O man drows on ground
Answer: @ rub the baby's fleee
Yann w/ Flipped-VQA
Yalv.o Fllpped-VQA
Answer: O plane take ell
Answer: @ keeps it back
(a) Causal question
(b) Temporal question
Figure 5: Examples of alleviation on linguistic bias.
Here, the correctness of linguistic prior is de- fined as the accuracy of the QA model that pre- dicts answers solely based on the language, i.e., P( AIQ Y). Secondly, we analyze how effec- tively LLaMA-VQA mitigates linguistic bias by the following metrie:
4.3 "lipped-VQA for mitigating linguistic bias
P Y +X).
(10)
We observed that Flipped-VQA is crucial to ad- dress linguistic bias. So we finally analyze how Flipped-VQA alleviates linguistic bias (Q3) by pro- viding detailed quantitative and qualitative analy- sis.
This measures the accuracy of the VQA model when the linguistic prior is wrong, i.e., YAIQ # Y. Tab. shows that Flipped-VQA improves the accu- racy on the samples with linguistic bias (inaccurate linguistic priors) by 3.1%. Our in-depth analysis of attention and embeddings also shows that Flipped- VQA encourages LLMs to leverage more visual content and better align the visual embedding space with LLMs' text embedding space, see Sec. F for details.
Quantitative results of bias mitigation. Although LLMs' strong prior of temporal and causal reason- ing is beneficial to exploit linguistic shorteuts for challenging questions, this sometimes leads sub- optimal results by forcing the model to overly rely on the text question while ignoring the visual con- tent. This problem, linguistic bias, is commonly observed in visual question answering (Niu et al., 2021; Ramakrishnan et al., 2018; Cadene et al., 2019). Our extensive analyses show that Flipped- VQA mitigates the linguistic bias while effectively leveraging linguistic shortcut. We first analyze how effectively LLaMA-VQA exploits linguistic short- cuts. Specifically. Tab. shows that Flipped- VQA improves the accuracy of LLaMA-VQA on NExT- QA by 4.6% when the linguistic prior is correct. It is measured by the following equation:
Qualitative results of bias mitigation. We here further analyze the effectiveness of Flipped-VQA in mitigating linguistic bias with qualitative results. Given incorrect linguistic prior, i.e., YAIQ * Y, in other words, when the prediction by the language- only model is wrong, our model trained with Hhipped-VQA better rejects the wrong linguistic bias than the one trained without Flipped-VQA. For example in Fig. 5a the language-only model out- puts "man draws on ground" for the causal question "Why are there lines everywhere on the ground?". LLaMA-VOR trained without Flipped-VO2 fails to reject the wrong linguistic prior and chooses the
P (VAvvo=Y/b =Y).

---

plausible-sounding answer based on the common knowledge in the pretrained language model with- out referring to visual content. This can be viewed as the hallucination and ungrounded guess problem observed in Alayrac et al. (2022). On the other hand, LLaMA-VQA trained with Flipped-VOO VQA refers to the visual content that depicts a plane leaving traces the snow as it is taking off and successfully predicts the actual cause "plane take off". The enhanced attention between the answer and visual tokens supports that Flipped- TiipeedVQAA VQA en- courages the model to refer to visual content, see Sec. for more details.
Limitations
We propose a Flipped- VQQ which can be widely adaptable to decoder-only LLMs by improv- ing their performances on challenging Vi Flipped-VQA effectively leverages LLMs prior knowledge of temporal and causal reasoning with generative objectives of next token prediction. Ex- tending it to incoder-decoded LLMs with ob- jective other than the next token prediction can be interesting. Also, although the number of trainable parameters of LLaMA-VQA is only 4.5M 9.2M, the total number of parameters is inherently large (7B ~ 33B), which mainly comes from backbone LLMs, LLaMA. This leads to massive memory usage during training/fine-tuni and inference.
Similarly. LLMs' temporal prior occasionally disrupts identifying the action followed by an event. For the temporal question in Fig. 5b, the act of wiping off follows the act of "pick[ing] up the towel" in general. Hence, the language-only model YaiQ and LLaMA-VQA YaII.Q without Flipped- VQA predict "rub the baby's s face." In contrast, the proposed method with Flipped-VOL VQA accurately predicts "keeps it back" by referring to the video. These results demonstrate that Flipped- VQQ en- courages the answers grounded on visual informa- tion and mitigates linguistic bias.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: visual language model for few-shot learning. In NeurlPS.
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022. Revisiting the" video" in video-language understand- ing. In CVPR.
5 Conclusion
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, 2019. Rubi: Reducing unimodal biases for visual question answering. In NewrIPS.
In this paper, we investigate the large language models' (LLMs') temporal and causal reasoning abilities on the challenging multi-modal Video Question Answering (VideoQA) task. We observe that the larger LLMs possess more powerful prior knowledge of temporal and causal reasoning in ad- dressing complex VideoQA. Moreover, we propose a novel framework, Flipped- VQA, that effectively leverages the LLMs' knowledge of temporal and causal reasoning on understanding the complex (V. Q. A) triplet by introducing three generative objectives: Lera: fiva.. and Lynv- Our in- in-depth anal- yses show that "lipped-VQA not only enhances the exploitation of linguistic shortcuts but also miti- gates linguistic bias that causes hallucination and ungrounded guess problems.
Shanqing Cai, Subhashini Venugopalan. Katrin
Tomanek, Ajit Narayanan, Meredith R Morris, and Michael P Brenner. 2022. Context-aware abbrevi- ation expansion using large language models. NAACL.
Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! large language models speak too positively about negative commonsense knowledge. In ACI..
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint 2107.3307.0334
Acknowledgments. This work was partly sup- ported by ICT Creative Consilience program (IITP.
Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mo- hit Bansal, and Gedas Bertasius. 2023. Vindlu: A recipe for effective video-and-language pretraining. In CVPR.
2023-2020-0-018199 supervised by the IITP, the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2023R IA2C2005373). and KakaoBrain cor- poration.
Seongho Choi, Kyoung-Woo Woon On, Yu-Jung Heo, Ah- jeong Seo, Youwon Jang, Minsu Lee, and Byoung- Tak Zhang, 2021. Dramaqa: Character-centered video story understanding with hierarchical qa. AAAI

---

John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while main- taining accuracy: Text data generation with large language models and human interventions. ACL.
Yao Jin, Guocheng Niu, Xinyan Xiao, Jian Zhang, Xi Peng, and Jun Yu. 2023. Knowledge-constran answer generation for open-ended video question an- swering. In AAAI.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv prepriat
Chen Ju, Tengda Han, Kunhao Zheng. Ya Zhang, and Weidi Xie. 2022. Prompting visual-language models for efficient video understanding. In ECCV.
Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafici. 2023, Evaluating open-domain ques- tion answering in the era of large language models. In ACI.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, 2021. Training verifiers solve math word problems. arXiv preprint arXiv:2110. 14168.
Emre Kıemman, Robert Ness, Amit Sharma, and Chen- hao Tan. 2023. Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint arXiv:2305.0005.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. ACL.
Seonhoon Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak. 2021. Self-supervised pre- training and contrastive representation learning for multiple-choice video qa. In AAAI.
Difei Gao, Luowei Zhou, Lei Ji. Linchao Zhu, Yi Yang. and Mike Zheng Shou. 2023. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. In CVPR.
Dohwan Ko, Ji Soo Lee, Miso Choi, Jaewon Chu, Jihwan Park, and Hyunwoo J Kim. 2023. Open- vocabulary video question answering: A new bench- mark for evaluating the generalizability of video ques- tion answering models. In ICCV.
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In ICLR.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2021. Hierarchical conditional relation networks for multimodal video question answering. IJC.
Namgyu Ho, Laura Schmid, and Se- GouYo Yun. 2022. Large language models are reasoning teachers. In ACL.
Marius Hobbhahn, Tom Lieberum, and David Seiler. 2022. Investigating causal understanding in llms. In NewrIPS Workshop.
Jie Lei, Tamara I Berg, and Mohit Bansal. 2023. Re- vealing single frame bias for video-and-language learning. In ACL.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv prepriat arXiv:2203. 15556.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg, 2018. Tvqa: Localized, compositional video ques- tion answering. In EMNLP.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. What is more likely to happen next? video- and-language future event prediction. In EMNIP.
Edward Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. In ICLR.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In ICML.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Qiao. 2023b. Videochat: Chat-centric video un- derstanding. arXiv preprint wrXiv:2305.0633.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. 2022. Visual prompt tuning. In ECCV.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang. Yinan He, Limin Wang, and Yu Qiao. 2023c. Unmasked teacher: Towards training-efficient video foundation models. In ICCV.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Lim-blender Ensembling large language models with pairwise ranking and generative fusion. In ACI..
Lei Li, Yuwei Yin. Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. 2023d. M3it: A large- scale dataset towards multi-modal multilingual in- struction tuning. arXiv preprint arXiv:2306, 043387.
Pin Jiang and Yahong Han. 2020. Reasoning with het- erogeneous graph alignment for video question an- swering. In AAAI.

---

Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoff- mann, Cyprien de Masson d' A'ttumee Phil Blunsom, and Aida Nematzadeh. 2022. A systematic investiga- tion of commonsense knowledge in large language models. In EMNLP.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397
Xingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng. Shafiq Joty, and Lidong Bing. 2023e. Un- locking temporal question answering for large lan- guage models using code execution, arXiv preprint MrXiv22305.15144
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: / http:/ggttuuuuooiigggg////o////e com/ mash-trranfformer-jaa
Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui. Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, al. 2023a. All in one: Exploring unified video-language pre- training. In CVPR.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023, Evaluating the logical reasoning ability of chatgpt and gpt-4, arXiv preprint arXiv:2304.03439.
Stephamie Long, Tibor Schuster, Alexandre Piché, Ser- viceNow Research, al. 2023. Can large lan- guage models build causal graphs? arXiv preprint arXiv:2303.05279.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b. Plan-and-solve prompting: Improving zero- shot chain-of of-thouggtt reasoning by large language models. In ACL.
Ilya Loshchilov and Frank Hutter. 2017. pled weight decay regularization. arXiv preprint arXiv:1711.0510..
Decou-
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, 7un Wang, al. 2022. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counter- factual vqa: cause-effect look at language bias. In CVPR.
Yiming Wang. Zhuosheng Zhang, and Rui Wang. 2023c. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of- thought method. In ACL.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.03748
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenen- baum, and Chuang Gan. 2021. Star: A henchmark for situated reasoning in real-world videos. In NeurIPS.
Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Nebo- jsa Jojic. 2023. Thinksum: Probabilistic reasoning over sets using large language models. In ACL.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- ur. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In /CML.
Junbin Xiao, Pan Zhou, Tat-Seng Chua, and Shuicheng Yan. 2022. Video graph transformer for video ques- tion answering. In ECCV.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Ste- fan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In NeurlPS.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined atten- tion over appearance and motion. In ACM Multime- dia.
Swarnadeep Saha, Peter Hase, Nazneen Rajani, and Mohit Bansal. 2022, Are hard examples also harder explain? a study with human and gener-egeeertte explanations, In EMNLP.
Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. 2021. Just ask: Learning to answer questions from millions of narrated videos. In ICCV.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022. Zero-shot video ques- tion answering via frozen bidirectional language mod- els. In NewrIPS.
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In ACL.
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang, 2022. Hitea: Hier- archical temporal-aware video-language pre-training. arXiv preprint arXiv:2212. 14546.

---

Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2023. Self- ceffnchained image-lane languae model for video localization and question answering. arXiv preprint arXiv:2305.068. 06988.
on visible contents such as places, and objects/at- tributes. About 5K videos of average 44s and 48K QA pairs with five answer candidates are given. Further examples of each question type are pro- vided in Sec. D.
Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji, Lijun Wu, Nong Xiao, and Nan Duan. 2021. Learning from inside: Self-driven siamese sampling and reasoning for video question answering. In NeurIPS.
DramaQA (Choi et al., 2021) features video story understanding with hierarchical difficulty levels. The level is determined by the required length of the clip (shot or scene) and the number of logical reasoning steps to answer the question. The dataset contains 24K video clips and 18K QA pairs with five answer candidates. Average video lengths are 3.7s for the shot and 91 .3s for the scene.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu. Zhou Zhao, Yuet- ing Zhuang. and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. Proceedings of the AAAI Con- ference on Artificial Intelligence.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can machine really fimish your sentence? ACL.
STAR (Wu al., 2021) is designed for situational reasoning with questions that tackle interaction, se- quence, prediction, and feasibility of events. There exist 60K QA pairs with four answer candidates and 22K video clips.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngiae Yu. Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. In NeurIPS.
VLEP (Lei et al., 2020) uses TV shows and YouTube Vlogs with average of 6.1s that contain rich physical interactions and dialogues between people. The challenge is to determine which of two future events is likely to occur in the given video (with dialogue). It is comprised of 29K QA pairs with 10K vidco clips.
Cheng Zhang, Stefan Bauer, Paul Bennett, Jiangfeng Gao, Wenbo Gong, Agrin Hilmkil, Joel Jennings, Chao Ma. Tom Minka, Nick Pawlowski, al. 2023a. Understanding causality with large language mod- cls: Feasibility and opportunities. arXiv preprint arXiv:2304.05524.
Hang Zhang, Xin Li, and Lidong Bing. 2023b. Video- Ilama: An instruction-tunee aidia- language model for video understanding. arXiv preprint arXiv:2306.02858.
TVQA (Lei et al., 2018) is built on long video clips (60-90s) of six different TV shows with various social interactions and activities. provides dia- logues for each video with 153K QA pairs and 22K video clips.
Michael JQ Zhang and Eunsol Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into qa. In EMNLP.
B Implementation details
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023c. Llama- adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint 2rXiv:2303. 16199.
Training details. LLaMA-VQA is trained for five epochs on all datasets with a batch size of 32. LLaMA-VQA 7B and 13B are trained with 8 x A6000 GPUs and LLaMA-VQA 33B is trained with 8 x A100 GPUs. AdamW opti- mizer (Loshchilov and Hutter, 2017) is used with 3 = (0.9, 0.95). We search learning rate and weight decay in [0.05, 0.1] and [0.15, 0.25], respec- tively. Following LLaMA-Adapter for each layer of LLMs, 10 adapter tokens are used, i.e. = 10. The number of video frames N₂ is set to 10. Each frame is resized by 224 x 224 and fed into CLIP VIT-L/14 (Radford et al., 2021) to extract frame features. The total sequence length of the con- catenated visual, question, and answer tokens, N. + N₂ + N.. is 128, 128, 384, 256, and 512 for NEXT-QA, STAR, DramaQA, VLEP, and TVQA respectively. Each dataset optionally provides dia- logues. We append dialogues as prefix tokens for
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diah, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 0050002205.01068.
Appendix
A Dataset details
NEXT-QA (Xiao et al., 2021) consists of three types of questions. Causal questions ask for the intentions of earlier actions or reasons for succeed- ing ones. Temporal questions determine the rela- tionships between actions that are solely based on the sequence of occurrence (e.g. what do af- ter/before/while..... ...). Descriptive questions focus

---

[SOS] Video: <n>> <1n2.... <U
[SOS] Question: <question>
Question: <question>
Choices:
Choices:
(A) coption 1>
(A) <option 1>
(B) <option 2>
(B) <option 2>
(C) coption 3>
(C) <option 3>
(D) <option 4>
(D) coption 4>
(B) <option 5>
(E) <option 5>
Answer: The answer is <answer>
Answer: The answer <answer> [EOS]
Video: <リソ> <o2>... <UN,>
Table 6: Input Prompt of VQ A.
Table 8: Input Prompt of QA -
[S0S] Video: <i1> <02>
<UN,>
where N = Ny. + No. Note in our work, according to each objective of HippeedVQAA the order of q. and a is permuted. Then, the scaled dot- dot-product between Q and K is calculated as:
Choices:
(A) <option 1>
(B) <option 2>
(C) coption 3>
L-QK/VD RNN((N++)), (12)
(D) <option 4>
(E) <option 5>
S in Eq. (12) can be divided into two groups as:
Answer: The answer <answer> [EOS]
Question: <question> [EOS]
,
(13)
Table Input Prompt of VA + Q-
where SN RNNN,, is the attention scores of adapter tokens p and S E NNN is for v, q. a tokens.
VLEP and TVQA. Lvag is not applied in VLEP since questions of all samples in VLEP are consis- to "Which event is more likely to happen right after?".
Moreover, to adjust the contribution of newly adopted tokens p at the beginning of training. LLaMA-Adapters introduce a zero-init attention gate as:
Prompt details. The general input prompt of LLaMA-VQU is provided in Tab. 1. Also, Tab. 6, Tab. 7, and Tab. 8 provides detailed input prompt of each task in Flipped- VQA, respectively. In those tables. non-prefix tokens, which the model needs generate, are highlighted in red and the rest are prefix tokens.
S = [Softmax(S*%) g. Softmax(s*) (14)
where g initialized zero. Eq. (14) preserves the LLMs' knowledge at the beginning of training and gradually increases the influence of adapter tokens p. Finally, the output of LLaMA-Adapter is as follows:
C LLaMA-Adapter
H - Linear NXD
(15)
LLaMA-Adapter (Zhang al., 2023c) adopts set of learnable adapter tokens p [pI,....,PN. PN,] E N,x efficiently fine-tune LLaMA (Touvron et al., 2023), where Np is the number of adapter tokens and D is a feature dimension. The adapter tokens are then concatenated as prefix tokens for the key and value of each self-attention layer, for- mulated as:
D NEXT-QA
In NEXT-QA, causal and temporal questions ac- count for 48% and 29% respectively the dataset. Specifically, there exist three types of questions: Causal, Temporal, and Descriptive. In general, questions and answers for temporal and causal types longer than descriptive ones.
Causal questions seek for event A which hap- pens in advance of event B and is also responsible for B's occurrecce in the video. These questions are broken down into asking "how" such an event
Q Linear, y ([v, q. a)) E N
K Linear, ((p.v.q. No+N
(11)
V
al) E NN+,

---

Canaal Quettion:
Temperal Qunetio::
Descrppiie Question: the misticrahip between not and the stack?
why did shones mouth nor the video?
doet the beflee white nhèoe?
Choicee:
loader
Cholcec jumps rake
Choloee
trainee
to balance
profucer and cluld colleagues
shockted
board
Figure Examples NEXT-QA.
occurred or "why" the object acts in certain way. For instance, Fig. 6 (left) asks "Why did the short- est girl cover her mouth near the end of the video?". Temppral questions are closely related causal- ity but require reasoning solely based on the se- quence of occurrence (present, previous, or next actions) and further ask to focus on interactions of multiple objects. For example, question regard- ing the previous action asks "What does the man in red do before the man in white cut the ribbon?" in Fig. (middle).
2:10
Attention Score
2.34
vinal arlippddvcc Lanpupplsterr
10.00
2008
2003
300
Ioraaionn
(a) T-SNE
(b) Attention
Figure 7: Visualization of T-SNE and attention score. (a) Each input token embeddings of visual and ques- tion q is visualized. (b) Attention score between answer query tokens and visual key tokens visualized.
Lastly, descriptive questions tend to ask about the video general (i.e., the places, objects/at- tributes, and main actions/events). For instance, Fig. (right) asks "What is the relationship be- tween the guy in red and the rest of the people in black?".
F Embedding space alignment of Flipped-VQA
To bridge the visual encoder embedding space with LLMs text embedding space, we adopt a simple linear projection layer f. We observe that without Flipped-VQA, f is not trained effectively the visual tokens are just used prefix tokens which are excluded from the generation target of LLMs. However, with Flipped- Bipped-VQQ VQA which directly prop- agates the loss to the visual tokens (red arrows of Lyav in Fig. 2), f is trained to align visual encoder embedding space with frozen LLMs text token em- bedding space. We visualize the embedding space of question tokens q and visual tokens with and without Flipped-VQA in Fig. 7a, and show that the embedding space of visual tokens with Flipped- VQA (orange) is closer to LLMs' text embedding space (red) compared to the one without Flipped- VQA (blue).
E Discussion on Flipped-VQA
The primary task of VideoQA, predicting the an- swer given the video and question, can be rewritten
P(alv, q) ((vla,q)P(app x P(vla,g). P(vlq)
(16)
Eq. (16), we observe that the auxiliary task of predicting visual tokens v given and can be viewed as maximum likelihood estimation (i.e., P(vja, q)) and the primary task as the maximum a posterior (MAP) estimation (i.e., P(a|v, q)), re- spectively.
Similarly, the auxiliary task of predicting the question token q given a and be correlated with the primary task as:
Furthermore, in Fig. 7b, we plot the attention score between the answer query tokens and visual key tokens, i.e., measuring how much visual tokens affect answer tokens As training proceeds, the attention score of both LLaMA-VOL with and without Flipped- VQQ gradually increases, repre- senting that the model leverages more visual con- tent to answer the question. However, after the entire training iterations, the attention score with
P(alv, q) P(qla, P(aalv) x P(qla, v). P(glv)
(17)
Therefore, by maximizing the likelihoods of auxiliary tasks, CFlipped-VQA in Eq. (8) aims to strengthen the performance on the main task.

---

I external visual-text data samples
questions by Lvag in Fig. For the example in the middle of the first row, the generated question successfully describes the video, where the man moves across the muddy area, so the model can an- swer "jeep" based on this question. Also, LLaMA- VQA generates questions by referring to different timestamps of the video. In the right example of the first row, the original question asks the reason why the lioness bends its head in the middle of the video lead to the answer "drink water". However, the lioness also touches the river to drink water at the beginning of the video, and the generated ques- tion asks the reason for touching the river to obtain the answer "drink water". This suggests that Lvaq of Flipped-VQA helps to understand the complex relationship between video, question, and answer with LLMs' prior knowledge.
Models
WUPS
HGA Ulong and Han, 2020)
0
25.2
KcGA (Jin 2023)
Flamingo O-shot (Alaynac all,, 2022)
2.18
26.7
Flamingo 52-hot (Alayrac et al., 2022)
2.18
33.5
LLaMA-YQA (Ours)
Table 9: Results of NEXT-QA.
external visual-text data samples
Models
Accuracy
JustAsk (Yang al., 2021)
SiaSamRea (Yu 2021)
5.6M
MERLOT (Zellers et al., 2021)
180M
41.4
FrozenBiLM (Yang et al., 2022)
10M
43.2
Singularity (Lei et al., 2023)
17M
44.1
FrozenBiLM+ (Ko et al., 2023)
10M
44.8
UMT-L al., 2023c)
25M
LLaMA-YQA (Ours)
Table 10: Results of Act.vityNet-QA.
Bias alleviation of Flipped-VQA. In addition the examples in Fig. 5, we provide further qualita- tive results that FIp enables the allevia- tion of linguistic bias. For the left example of the first row in Fig. 9, the model trained with question- answer pairs (Y (Ya))) predicts "microphone" for the temporal question "What are the men holding on to when they speak?", based the prior knowl- edge that people usually the microphone when they are speaking. LLaMA- VQQA trained without Flipped-VO VQA still fails to reject the wrong linguis- tic bias and predicts the same answer. On the other hand, LLaMA- -QAM trained with Flipped- VQA ac- curately outputs the answer "glasses" by taking into account the video where the men are holding glasses, resulting in the reduction of hallucination problems with the mitigation of linguistic bias.
VIA (red) three times larger than the one without Flipped-VOR VQA (blue), indicating that Flipped- VQA plays key role in transferring the visual representation into the LLMs embedding space and enhances the utilization of visual content LLMs. These results demonstrate that Flipped- VQA encourages text-only trained LLMs to un- derstand visual content by utilizing the strong rep- resentation power of a pretrained visual encoder, effectively aligning the visual embedding space with the text embedding space.
G Further quantitative results
We conduct an additional experiment on two generation-based VideoQA benchmark datasets: NExT-QA open-form generation (Xiao al., 2021) and ActivityNet-QA (Yu et al., 2019). WUPS and Accuracy are used for evaluation metrics in NExT- QA open-form generation and Act,,ityNet-QA, re- spectively. In Tab. 9, our LLaMA-VQA outper- forms mon-sLds-based models HGA and KcGA by large margin. Also, compared with LLMs- based Flamingo which is further trained with 2.1B external visual-text pairs, the performance gain is 0.8%. Furthermore, in Tab. 10, LLaMA-VQA also outperforms all the baselines although those use large visual-text data for extra training in ActivityNet-QA.
H Further qualitative results
Question generation of FQipped-VQA. We here show further qualitative examples generated

---

Choices:
Choices:
playing with them
① cycle up slope jeep
T playing
colour
distracted
read them books
① using the slod handles + spread arms out
@ pick somothing
listening music logother
drink
pot them to sleep
1 to make the dog jump
Answer:
Answer: jeep
Answer:
books
drink
Original Question:
Original Question:
Original Question:
Why does the man in place children his lap?
man transport
Why lioness its head down the middle?
accoss?
Generated Question:
Generated Question:
Generated Question:
Why are the babies sitting on the man's
How the people move across muddy area?
Why the lioness touch the river the beginning?
Cholcess
Cholees:
Choices:
playing with ball watch
① bite
C approsch lady sitting there
watching television reading book sleeping
unwrap it
O ran away
playing with toy train gesture something
① play with hond
① play with cat
Answer: watch
Answer:
unwrap it
@ play with the hand
Original Question:
Original Question:
Original Question:
What person the boy doing the boy plays with
Why did the pick from the group of them and move sofa?
puts the hand on the floor?
Generated Question:
Generated Question:
Generated Question:
doing?
does bay bend down and
How did the person interact with the
pick up the thing the start of video?
Choices:
Choices:
white cake
① cowwling
to ball plck It up
cardhoard box
grabbing on the cot
letters
@ sit tear by bit @ sitting on table
step
birthday gift food
to bounce the balloon
? kick balloon away
Answ
Answer:
Answer:
pick 31 up
Original Question:
Original Question:
Original Question:
What is on the table?
How does the kid open the present?
Why does the buy run towards the balloon
ground?
Generated Question:
Cencrated Question:
girl?
How did the boy red disassemble wrapping paper from the toy the start?
Generated Question:
Why does boy bend down towards the end of the video"
Figure Examples of question generation.

---

Question:
Question:
Questies:
What are the men holding onso when they speak?
Why baby climbing
What did the giri de after she
the floer aller walking for while the start of the vidco?
picked
Choices:
Choices:
① phone
Choices::
① take out cundle
① glasses
① from balloon
① puts baking
cumera
down the
2 walks away
① micnophene
@ tuys
① the fork her mouth
① milk pack
① fall over
① forward
① drink
YAI
YAι
Answer: @ microphone
Answer: @ put the fook ber mouth
climb down the
YaIvo Flipped- YQA
YaIViQ Flipped-YQA
YAIV:Q w/o Filpped-VQ.A
Answer: microphone
Answer: @ put the fork her mouth
Answer: climb down the stairs
YaIno w/ PRI
Y YAlvi w/ Flipped-VQA
8 Yiyvo Fllppod-VOAA
Answer: @ glasses
Amswer: @ moves forward
Amswer: over
Question:
Question:
Why did the mann with black suit and shades touch his after blowing trumper?
Wilyy the yacht floating
How did the girl in blue respond each time in multicoloured
pick some
Choices:
Cholces:
get boat
Choices:
cover his cars
help divers swim
put carphone
playing with water
@ sit down
enjoy the music
+ knock away the sand
o serotch
eajay beauty
@ pull her away grab sand
carpleg
YAio
YAi
Amswor: @
Answer: adjust earping
Answer: @ grab the sand
FAIV.Q FIipped-VQA
YaIKo wio Flipped-Ve YQA
Yaiviq Flipped-YQQ
Answer: O adjust carplug
Answer: @ aca waves
Answer: @ grab the sand
YaIv w/ Flipped-Y YQA
Y Yajoo wi VQp
YAIVi w/ Flipped-YQA
Answer: enjoy scenic heauty
Answer: @ knook the sand
Queation:
Question:
Question:
Why did the woman pass the phoce
What the baby in yellow doing
How did the boy operate the car?
flooe?
Choices:
Choices:
Choices:
haby the call baby
fell down
@ leg apainst pedal
① wanted carried
@ crawling
@ bend down and grab
① end call
C clapping playing
① remote control
baby
lap
jump baby
Aii
YAIQ
PAςQ
Answer: O let baby unswer the call
Answer: @ crawling
Answer: @ cemote control
YaIvO Flipped-YQAA
YaIVO Flipped-VQA
YAIVOO Tilpped-VQQ.
Answer: @ let baby answer the call
Answer: O crawling
Answer: C romotc oxetrol
Yaivo w/ Fllpped-VQA
Yaina w/ FII
Yaiva w/ Elipped-VOA
@ involve the
Answer: @ playing
Answer: press log against the podal
Figure 9: Examples of alleviation on linguistic bias.


Reviews:
[{"review_id": "7PkYaXrAA4", "reviewer": "Reviewer_iJms", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your strong acknowledgment of our paper. We will provide further implementation details in terms of reproducibility, hoping for more vigorous support for our paper.\n\n---\n\n**Response 1: Further implementation details.**\n\nFirst, the general input prompt of LLaMA-VQA is provided in Tab. 1 of the main paper. Specifically, the table represents the prompt for VQ → A, where the prompt from `[SOS]` to `Answer:` serves as a prefix and the model needs to generate the tokens after `Answer:`. Similarly, for VA → Q, `Video: ...`, `Choices: ...`, and `Answer: ...` tokens serve as prefix tokens and onwards is for a generation. The same manner is applied to QA → V.\n\nAlso, LLaMA-VQA 7B is composed of 32 heads and 32 layers and the feature dimension $D$ is 4096. The total number of vocabularies used is 32,000 from SentencePiece Tokenizer. For visual encodings, each frame is resized by 224 $\\times$ 224 and fed into CLIP VIT-L/14 to extract frame features. \n\nThe hyperparameters of our main experiments (Tab.2 of the main paper) are shown in the table below.\n|  | Learning Rate | # adapter tokens ($N_p$) | Batch Size | Weight Decay | Epoch | Warmup Epoch | # frames ($N_v$) | Max Sequence Length ($N_v + N_q + N_a$) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| NExT-QA | 0.0225 | 10 | 64 | 0.16 | 5 | 2 | 10 | 128 |\n| STAR | 0.0175 | 10 | 64 | 0.16 | 5 | 2 | 10 | 128 |\n| DramaQA | 0.0225 | 10 | 64 | 0.16 | 5 | 1 | 10 | 384 |\n| VLEP | 0.02 | 10 | 64 | 0.25 | 5 | 2 | 10 | 256 |\n| TQVA | 0.00875 | 10 | 32 | 0.02 | 5 | 2 | 10 | 512 |\n\nWe will include further implementation details in our final version and release our code if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are thorough and the results are convincing.\n2. The authors decouple the reasoning ability of LLMs into temporal and causal and show good insights into reducing the linguist bias.", "reasons_to_reject": "No particular reason to reject. The paper conveys good results and is well written. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors show that even though LLMs can exploit linguist shortcuts in VQA tasks, they can leverage the powerful LLMs with further tuning to reduce the linguistic bias and boost results. To achieve this, the authors propose to further train the LLM model with V-Q-A triples permuted and LLM frozen. Through extensive experiments, the authors show that the proposed Flipped-VQA achieves very good results on many benchmarks including NExT-QA, STAR, 332 DramaQA, VLEP, and TVQA.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is generally well written."}, "extra_text": {}}, {"review_id": "krsO1S2k2i", "reviewer": "Reviewer_peaG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your strong acknowledgment of our paper. We will address your concerns and provide further implementation details in terms of reproducibility, hoping for more vigorous support for our paper.\n\n---\n\n**Response 1: Experiments on generation-based VideoQA.**\n\nWe conduct an additional experiment on two generation-based VideoQA benchmark datasets: NExT-QA open-form generation and ActivityNet-QA. WUPS and Accuracy are used for evaluation metrics in NExT-QA open-form generation and ActivityNet-QA, respectively. In the below table, our LLaMA-VQA outperforms non-LLMs-based models HGA and KcGA by a large margin. Also, compared with LLMs-based Flamingo which is further trained with 2.1B external visual-text pairs, the performance gain is 0.8%.\n|  | # external visual-text data samples | NExT-QA open-form generation (WUPS) |\n| --- | --- | --- |\n| HGA | 0 | 25.2 |\n| KcGA | 0 | 28.2 |\n| Flamingo 0-shot | 2.1B | 26.7 |\n| Flamingo 32-shot | 2.1B | 33.5 |\n| **LLaMA-VQA (Ours)** | **0** | **34.3** |\n\nFurthermore, in ActivityNet-QA, LLaMA-VQA also outperforms all the baselines although those use large-scale visual-text data for extra training.\n|  | # external visual-text data samples | ActivityNet-QA (Accuracy) |\n| --- | --- | --- |\n| JustAsk | 69M | 38.9 |\n| SiaSamRea | 5.6M | 39.8 |\n| MERLOT | 180M | 41.4 |\n| FrozenBiLM | 10M | 43.2 |\n| Singularity | 17M | 44.1 |\n| UMT-L | 25M | 47.9 |\n| **LLaMA-VQA (Ours)** | **0** | **48.6** |\n\nWe compared our method with peer-reviewed works in the table aboves. But we will include discussions/comparisons with more recent/concurrent works, *e.g.*,  VALOR [1] and VAST [2], in the final version.\n\n[1] Chen et al., Valor: Vision-audio-language omni-perception pretraining model and dataset, 2023.\n\n[2] Chen et al., VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset, 2023.\n\n---\n\n**Response 2: Adaptation of Flipped-VQA to non-LLMs-based models.**\n\nWe apply Flipped-VQA to InternVideo, one of the recent strong non-LLMs-based baselines, to verify the broad applicability of our method. We modify the question generation and video prediction task as a contrastive learning task, which encourages matching VA and Q (or QA and V) pairs among batch samples, in order to apply Flipped-VQA to non-LLMs-based model. In NExT-QA, the performance gain is 0.8%.\n|  | Total |\n| --- | --- |\n| InternVideo | 63.2 |\n| **InternVideo + Flipped-VQA** | **64.0** |\n\n---\n\n**Response 3: Missing references on Video-LLMs.**\n\nThank you for the suggestions. Our work is studied concurrently with those works for Video-LLMs. We will include the mentioned works in our final version.\n\n---\n\n**Response 4: Further implementation details.**\n\nFirst, the general input prompt of LLaMA-VQA is provided in Tab. 1 of the main paper. Specifically, the table represents the prompt for VQ → A, where the prompt from `[SOS]` to `Answer:` serves as a prefix and the model needs to generate the tokens after `Answer:`. Similarly, for VA → Q, `Video: ...`, `Choices: ...`, and `Answer: ...` tokens serve as prefix tokens and onwards is for a generation. The same manner is applied to QA → V.\n\nAlso, LLaMA-VQA 7B is composed of 32 heads and 32 layers and the feature dimension $D$ is 4096. The total number of vocabularies used is 32,000 from SentencePiece Tokenizer. For visual encodings, each frame is resized by 224 $\\times$ 224 and fed into CLIP VIT-L/14 to extract frame features. \n\nThe hyperparameters of our main experiments (Tab.2 of the main paper) are shown in the table below.\n|  | Learning Rate | # adapter tokens ($N_p$) | Batch Size | Weight Decay | Epoch | Warmup Epoch | # frames ($N_v$) | Max Sequence Length ($N_v + N_q + N_a$) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| NExT-QA | 0.0225 | 10 | 64 | 0.16 | 5 | 2 | 10 | 128 |\n| STAR | 0.0175 | 10 | 64 | 0.16 | 5 | 2 | 10 | 128 |\n| DramaQA | 0.0225 | 10 | 64 | 0.16 | 5 | 1 | 10 | 384 |\n| VLEP | 0.02 | 10 | 64 | 0.25 | 5 | 2 | 10 | 256 |\n| TQVA | 0.00875 | 10 | 32 | 0.02 | 5 | 2 | 10 | 512 |\n\nWe will include further implementation details in our final version and release our code if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. The Flipped-VQA idea is creative, expanding perspectives on utilizing VQA data. Interpreting the tasks as learning posterior and likelihood provides insight.\n3. Experiments are solid and comprehensive, with comparisons to strong baselines including InternVideo (non-LLm-based) and BLIP-2 (LLM-based), and improvements over these baselines are significant. The authors also conduct extensive ablation study to verify the effectiveness of the two auxiliary tasks.", "reasons_to_reject": "I don’t find significant flaws in this paper. There are some minor suggestions:\n\n1. The VideoQA benchmarks in the paper are all choice-based. It would be better to choose some generation-based VideoQA datasets like ActivityNet-QA to increase the diversity.\n2. I believe the Flipped-QA is a general framework for various generative VideoQA models.  However, the authors only apply this framework to LLM-based models. It would be better to further verify the effectiveness and  universality to non-LLM-based models like HiTeA and InternVideo.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel framework called Flipped-VQA, which trains a VideoQA model with three objectives: (1) predict the answer A based on video V and question Q, (2) predict question Q based on V and A, and (3) predict video V from Q and A. The authors show that adding objectives (2) and (3) improves performance on the primary VideoQA task while also reducing linguistic bias. The framework is implemented in LLaMA-VQA, which outperforms both LLMs-based and non-LLMs-based models on five VideoQA benchmarks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some recent studies on Video-LLMs can be incorporated as related work:\n\n[1] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. Zhang et al.\n\n[2] VideoChat: Chat-Centric Video Understanding. Li et al.\n\n[3] M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. Li et al."}}, {"review_id": "XA1SERcmtu", "reviewer": "Reviewer_W2ex", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your strong acknowledgment of our paper. We will address your concerns and provide further implementation details in terms of reproducibility, hoping for more vigorous support for our paper.\n\n---\n\n**Response 1: Exploration of other video prediction methods.**\n\nWe explore two additional video prediction methods: video raw pixel generation and video feature generation. Both adopt L2 regression loss. In NExT-QA, Flipped-VQA with next frame prediction outperforms both raw pixel generation and feature generation, suggesting that the next frame prediction task is more beneficial for the temporal and causal understanding task.\n|  | Total |\n| --- | --- |\n| Raw pixel generation | 65.6 |\n| Feature generation | 71.7 |\n| **Next frame prediction (Ours)** | **72.4** |\n\n---\n\n**Response 2: Further implementation details.**\n\nFirst, the general input prompt of LLaMA-VQA is provided in Tab. 1 of the main paper. Specifically, the table represents the prompt for VQ → A, where the prompt from `[SOS]` to `Answer:` serves as a prefix and the model needs to generate the tokens after `Answer:`. Similarly, for VA → Q, `Video: ...`, `Choices: ...`, and `Answer: ...` tokens serve as prefix tokens and onwards is for a generation. The same manner is applied to QA → V.\n\nAlso, LLaMA-VQA 7B is composed of 32 heads and 32 layers and the feature dimension $D$ is 4096. The total number of vocabularies used is 32,000 from SentencePiece Tokenizer. For visual encodings, each frame is resized by 224 $\\times$ 224 and fed into CLIP VIT-L/14 to extract frame features. \n\nThe hyperparameters of our main experiments (Tab.2 of the main paper) are shown in the table below.\n|  | Learning Rate | # adapter tokens ($N_p$) | Batch Size | Weight Decay | Epoch | Warmup Epoch | # frames ($N_v$) | Max Sequence Length ($N_v + N_q + N_a$) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| NExT-QA | 0.0225 | 10 | 64 | 0.16 | 5 | 2 | 10 | 128 |\n| STAR | 0.0175 | 10 | 64 | 0.16 | 5 | 2 | 10 | 128 |\n| DramaQA | 0.0225 | 10 | 64 | 0.16 | 5 | 1 | 10 | 384 |\n| VLEP | 0.02 | 10 | 64 | 0.25 | 5 | 2 | 10 | 256 |\n| TQVA | 0.00875 | 10 | 32 | 0.02 | 5 | 2 | 10 | 512 |\n\nWe will include further implementation details in our final version and release our code if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Using LLMs for visual tasks is interesting and maybe an approach for more general artificial intelligence.\n2. The proposed framework Flipped-VQA is simple but effective, as shown in Table 4, every part is beneficial to the final performance.\n3. Extensive experiments have been conducted and the comparison between many existing methods have shown the superiority of their method.", "reasons_to_reject": "1. There maybe other video prediction (QA->V) methods, e.g., a real video generation head. More exploration on this task is better.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This paper proposes to utilize an LLM with a novel framework for video question answering.\n2. They propose Fipped-VQA with three tasks VQ->A, VA->Q and QA->V, which is novel and efficient.\n3. Extensive experiments have been conducted to verify the effectiveness of their methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are unanimous that paper is sound and exciting. There is consensus on the quality of the evaluation and the proposed Flipped-VQA method as a sample effient means to train model and help understand their causal and temporal reasoning.
Decision: Accept-Main