Answer Me if You Can: Debiasing Video Question Answering via Answering Unanswerable Questions
Authors: Dohwan Ko
Heeji Won
Jongha Kim
Miso CHOI
Byungseok Roh
Hyunwoo J. Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Video Question Answering
debiasing
causal inference

Abstract: Video Question Answering (VideoQA) is a task to predict a correct answer given a question-video pair. Recent studies have shown that most VideoQA models rely on spurious correlations induced by various biases when predicting an answer. For instance, VideoQA models tend to predict `two’ as an answer without considering the video if a question starts with ``How many’' since the majority of answers to such type of questions are `two’. In causal inference, such bias ($\textit{question type}$), which simultaneously affects the input $X$ ($\textit{How many...}$) and the answer $Y$ ($\textit{two}$), is referred to as a confounder $Z$ that hinders a model from learning the true relationship between the input and the answer. The effect of the confounders $Z$ can be removed with a causal intervention $P(Y|do(X))$ when $Z$ is observed. However, there exist many unobserved confounders affecting questions and videos, $\textit{e.g.}$, dataset bias induced by annotators who mainly focus on human activities and salient objects resulting in a spurious correlation between videos and questions. To address this problem, we propose a novel framework that learns unobserved confounders by capturing the bias using $\textit{unanswerable}$ questions, which refers to an artificially constructed VQA sample with a video and a question from two different samples, and leverages the confounders for debiasing a VQA model through causal intervention. We demonstrate that our confounders successfully capture the dataset bias by investigating which part in a video or question that confounders pay attention to. Our experiments on multiple VideoQA benchmark datasets show the effectiveness of the proposed debiasing framework, resulting in an even larger performance gap compared to biased models under the distribution shift.

Under review as a conference paper at ICLR 2023
ANSWER ME IF YOU CAN: DEBIASING VIDEO QUESTION ANSWERING VIA ANSWERING UNANSWERABLE QUESTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Video Question Answering (VideoQA) is task to predict correct answer given a question-video pair. Recent studies have shown that most VideoQA models rely on spurious correlations induced by various biases when predicting answer. For instance, VideoQA models tend to predict 'two' as an answer without considering the video if a question starts with "How many" since the majority of answers such type of questions are 'two'. In causal inference, such bias (question type). which simultaneously affects the input X (How many...) and the answer Y (two), is referred to as confounder Z that hinders a model from learning the true rela- tionship between the input and the answer. The effect of the confounders Z can be removed with causal intervention P(Y ((Y)) when Z is observed. How- ever, there exist many unobserved confounders affecting questions and videos, e.g., dataset bias induced by annotators who mainly focus on human activities and salient objects resulting in a spurious correlation between videos and questions. propose novel framework that learns unobserved
To address this problem, confounders by capturing the bias using wnanswerable questions, which refers to an artificially constructed VQA sample with video and question from two dif- ferent samples, and leverages the confounders for debiasing a VQA model through causal intervention. We demonstrate that our confounders successfully capture the dataset bias by investigating which part in video or question that confounders pay attention to. Our experiments on multiple VideoQA benchmark cltassets show the effectiveness of the proposed debiasing framework, resulting in an even larger performance gap compared to biased models under the distribution shift.
INTRODUCTION
Video Question Answering (VideoQA) task is multi-moda understanding task find the correct answer given a question-video pair, which requires an understanding of both vision and text modali- ties along with causal reasoning. However, recent studies (Ramakrishnan et al.) 2018 Cadene al.) 2019) point out that the success of the VideoQA models is due to its reliance on spurious corre- lations caused by bias instead of reasonable inference for answer prediction. In other words, the models concentrate on the co-occurrence between the question (or video) and the answer based on the dataset statistics and tend to simply predict the frequent answers. For instance, given a question that starts with "How many"" biased VideoQA model often blindly predicts 'two" as answer as depicted in Fig. 1b Fig. Ta faliutsststes the statistics of MSVD-QA dataset, showing that the majority of the answer to the "How many" questions are 'two'. In this case, 'question type' is acting as bias simultancously influencing the input question-video pair and the answer, which hinders the model from learning true relationship between the input and the answer.
In causal inference (Glymour et al. 2016), such variable, egg., question type, affecting both the input X and the answer Y is called confounder Z, which interrupts finding true causal relationship between X and Y. The causal intervention P(Y (d)))) intentionally cuts off the relation between X and Z via do-caleulus, which is also called 'deconfounding', remove the effect of the

---

Under review as a conference paper at ICLR 2023
founders Nevertheless, Z should be predefined to apply causal intervention but most confounders are unobserved in the dataset and hard to be applied to the causal intervention.
Therefore, introduce learnable confounder queries and train them capture the bias, and leverage the learned confounders for debias- ing through the causal intervention. To achieve this, we force the model answer the unanswerable question. An unanswerable question refers to an artificially constructed VQA sam- ple with video and question from two different samples mini-batch, along with an answer that corre- sponds to either the video or the ques- tion. When model answers the unanswerable question, the model in- evitably learns the bias of the specific modality that corresponds to the an- swer since another modality is ran- domly sampled and irrelevant to the answer.
8.38%
3.78%
3.24%
0.811%
Question How many boys standing in front of the burning car?
two
three five
Answer
four Baseline
four
eight
(a)
(b)
Figure I: Dataset statistics MSVD dataset and an example of the biased answer. (Left) The majority answers to the "How many" questions are 'two'. (Right) The model outputs the biased answer 'two' instead of the right answer 'four'.
To summarize, we propose novel framework Debiasing a Video Question Answering Model by Answering Unanswerable Questions (VoidQ) with causal inference. In order apply the causal intervention P(Y (d))),, we introduce learnable confounder queries. The proposed confounder queries are trained to capture the bias by answering the unanswerable questions. Our framework leverages the confounder queries and their outputs to debias our VQA model via causal interven- tion. We validate our models on three benchmark VideoQA datasets (TGIF-QA, MSRVTT-QA, and MSVD-QA) and demonstrate the effectiveness of our debiasing strategy. Also, ablation studies reveal that the performance gap between conventional biased models and the proposed model gets larger when the training and test distribution significantly differ, supporting the improved general- ization ability by the proposed approach. Lastly, visualization of confounders via our variant of Grad-CAM shows that the learned confounder queries adequately debias VQA model by taking into account falsely correlated keywords questions or salient regions in videos.
To up, our contributions
follows:
We propose novel debiasing framework for VideoQA model to predict correct answers based on causal inference by removing the effect of the confounders.
We also present training scheme encouraging the learnable confounder queries to capture the bias by forcing the model to answer the unanswerable questions.
Our extensive experiments demonstrate that the proposed framework outperforms previous models on various benchmark datasets, even with larger margin under the distribution shift where the biased models suffer significant performance degradation from.
We verify that our confounders successfully capture the dataset bias by investigating which parts in video or which words in question are utilized by confounder queries correct the predictions of a VQA model.
2 RELATED WORK
Video Question Answering (VideoQA). VideoQA task to infer correct answer, given video and a question. While models for the VisualQA task focuses on spatial information of an image (Antol al.) 2015) Yang al., 2016). VideoQA requires reasoning over both temporal and spatial dynamics, making it a more challenging task. Previous works have applied spatio-temporal
contextual attention various scenarios (Jang et al. 2017: Xiao et al. 2022. Z.ho et al. 2017).
Additional descriptions about the causal inference in Sec. 3.1
2

---

Under review as a conference paper at ICLR 2023
Another line of research has proposed the end-to-end pretrained models on the large-scale dataset to improve the performance of various downstream tasks including the VideoQA. Fu et al. (2021)
builds additional cross-modal encoder well as the video and text encoder and Wang et al. (2022)
introduce a token rolling operation to efficiently perform the temporal attention on the cross-modal encoder. However, existing models still suffer from dataset bias (Ramakrishnan et al. 2018 Cadene itt al., 2019). Therefore, in this paper, we propose a debiasing framework for VideoQA to improve generalization power by reducing the dataset bias even under distribution shift.
Debiasing from the biased model. The first approach to alleviate the bias is to directly augment the dataset to remove statistical "hints' and enlarge the size and diversity of the training set. [Gokhale] et al. (2020), Chen et al.] (2020), and Kil et (202T) propose to augment input images or questions to generate counterfactual or paraphrased QA pairs. On the other hand, there exist attempts to learn the text or image bias through additional branches by training the branch only with a single modality. Outputs from the biased branches are then utilized to debias training (Ramakrishnan et al. 2018 ZZhng et al.) 2021; Cadene et (019). However, these approaches require manually designed heuristic rules or separate branches to capture bias from a specific modality. We propose a novel unified debiasing framework that leverages the unanswerable questions, allowing the learnable confounder queries to capture the biases related to both modalities without any heuristics.
Causal Inference. Causal inference, a method to find the true effect of a particular variable target variable without being interrupted by any other variables, is being widely adopted in Visual QA tasks. Previous works proposed augmenting the data to remove unwanted effects of a specific variable by utilizing the Structural Causal Model (SCM) (Glymour et al., 2016) which defines causal relationships between variables. Specifically, existing works generate counterfactual samples Tang
let al. |020: Abbasnejadd et al. [020;] Yue et al.] (021) or negative samples (Wen et al. 2021 Teney
al. 2020 measure and remove an effect of specific confounding variable. Besides, causal intervention is also used to directly remove the effects of a predefined confounding variable that hinders proper reasoning. Unfortunately, since the confounders are usually unobserved, most ex- isting methods manually predefine the confounder sets as object classes (Zhang et al., (020) or verb-centered relation tuples from caption data (Nan al., 202T) in order conduct the causal intervention. Unlike these works, we directly train the confounder queries so that they can capture various types of bias, instead of manually predefining what confounder should be.
3 METHOD
Bias misleads the model to become reliant on spurious correlations, resulting in poor generalization ability. Therefore, in this section, we propose a novel framework Debiasing a Video Question Answering Model by Answering Unanswerable Questions (VoidQ) with causal inference. Firstly, we briefly revisit the basic concepts of the VideoQA and causal inference. We then present the debiasing framework with learnable confounder queries based on the causal intervention. Finally, we introduce the training objective using unanswerable questions to let the confounder queries learn the bias.
3.1 PRELIMINARIES
VideoQA. VideoQA is a task to predict the answer Ý given a question-video pair X = (Iig. Te). There are two types of tasks in the VideoQA: multi-choice question answering (MCQA) and open- ended question answering (OEQA). For the MCQA, the model predicts the answer among five options in general. Each option concatenated with the question and the model calculates the similarities between each concatenated text and the video to output the final prediction. In the OEQA setting, the task is mostly converted to the classification task to predict the correct answer among the predefined global vocab-set containing all the candidate answers. For simplicity, we will explain concepts only with OEQA. Further details including MCQA are in the supplement. The prediction Y under the OEQA setting given a pair X = (Igg I.) can be written as:
Ý = P(Y))) = Mx((x)),
(1)
where f is feature encoder and h is classifier.
3

---

Under review as a conference paper at ICLR 2023
f
Answer
FFN
h
FRV byy
sway?
P(
Inee Fil contraa
Stord
Question
tolkeiize
data encoder f
confounder encoder 9
Input Video
X
Igg
T.
Confounder
Figure 3: VoidQ Architecture. We construct tokenized text tokens Zg and patchified video tokens By from input question and video, then concatenate them to form and feed them into the data encoder f. Learnable confounder queries Z are fed to the confounder encoder which are cots-dt with output features of f. dataset bias by minimizing Loonfonder between the ground truth and biased prediction y, generated from Z through hg. Causal intervention utilizing learned confounders Z and 2 is applied to generate the final debiased prediction Y, from X through ht.
Le., X. trained to learn
Causal Inference. To train and h in Eq. most previous approaches (Fan et al. [2019] Gao
Z
Z
2018, Jiang al.) 2020 Jiang Han Le et al. 2020) have adopted the standard cross entropy (CE) loss as:
al.
2020
X
Y
X
Y
min CE(Ý, Y). f.h
Eq. aims simply to minimize CE between the ground-truth Y and the prediction Y. in which and & naturally learn the spurious correlation between X and Y.
(a) X
(b) P(Y do()))
Figure
Causal Inference. X cause, Y effect, and a set confounders.
To address this problem, some recent works (Tang et al. 2020 Zhang et al. 2020: Niu et al. tried applying causal inference (Glymour et al. 2016) to alleviate the bias. As shown in Fig. 2a conventional approaches have calculated the likelihood P(Y to predict the answer as:
2021
X(Y Σ P(Y(x.)))((xX))
2EZ
On the other hand, the predicted answer with causal intervention P(Y(do(X)) where the connection between X and is cut-off is defined as:
P(Y
Σ 2) ZEZ
(4)
Unlike in Eq. the causal intervention removes the relation between X and Z, allowing the model to reason about the real causation of X to Y by considering the prior P(z) instead of P(=[X) in Eq. Note that set of confounders Z must be known in advance to calculate Eq.
3.2 OVERALL ARCHITECTURE
VoidQ consists of two encoders. A data encoder f follows the Transformer (Vaswani et al. 2017 encoder based on the self-attention mechanism. Unlike f, a confounder encoder g is composed of cross-attention layer followed by feed-forward networks (FFN). For the input of f. we concatenate text tokens F and visual tokens F., iee,, X = (xgg x.,) E NDD where N the number of input tokens and is the feature dimension. Then, the output feature can be calculated as:
X f(X) NXD
(5)
4

---

Under review as a conference paper at ICLR 2023
X
Cansal
What the doing?
(u
Y
Y
g
GCE
(v,0)
GCE Y
connounndr
X
(1.9)
Y
causal
Y
+++
What is diving in?
GCE
CE
Y
GCE
X
Figure 4: Objective functions with unanswerable questions. The first letter in the superseripts of the four outputs Y (+,))) from stands for the modality of an input from the pair X, ie.. p(a+)) and Y denote outputs from given input X, = 2%)) and X, (1 Lo), respectively. The second letter denotes the modality an output is biased towards, which is also represented in color. (1-e) and {((,0) denotes that the output are biased towards text and video, respectively.
For unobserved confounders, we additionally introduce set of learnable confounder queries Z E xMMD as an input g. where M is the number of confounders and is the feature dimension. We also add two different modality encodings to inject information about each token's 's modality so that they could learn modality-speciin bias. Concretely, a text-type encoding and a video-type encoding are added to individual confounder queries Z(0 M/2] and M[M/2: : MM] The output of f, i.e. X, also used as an input of g and cross-attended with Z. detail, Z is adopted as query, and X is used as key and value the encoder Then, the output Ž can be written as:
Ż = {2| g(Xx2)) V2 E 2) E MxD.
(6)
We additionally introduce two FFN h and hg as prediction heads of f and g, respectively. In short, the data encoder f and h perform the main VideoQA task, while the confounder encoder and hg learn and encode the bias confounder queries Z, which will later be removed via causal intervention. The detailed objective functions train each component introduced in Sec. 3.3
Fig. 3 Billustrates the overall architecture of our proposed framework.
3.3 TRAINING OBJECTIVE
Debiased prediction. Conventional approaches used P(Y[X) in Eq. 3 as the output logit which simply calculated with an additional FFN à on the top the encoder f. i.e., Ý = X) = h(f())) On the other hand, to remove the effeet of confounders, with causal intervention. Since P(Y]do(X)) is calculated with the Softmax function, can be approximated by Normalized Weighted Geometric Mean (NWGM) (Xu et al. 2015) follows:
logit P(Y (()))
y = P(Y do(X)) = Σ P(Y)),))((2) R P(Y)
Σ (X + 2))()))
zEZ
ZE2
Then, the debiased output y, in Eq, can be calculated with FFN hy:
P(Y) ((x+2)P(2)) h 2E%
Σ
x (x+ + + o(x, p(z) hy Σ (x++2 (x+2+2) P(z)
+E7
(8)
Note that CLS token from X is used when calculating X 2 + in Eq. 8 In general, confounders are defined to be data-agnostic, i.e., each sample in a dataset shares the same confounders. However, Eq. g consider not only data-agnostic confounders
where X = ((X) defined in Eq.
Z but also data-modulated confounders Z. While debiasing the data-agnostic confounders leads the model to alleviate dataset bias, by introducing additional data-modull modulated confounders Z, also mitigate sampee spurious correlations, e.g., models tend to select the option which includes the visually salient 'object' in the video as an answer. We then apply the standard CE loss to perform the VideoQA task by using debiased logit with causal inference as:
Leansal CB(Ý, Y).
5

---

Under review as a conference paper at ICLR 2023
Training confounders with unanswerable questions. To alleviate the effect of confounders as in 7 it is important to let the confounder queries Z E 2x x capture the bias during training. To achieve this, we first construct two unanswerable questions i.e., X, (Zy x1,) and X., (x F.) by pairing text Eg and video 2i from sample X with label Y with another video x'g and text oy from a different sample X' in mini-batch having label Y'. We then force the model to predict Y or Y from these unanswerable questions. Since the model is unable to predict the proper answer given the unanswerable pair by relying only on the single modality but is forced to do so, the model inevitably learns text or video bias. Therefore, the model gets to only consider the spurious correlations to predict an answer. Fig. shows the loss functiion train confounders with unanswerable questions.
In detail, two unanswerable pairs are forwarded to the encoder f and with the confounders Z as:
Xqi X2 f(X)), f(X,)
(10)
Zg. 2. { g(x, 4 2). V2 E 2}. { = g(Xu, 2), Z}.
where Zy. %. E MRDD In Eq. 10 7g and Z., indicate that output features of confounders Z, which are cross-attended with unanswerable pairs X, and X., respectively. As mentioned above, since the confounder is divided into two part to learn the text and video bias, we feed both 2, and 2.,. being separated into two parts respectively, to the FFN layer hg to output modality-biased predictions as.
Zg Zu.y Zgwor Zu = 2,(0:M/27 2o10: M M/2). 2 : M), [o[M/2 M]
(11)
fr(4.)), A((aa) f((a"), (f hy Z4.)) hg( hg (Z,,)) kg(Zo.))
Here, the former letter in the superscript of )-( denotes the input modality which comes from the original pair X when constructing the unanswerable pair (e.g., An input question of ((a,*) is taken from X,) and the latter denotes the modality an output would be biased towards. For instance, (-(0.4) an output of y given an input X, = (xrg: T.)), is desired to be text-biased, while ((,,)) is also an output of g given an input X, = (+ x.)), but desired to be video-biased. In other words, ((m.a)) and f7(*.9) denote text bias and a text-biased output from g. Similarly, Z(,,0) and (/(,,) denote video bias and video-biased output from g.
g
Then, the loss function for training confounders to satisfy properties mentioned above is as follows: Coonfounder = -GCE(Ý Y) GCE(Ý Y") + GCE() Y') + GCE() Y), (12)
where GCE is the Generalized Cross Entropy (Zhang & Sabuncu 2018) loss which will be further discussed below. For GCE(), (q.4) given an input Xg = (Izg, x,),,, i.e., is forced to learn the text bias regardless of the video, since the question F. and the answer Y are from the same pair, while the irrelevant video z' is from another
(a.)) YY in Eq. 12
(14-4) is text-biased output desired to match Y
Y', where the input corresponding to ( +()) is encouraged learn the video bias because xi, and Y' are from the same pair. We can therefore train the confounders to learn both text bias and video bias with unanswerable questions in Eq. [2]
sample in mini-batch. In the same way, GCE( g
We can also amplify the bias and train the confounders to be more bias-toward by adopting GCE as:
GCE(p(z;0)..) = 1- (2; f)T
(13)
where p(z: f) is an output probability parameterized by 6, q E (0. 1] is smoothing parameter, and is a ground truth. The gradient of GCE loss is Py (a:; 8yd times larger than the gradient of the standard
CE loss, iee., 8GCF
20 (2; 0)9. OCE Therefore, GCE loss leads the model to be biased by placing
"0
larger weight on 'easier' samples with high confidence score (Lee et al. 2021 Nam et al. 2020). inducing the model to be overfitted easy shortcuts. Further details of GCE are in the supplement.
Then, the final loss function of our proposed algorithm is as follows:
L Leausal + Levafounder-
(14)
At the inference phase, we only use y,, the output of causal intervention for the prediction.
the first M/2 confounders 7(0 : M/2) denotes comfoundersss and the latter M/2 confounders M|/ M] means video confounders

---

Under review as a conference paper at ICLR 2023
Table I: Comparison on TGIF-QA, MSVD-QA, and MSRVTT-QA. We report the accuracy for all datascts.
TGIF-Action and TGIF-Transition MCQA and TGIF-Frame, MSVD-QA, and MSRVTT-QA OEQA.
TGIF-OL QA
MSVD-OO QA
MSRVTT-QA
Method
Action (MC)
Transition (MC)
Frame (OE)
OF
OB
ST-VQA (Jang et al.
2017
62.9
69.4
49.5
Co-Mem Gao et al. 2018
68.2
74,3
51.5
31.7
31.9
HCRN (Le al. 2020)
75.0
81.4
55.9
36.1
35.6
HGA (Jing & Han 2020
75.4
81.0
55.1
34.7
35.5
QueST (hang et al 2020
75.9
81.0
59.7
36.1
34.6
Bridge2Answer (Park al.
2021
75.9
82.6
57.5
37.2
36.9
ClipBERT (Le
2021
82.9
87.5
59.4
37.4
VIOLET (Fu
92.5
95.7
68.9
43.1
MASN Seo
84.4
87.4
59.5
38.0
35.2
QESAL Liu el al.
202T
76.1
82.0
$7.8
36.6
36.7
CoMVT Seo
2021b
39.5
42.6
IGV (Li ct
2022
40.8
38.3
HD-VILA Xue ct al.
2022
84.3
90.0
60.5
40.0
HRNAT Gao et al.
2022
38.2
35.3
CASSG Ju
2022b
77.6
83.7
58.7
36.5
36.1
HGQA Xiao
2022
76.9
85.6
61.3
38.6
CMCIR
2022a
78.1
82.4
62.3
43.7
38.9
Mao
2022
84.6
90.1
62.5
41.6
VoidQ (Ours)
93.5
97.3
67.0
46.2
43.2
Prior probability P(z). As for the prior probability P(=). we introduce a learnable parameter E Rd ie., P(z) Softmax(c). However, since large variance of P(=) can make the training unstable, we apply exponential moving average (EMA) on P(z) to stabilize the training. We also apply Dropout (Srivastava et al. 2014) P(z) and regularize model from being overfitted on the particular confounders.
4 EXPERIMENTS
In this section, we evaluate the performance VoidQ for both MCQA and OEQA settings on three benchmark VideoQA datasets: TGIF-QA, MSRVTT-QA, and MSVD-QA. In TGIF-QA, TGIF- Action and among five options. For OEQA on TTGIF-Framc MSRVTT-QA, and MSVD-QA, we follow the con- ventional settings (Fu et al., 2021] Lei et al.) 202T) construct the answer candidates. detail, the answer candidates of TGIF-
are conducted under the MCQA setting, predicting the proper answer
consist of 1.540 most frequent answers in the training set. Simi- larly, 1,500 and 1,000 most frequent answers the training set are selected as the answer candidates for MSRVTT-QA, and MSVD-QA datasets. We also perform ablation studies to show that VoidQ is robust and generalizes well under distribution shifts. Our extensive qualitative analyses demonstrate that learnable confounder queries successfully capture the dataset bias, illustrated by our variant of Grad-CAM. Descriptions of datasets and implementation details are in the supplement.
4.1 QUANTITATIVE RESULTS
TGIF-QA, MSVD-QA, and MSRYTT-QA. We compare VoidQ with previous VideoQA meth- ods in Tab. I On TGIF-QA, VoidQ outperforms ClipBERT and VIOLET, which are foundation models pretrained on the large-scale dataset, especially by margin of 1.0% TGIF-Action, and 1.6% on TGIF-Frame compared to VIOLET. The performance VoidQ also improves by 0.6% on MSRVTT-QA compared to CoMVT which is specifically designed for VideoQA tasks. On MSVD- QA, VoidQ obtains a 2.5% improvement over CMCIR which also conducts causal intervention.
Ablation studies. Tab. demonstrates the ablation studies in terms of the three components: con- founder encoder unanswerable questions, and GCE loss. Without the unanswerable questions to train confounders, adding a confounder encoder degrades the performance from (a) 43.5% (b) 41.9%. This result also evidences that the performance gain of VoidQ is not solely from increasing the model complexity. On the other hand, row (c) shows that the unanswerable questions signif- icantly improve performance by margin of 3.2% compared to (b). Adopting GCE loss (d) also leads to further improvement in performance, since GCE loss helps the model better learn the confounder Z by amplifying the dataset bias.

---

Under review as a conference paper at ICLR 2023
Ablation studies on MSVD. '-" at g. the con- founder encoder, denotes that we do not conduct the causal intervention and use the conventional likelihond P(YIX) (X) for the prediction. UQ denotes the unanswerable questions with the standard CE loss. 'v' on both UQ and GCE stands for Leonfoundrr
Table 2:
A (%)
UQ
GCE
MSVD
MSRVTT--MSVD
0.3
0.5
JSD
(a)
43.5
36.3
Figure 5: Distribution shift. X-axis: JSD between the train and modified test sets. Y- axis: Accuracy difference of the haseline and ours.
(b)
41.9
(c)
45.1
(d)
46.2
41.0
Q: How many men are dancing?
Q: What is a person giving to baby pandas?
P()
P(Y[X)
animal
milk 6.00
three 0.02
two
0.46
four
hole in
food 0.05
five 0.03
video 0.00
kitchen 0.00
toy: 0.03
toy: 0.05
(a)
(b)
Q: What is a person doing?
Q: Where do a large group of people run?
P(YIX)
P(Y)d)(X))
P(Y[X)
P(Y (do())
play 6.42
skd 0.20
side
0.42
stage
0.64
jump
trick
field
Field
fall 0.05
perform 0.06
park: 0.04
0.05
stand 0.68
walk 9.08
stage: 0.02
silhouette 0.05
pasture
place
(c)
(d)
Figure 6: Qualitative results on MSVD. Confidence scores of the top-5 predicted answers using conventional likelihond P and causal intervention ((y]) Ground-truth answers are colored in red
To show the generalizability of VoidQ. we also conduct experiments under the distribu- tion/domain shift; we trained models on the MSRVTT training set and evaluated them on the MSVD test set (MSRVTT-MSVD)), Jensen-Shannon Divergence, i.e., JSD(P, Q)
(DKu(PlIB) + (Q)) where R = (P + Q). is adopted to quantify the label distribu- tion distance between training and test sets. We observe that VoidQ provides larger perfor- mance gain when the distribution shift increases. Where both training and evaluating MSVD, ISD between the train and test set shows the relatively small value of ISD(trainMSVD: (EstMSVD) 0.07. VoidQ obtains 2.7% improvement from (a) 43.5% to (d) 46.2% in such setting. Whereas, the improvement increases to 4.7% when training on MSRVTT but evaluating on MSVD, where (IaaiininSTTYTT: les|MSVD) 0.26.
We conduct additional experiments comparing model performances on the new test sets, including the standard test set, constructed by intentionally removing samples from the test set if the answer belongs in the top-1, 10, 20, 50, and 100 most frequent answer candidates. Such modification demonstrates how much model is statistically biased. If a model is highly biased towards the dataset statistics, the model would perform worse when the frequent answers are removed. We com- pare the performance of VoidQ against the base VideoQA model without any debiasing scheme, which corresponds to (d) and (a) in Tab. 2 respectively. Fig. [S) illustrates the results of such ex- periments. The performance gap between the base model and the proposed model increases as the discrepancy between the train and test set enlarges. The performance gap of 2.7% between VoidQ and the baseline when JSD 0.08 dramatically increases up to 6.79% when JSD is a larger value, 0.61. Two experiments done under distribution shift prove that VoidQ well alleviates the statistical bias, helping the model to successfully perform on the dataset that differs from the train set.
4.2 QUALITATIVE ANALYSES
How is the model debiased? Four examples in Fig. 6 illustrate how models' predictions are corrected via causal intervention P(Y [do(X)). Without the causal intervention, i.e, X(Y
(x())) ). the model is prone to answer questions based on a single modality. For instance, in Fig. 6a| X( predicts 'two' as the answer since about 84% answers to the 'How many' ques- tions in the training set are 'two' without considering an input video. Another example in Fig. 6b
P(Y X) predicts 'animal' or 'panda" answer, focusing on visually salient objects, overlook-
8

---

Under review as a conference paper at ICLR 2023
(xx2)) 87
V vix;2)
Vi
How
many
men are
How many men are
What is a person
dancing
dancing
giving to baby pandas
giving baby pandas
Debiased Prediction three
Debiased Prediction :mll milk
(b)
Figure 7: GradCAM on MSVD. visualize ((x,)) VJ and V Vxx; ff on two samples to show where confounders look in the imput.
ing the input question. However, after conducting the causal intervention P(Y (o))))) output corrected to 'three" and 'milk' by thoroughly considering the previously neglected video and text input, respectively. VoidQ also succeeds predicting the answer 'ski' in Fig. [c| although the answer 'play' appeared 51 times more than 'ski' during training. Finally, Fig. shows that VoidQ plau- sibly predicts 'stage", overcoming the dataset bias that 'side' and 'field" are the two most frequent answers to the 'Where' questions.
Where confounders look at? investigate which parts in a video or which words in question taken into account by confounder queries to debias the predictions of VideoQA model. To consider the gradient flows through Z, modify GradCAM and Counterfactual Grad- CAM Selvaraju al. 2017) denoted ((x:2)y and V V respectively. They are computed
OÝ!
0E
a
0
(xx:2 ReLU
V (x ReLU
(15)
05
0X
03
0X
8E2
JEE
where / is the target label in question. V
(x;2) illustrates where confounder ž focused on to bol-
ster correct predictions. Conversely, V
(x.Z) Y reveals where confounder Ž focused on to suppress
falsely correlated cues that cause bias. Fig. illustrates V 2x V and (ix22 V on two examples
shown in Fig- 6a and 6b which contains text and video bias, respectively. Fig. 77a (left) reveals that "How many" is strongly highlighted by V (xx2)) V/. implying the phrase negatively influenced to predict a correct answer. Interestingly, it matches ered as the confounder leading a model to predict 'two' as the answer. On the other hand, V ((x:2) Y highlights "dancing" most, meaning that VoidQ focuses the right context to output the number of 'dancing men'. Similarly in Fig. 7b including video bias, Ve (xx2) Vj and V V(x.2) V focus the 'panda' and 'milk' respectively, which matches our notion that looking at 'panda" in the video hinders the model predict correct answer looking at 'milk' in the video helps debiasing.
assumption that "How many" can be consid-
CONCLUSION
In this work, we propose a novel debiasing framework for VideoQA, dubbed VoidQ, which trains confounder queries by answering unanswerable questions and utilizes the trained confounders to remove the dataset bias via causal intervention. Concretely, we adopt causal intervention to cut-off the relation between confounders Z and input X so that the model predicts the correct answer Y with bias removed. Since the causal intervention is not applicable when confounders are unobserved, we additionally introduce a training scheme that leverages unanswerable questions to let learnable confounder queries capture the dataset bias. We demonstrate the effectiveness of our method by validating the proposed architecture on various benchmark datasets, and provide qualitative analyses showing that confounders well learned to capture the dataset bias and properly removed.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ehsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi, and Anton van den Hengel. Coun- terfactual vision and language learning. In CVPR, 2020.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. In NeurIPS, 2019.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training recognize long-tail visual concepts. In CVPR, 2021.
David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In ACI., 2011.
Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question answering. In CVPR, 2020.
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Hetero- geneous memory enhanced multimodal attention model for video question answering. In CVPR, 2019.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Vi- olet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 2021.
Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In CVPR, 2018
Lianli Gao, Yu Lei, Pengpeng Zeng, Jingkuan Song, Meng Wang, and Heng Tao Shen. Hierarchical representation network with auxiliary tasks for video captioning and video question answering. IEEE TIP, 2022.
Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Mutant: A training paradigm for out-of-distributtinn generalization visual question answering. In EMNLP, 2020.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR. 2017.
Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. Divide and conquer: Question-
guided spatio-temporal contextual attention for video question answering. In AAAI, 2020.
Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In AAAI, 2020.
Jihyung Kil, Cheng Zhang, Dong Xuan, and Wei-Lun Chao. Discovering the unknown knowns: Turning implicit knowledge in the dataset into explicit training examples for visual question swering. In EMNLP, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 1JCV, 2017.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In CVPR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. In NewrIPS, 2021.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for vidoo-and-lagguage learning via sparse sampling. CVPR, 2021.
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, 2022.
Yuncheng Li, Yale Song, Liangliang Cao. Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. Tgif: A new dataset and benchmark on animated gif description. In CVPR, 2016.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Fei Liu, Jing Liu, Richang Hong, and Hanqing Lu. Question-guided erasing-based spatiotemporal attention learning for video question answering. Neural Networks and Learning Systems, 2021.
Yang Liu, Guanbin Li, and Liang Lin. Cross-modal causal relational reasoning for event-level visual question answering. arXiv preprint wwXiv:220..12644 2022a.
Yun Liu, Xiaoming Zhang, Feiran Huang, Bo Zhang, and Zhoujun Li. Cross-attentional spatio- temporal semantic graph networks for video question answering. IEEE TIP, 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17X1v:1711.05101, 2017.
Jianguo Mao, Wenbin Jiang, Xiangdong Wang, Zhifan Feng, Yajuan Lyu, Hong Liu, and Yong Zhu. Dynamic multistep reasoning based video scene graph for video question answering. In NAACL, 2022.
Antoine Micch, Dimitri Zhukov, nann-Bapistte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef' Sivie. Howto100m Learning text-video g by watching hundred million narrated video clips. In ICCV, 2019.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. NeurlPS, 2020.
Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu. Interventional video grounding with dual contrastive learning. In CVPR, 2021.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counter- factual vqa: A cause-effect look at language bias. In CVPR, 2021.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 1m2text: Describing images using million captioned photographs. In NeurIPS, 2011.
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Bridge to answer: Structure-aware graph interac- tion network for video question answering. In CVPR, 2021.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in visual question answering with adversarial regularization. In NewrIPS, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In ICCV, 2017.
Ahjeong Seo, Gi-Cheon Kang, Joonhan Park, and Byoung-Tai Zhang. Attend what you need: motion-appearance synergistic networks for video question answering. IJCNLP, 2021a.
Paul Hongsuck Sco, Arsha Nagrani, and Cordelia Schmid. Look before you speak: Visually textualized utterances. In CVPR, 2021b.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
11

---

Under review as a conference paper at ICLR 2023
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, 2020.
Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton Van Den Hengel. On the value of dittr--diitribttion testing: An example of goodhart's law. In NeurlPS, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in onc: Exploring unified video-language pre- training. arXiv preprint XXiv:2203.07300 2022.
Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. Debiased visual question answering from feature and sample perspectives. In NeurlPS, 2021.
Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video conditional graph hierarchy for multi multi-granulat question answering. In AAAI, 2022.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACMMM, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun. Bei Liu. Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, 2022.
Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks. In CVPR, 2021.
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016.
Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero- shot and open-set visual recognition. In CVPR, 2021.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In NeurIPS, 2021.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal intervention for weakly-supervised semantic segmentation. In NeurlPS, 2020.
Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Towards debiasing temporal sentence grounding in video. arXiv preprint arXiv:2111.0433 2021.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NewrIPS, 2018
Zhou Zhao, Jinghao Lin, Xinghua Jiang, Deng Cai, Xiaofei He, and Yueting Zhuang. Video question answering via hierarchical dual-level attention network learning. In ACMMM, 2017.
12

---

Under review as a conference paper at ICLR 2023
Z
X
(a) Structural Causal Model
(b) The definition of do
Figure Illustration of Structural Causal Model (SCM) do-calculus definition.
A APPENDIX
B ADDITIONAL PRELIMINARIES
B.I CAUSAL INFERENCE
Structural Causal Model (SCM). SCM is statistical model representing the causal relationship between variables in the graph structure Glymour al. (2016). In the causal graph, each variable is denoted by nodes, and 'causation' between two different variables is denoted by directed edge between nodes. Fig. Salillustrates an example of SCM representation in graph form. The edge X Y in the graph implies that X is the 'cause" of Y. Also, Z in the graph represents 'confounder', which simultaneously affects both X and Y. therefore making it difficult to find a true effect of X on Y. Such confounder induces the spurious correlation between X and Y through the backdoor path between and Y. A backdoor path formally defined as any path from to Y that starts with an arrow pointing X (Yang et al.) ) such as X <- Z Y in Sa To find out the true causal relationship between X and Y, the causal intervention with do-calculus P(Y]do(X)) )) is applied to cut-off the relationship Z - X, as illustrated in 8b] therefore removing the spurious correlation induced by Z. Backdoor adjustment is a widely adopted approach to deconfound the effect of the confounders using the do-calculus, which we further concretize in the very following section.
The backdoor adjustment. Given a directed acyclic graph consisting of X, Y, and Z as &a backdoor adjustment can be applied to reveal the true causal effect of the X on Y given the con- founder 7. By Bayes' theorem, X) can be expressed as follows:
P( = Σ = x|X).
(16)
=EZ
The causal intervention with do-calculus P(Y]do(X)) mentioned in the previous section is then formally defined as below:
P(Y)do(X)) Σ P =EZ
(17)
Through the backdoor adjustment, the true causal relationship between X and Y, which is denoted as P(Y| do) is measured without any effect of the confounder Z.
Normalized Weighted Geometric Mean (NWGM). To approximate P(Y ((Y))).. we use
NWGM. Before dealing with NWGM, we first revisit the definition of Weighted Geometric Mean (WGM). Given a discrete variable X and its distribution P(X), the expectation of f(x) is defined as:
Ex [x())]
P())
(18)
BEX
The Weighted Geometric Mean (WGM), an approximation of E, [x())] defined as follows:
WGM(f(>) II f())
(19)
#EX
13

---

Under review as a conference paper at ICLR 2023
If the activation function of f(z) is composition of a function g(x) followed by exponential function, i.e., f(z) = exp(g(2) (z)). Eq. 19 can be reformulated as:
WGM(
I explo(>))
II explg(x)P(x)) =eX
(20)
expl Σ {()) exp LE و [())]}
Interpreting WGM in the perspective deep learning, f(z) can be regarded as a neural network whose last activation function is the softmax function. Therefore, Xu et al. (2015) and Yang et al.
2021 approximate the expectation the f(x) using the WGM as follows:
Eg [f()]] (GM(f(2)) (xp
(21)
To guarantee that output logits can be interpreted as probability, NWGM, normalized version of WGM, applied so that the sum of output logits adds up to one, and it formally defined as:
NWGM((<x) = II. exp(o(x))
IL exp(g(x))
(E. [xx))
(22)
Σ, exp(1 (x))())))
= Softmax (F. (Frr(((r))
Adopting the WGM defined above to our model, P [X)) can be approximated as below, where P(Y X, Sof (m(a( x exp(s(X, =)): P(Y]do(X))
E. Ex/eex(g((X, xX),
exp( (g(,,,oo(XX11,00 x)])
(23)
expl Σ (J(X) +2
sEZ
where 2(X.2) f(X) + + 2. Then, we apply NWGM to normalize Eq.
23 as to get final
deconfounded prediction probabilities P(Y (o(())()) as follows:
P(Y]do(X)) Softmax (E_ (x(,,))))
Softmax (f(X) + 2)
(24)
=EZ
B.2 GENERALIZED CROSS ENTROPY (GCE) LOSS
GCE. GCE loss was first proposed as a generalized loss taking advantage of both Mean Absolute Error (MAE) loss, and Categorical Cross Entropy (CCE) loss by Zhang & Sabuncu (2018). Given an input æ, the ground truth one-hot vector y, and the set of parameters 0 of the classifier f. MAE and CCE loss are formally defined as below the common case where the softmax is followed by the classification layer:
(MAE(f(I;0), 0), = 11v f(x; 0)1l1
(25)
(CCE(((x;
U/ log [y(2; 0),
where denotes the number of target classes, Yj and fj denote the j-th element of y and the j-th prediction of f. The gradient of loss functions with respect to parameter 0 is as follows:
0 M AE (J 0). (z;
00
(26)
(f 0),u)
f, Volu(E;0),
09
14

---

means how much affects the prediction 8j and 02 ax means what the confounder Ž looks at in the input X, so V (xx22 8J indicates Under review as a conference paper at ICLR 2023
where fy denotes the element of the output logit corresponding to the ground-truth label. As for- mulated in Eq. 26 CCE emphasizes samples with larger 1/fy (z; f), or smaller (2; On the contrary, MAE equally treats every sample with the same weight. The fact that MAE does not place a larger weight on difficult samples makes MAE robust to noisy labels, but it also makes training difficult since every sample is treated equally so that challenging examples are not learned enough. In contrast, optimizing a model using CCE is easier due to larger weights being given to challenging samples. However, CCE is sensitive noisy labels, since the model could easily be overfitted to such noisy samples which are intrinsically difficult due to label noise. Then GCE loss can be viewed as generalization between MAE and CCE loss, and is formally defined as below:
Cf y) 1 - (x 0)0
(27)
q
where E [0, 1] is a smoothing parameter. The gradient of LaCE with respect to 0 is as follows: ÖLoce(f(r; (x:;)).)) =f f,(
fz(2;0 Vofu(z; = fu 0)2 dCCCE
00
00
(28)
= Vall(())) sy<; 0LMAE
09
Therefore, LoCE additionally weights each sample by [,; 0) times compared to CCE loss, weighting difficult samples less. Also, weights each sample by fa ((:;))=- times compared to MAE loss, giving larger weight to difficult examples compared to MAE loss. If q is properly cho-
sen, GCE can therefore act as generalized loss that is more robust than CCE and easier to train than MAE, achieving balanced trade-off between two losses.
GCE Loss in Computer Vision. By the fact that GCE loss gives smaller weights 'difficult' examples compared to conventional CCE loss, Lee et al. (021) and [Nam et al. (2020) propose capturing bias in the model by leveraging GCE loss to train 'biased network', which is overfitted to easy samples, which corresponds to "bias" or "spurious correlation' existing in the dataset. Both works train the model with GCE loss to achieve the model to be biased by focusing on the "casier" samples compared to the conventional CCE.
B.3 GRADCAM
The standard GradCAM Selvaraju et al. calculated as:
2017) of prediction f with respect to input X can be
ori 0x
O 0x ox
ov:
ož
V Yj = ReLU
ReLU
(29)
0x
07
0x
since the information of input X is divided into two streams, i.e., X and 2, and merged to make the prediction ff. Here, is the target label in question visualization of Eq. 29 illustrates which parts in the input affect predicting the label t. However, Eq. 29 takes into account the gradient flows through both X and Ž although we want to know only the flows through confounders 2 to visualize where the confounders look at. So we define and visualize the gradient through as:
O 07
az
0z
(x:2) = ReLU
ReLU Σ
(30)
0x
0z
0x
8EZ
which is consistent with Eq. 15 of the main paper. Here, at, oz
what the confounder Ž, affecting the prediction Y looks at in the input X. Compared to V (x:2) 8/. Y is defined as:
by simply adding a negative sign, Counterfactual GradCAM Ve
(x:2)
OÝ; 0z
(x:2) Yf = ReLU
Σ
(31)
0
0X
3E
meaning that where confounder 7 focuses suppress the prediction 4/.
15

---

Under review as a conference paper at ICLR 2023
Algorithm Overall Algorithm
Inputs:
sample (X = (zg: z. Y) }. negative sample {X'' = (z))) a'),, Y'), confounder queries Z, number of confounder queries M
Parameters: prior probability c, data encoder f. confounder encoder g. FFN {hy, hgp
Xyx Xy t (x)) (z))
2: X, Xq. X. + f(x), f(Xg): f(X.)
3: 2, Zg: Zo + (ē =g(X, Vz Z}, {Elz g(X, 2), E - 2) E Z}
4: Yy + hi {Exez X (x+2+ā) C2)
D Cc2 is prior probability of
5: Zg.g. Zg.u: Zo.go Zo.v + 2g MM/], Lg M), MM2], 2oIM/2: M]
6: g(1.4)
((a..) ((v1)) (v,0,
t- hg Zgg)) ng (z. (Z(z))) hg 1z,
g
7: Lemusal CB(Y,,
(1.9) 8: Leonfounder + GCE(PA Y) + GCE( YY' GCE( ,Y) + GCE(Ý .Y)
9: L f- Leausal + coafounder
10: return L
C EXPERIMENTAL SETTINGS
C.I DATASET
We validate the proposed model on four benchmark datasets: TGIF-QA (Li al. 2016 Jang al. 2017).
MSVD-QA (Chen & Dolan 2011 Xu et al. 2017). and MSRVTT-QA (Xu et al. 2016 2017 TGIF-QA consists of 103.9 13 QA pairs from 56,720 GIFs and includes three multiple-
choice VideoQA tasks: repetition count, repeating action, and state transition, along with an open- ended frameQA task reasoning on a single frame. MSVD-QA and MSRVTT-QA are both open- ended VideoQA datasets with descriptive QA tasks, while MSRVTT-QA consists of more complex and longer 10,000 trimmed videos and larger 243,000 QA pairs compared to MSVD-QA with 1,970 trimmed videos and 50,500 QA pairs.
C.2 IMPLEMENTATION DETAILS.
Model architecture. We adopt the Transformer (Vaswani et al. 2017) architecture with 12 layers for both the data encoder and the confounder encoder g. Concretely, for the data encoder f. visual tokens X, and text tokens X, are concatenated with an additional [CLS] token to form an input X = (xg, T.) E RND To build Tes we sample 3 frames per single input video. Each frame has a spatial resolution of 224x224, and is patchified into 14x 14 patches with the size of 16x 16 for each. For text token Tg. we set 40 as the max length of the input text sequence. An input text is then tokenized to have hidden dimension of D 768. After concatenating T. and Tes modality encoding is added to input tokens having corresponding modalities. When conducting cross-attention in g. we apply a stop-gradie -oogggrddeent operation to X so that it could not be affected by Leonfoustrr- Also, use M 128 for the number of confounder query tokens.
Training details. For training, the initial learning rate is set to 10-1 with cosine decay and warmup applicd until 10% of the total training step is done. We train the models with AdamW (Loshchilov) & Hutter 2017) optimizer with weight decay rate of 0.01. The probability of confounder dropout is 0.15. Our backbone encoders are pretrained on Webvid (Bain et al. (021).. YT-Temporal 180M (Zellers et al., 2021). HowTo100M (Miech
al. 2019). CC3M (Sharma et al. 2018).
CC12M (Changpinyo et al. 2021), COCO (Lin et al. 2014). VisualGenome (Krishna et al. 2017
and SBU (Ordonez et al.] 2011) 1) as et al. (2021 and Wang et al. (2022). All the experiments
are conducted on 4 x Tesla A00 GPUs.
MCQA details. We concatenate each option and the question and insert the [SEP] token between them to construct the text token sequence. To efficiently calculate Cconfounders we only take into ac- count two negative pairs (X,, Y) and (X,, instead of four negative pairs including (Xg: Y') and
16

---

Under review as a conference paper at ICLR 2023
Q: What does the man do 4 times
Airide(X))
shake head
1.00 rub something with fingers
0.70
rub something
fingers
0.00
head
chew seed 0.00
bare teeth
0.00
bare teeth
steps
take steps 0.00
(a) TGIf-Action
Q: What does the man do after lift bucket ?
PYM)
PLrid00)))
smile
0.75
dump water himself
0.84
dump ice water
on himself
0.23
exhale 0.15
shrug towards
towards the 0.02
the crowd
smile 0.00
speak
speak
(b) TGIF- -raasstrnittonn
Q: What is the small kitten licking while laying in the covers
P17]dt1X))
0.23
0.35
paw 0.17
paws 0.12
tail 0.09
cat 0.10
dog 0.07
dog 0.08
paws
(c) TGIF-Frame
Figure 9: Qualitative results on TGIE. Confidence scores the top-5 predicted answers using conventional likelihood (( and causal intervention P(Y do(X)). Ground-truth answers are colored in red.
(X,, Y), i.e., Loonfounder = GCE(()(9.9) Y)+ GCE(( Y'). This because it is cumbersome to forward all the combinations of negative pairs including concatenated text token sequences for each option.
D OVERALL ALGORITHM
The overall algorithm to train our proposed framework is formulated in Alg.
E FURTHER QUALITATIVE ANALYSES
E. DEBIASED PREDICTION
Fig. iillustrates how models' predictions are corrected via causal intervention P(Y)do(X)). We discuss the detected biases three representative question types TGIE.
TGIF-Action. As shown in Fig. 9a the model tends to predict 'shake head" without considering the visual context before the causal intervention. On the other hand, the prediction is corrected to 'rub something with fingers' after the causal intervention. We believe that this case is biased to text since the 'shake head' co-occurs 141 times more than 'rub' with the word 'man' in the question. Here, the word 'man' serves as the confounder inducing the text bias.
-raansrrnnition. Fig. 9b shows the text bias. In this case, the 'smile' & 'man' pair co-occurs 169 times more than 'dump & 'man' pair, which leads the model to predict 'smile' only considering the text. However, after the causal intervention with P(Y)).(X)) the model predicts the answer correctly.
17

---

Under review as a conference paper at ICLR 2023
Question:
V (x. 87
0.0
What
diees
Mho
after
beckked
Original Prediction:
Question:
62
Debiased Prediction: dump ice water on himsolf
Traa-Trrnsition
Question:
What
smal
Scking whlle
laying
Original Prediction: cat
66
Question:
Deblased Prediction: paw
00
(b) TGIF-Frame
Figure 10: GradCAM on TGIE, We visualize V (x:2) Vj and V ((x,2) Ej on two samples to show where the confounders look the input.
TGIF-Frame. Fig. 9b illustrates the video-biased case. P(Y X) is likely to focus on the visually salient object 'cat' without considering the question. By applying causal intervention, the video bias alleviated and the prediction is corrected to 'paw' from 'cat' considering both the video and question.
E.2 GRADCAM VIUSALIZATION
In the main paper, using variants of GradCAM, we have investigated which words a question or which parts in video taken into account by confounder queries debias the predictions of VideoQA model with GradCAM V ((x:2) Yf and Counterfactual GradCAM V (x,2)* Yj for MSVD. For TGIF-QA, Fig. T0a shows the same QA pair with Fig.e e 9b which is biased to the text. The word 'man' is strongly highlighted by counterfactual GradCAM Ve (x:2) Y implying it negatively
influences predict the correct answer. On the other hand, GradCAM V (xx2) V focuses the word 'bucket' to output a correct answer 'dump ice water on himself". This consistent with our observation that the word 'man' is considered as the text confounder hindering the model from predicting correctly. Fig. [0b] the video-biased QA pair come from Fig. 99c| Counterfactual GradCAM V (x:2) y7 shows that the object 'cat/kitten' which is visually salient object in the video
hinders the model to predict the proper answer. However, GradCAM V V(x.2) focuses on the object 'paw' in the video so the model correctly predicts the answer. This indicates that the object "cat' serves as the video confounder in the video-biased sample.
F FURTHER ABLATION STUDIES
F. CONFOUNDER QUERIES
As shown in Tab. 3 the performance slightly decreases by 0.8% when only using the text founder queries. On the other hand, the performance decreases by 1.9% when only using the video confounder queries. This indicates that the dataset has stronger text bias than video.
18

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study the type of confounders. Z MM 2 : M/ and 2|M/2 MM refer to entire
confounder queries, text confounder queries, and video confounder queries. respectively.
Z M]
Z MM2]
Z[M/2 M]
MSVD
46.4
45.2
44.5
F.2 NUMBER OF CONFOUNDER QUERIES
As shown in the Tab. 4 the model performs best when the number confounders is near 64 or 128. In our experiments, we used 128 confounder queries.
Table 4: Ablation study on the number of confounder queries.
4
16
32
64
128
256
512
1024
MSVD
44.8
45.0
45.7
46.3
46.4
46.4
46.2
46.1
45.4
19


Reviews:
[{"review_id": "qQj8sZRpSe5", "reviewer": "Reviewer_aBdW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies VideoQA through the lenses of causality where it attempts to break the spurious correlations caused by biases (e.g., linguistic biases, visual biases etc.) when predicting answers. It proposes to identify these biases by forcing VideoQA models to respond to unanswerable questions obtained by pairing videos and question-answer pairs of different random samples. The proposed method demonstrates its effectiveness on several VideoQA datasets.", "summary_of_the_review": "The paper has contributions to VideoQA task specifically but similar ideas have been used in related tasks.", "strength_and_weaknesses": "1. Strengths:\n\n+ The paper is well written and easy to follow. Although similar works have been done in VQA, not much work on debiasing and robustness has been done in VideoQA settings.\n+ The proposed method achieves state-of-the-art performance on major VideoQA datasets.\n\n2. Weaknesses:\n\n+ Similar studies have been done in VQA and image captioning [1, 2]. The idea of sampling “unanswerable” QA pairs to detect the shortcuts that VQA models exploit when answering questions has been studied. The proposed method would be more interesting if it focuses on identifying biases that are unique to video data.\n\n[1] Wen, Zhiquan, et al. \"Debiased Visual Question Answering from Feature and Sample Perspectives.\" Advances in Neural Information Processing Systems 34 (2021): 3784-3796.\n[2] Yang, Xu, Hanwang Zhang, and Jianfei Cai. \"Deconfounded image captioning: A causal retrospect.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).\n\n+ It is unclear what biases are detected and reduced by the proposed method. Examples of only “how many” question type do not seem convincing enough. \n\n+ The explanation of how the authors detect visual biases and linguistic biases could be clearer. They also should provide more detailed analysis of how reducing the negative impacts of these biases impacts the overall performance.\n\n+ How do the authors make sure the newly created pairs of video-question are unanswerable? What if questions are general-purpose such as “what is in the video?” or “what action is performed in the video?”?\n\n+ I have a doubt about the results of TGIF-QA dataset as if you train VideoQA models on each question type one by one, what biases do you detect? What makes the improvements so large? What about the Count questions? I do not see it in Table 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dWVTahXmfKD", "reviewer": "Reviewer_LFSv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to fix bias in VideoQA models through causally approaches. The authors noticed that VideoQA models tend to just learn the dataset statistics (\"How many ...\" --> \"2\") and look to rectify that by forcing the model to answer questions that don't make sense for the input in order to disrupt the spurious correlations. The goal here is to enforce that the model only takes advantage of causal relations between the question and the video and makes use of both modalities.\n\nThey go on to run experiments on three benchmark VideoQA datasets and show favorable results, as well as apply GradCAM to assess what parts of the modalities are being taken into account.", "summary_of_the_review": "My review of this paper is overwhelmed by how hard it is to grok the idea. This doesn't pass the bar and would take great revision to fix it. I do actually think it's possible in this round, but it's going to take a wholesale rewrite of the first 2/3. I strongly suspect the paper will be 100x better after doing that.", "strength_and_weaknesses": "Strengths:\n1. A strength of this approach is that it appeals to a well founded base vis a vis the causal methods.\n2. I'm not sure how novel the larger approach is, but applying this to VideoQA in order to debias the question and the video is a good idea that I have not seen before.\n\nWeaknesses:\n1. Boy oh boy is this hard to read. The intro section has three different summaries. It's not until page 6 that the authors finish explaining how their method works, and it starts on page 3. Figure 3 does not help clarify much. In fact, I think it made my understanding even worse. Why do you even need Figure 2 showing the graphical interpretation of the do calculus? In fact, why do you need most of the causal inference preliminaries? A lot of what the authors write is fairly basic and can be summarized / pointed to in other works. Figure 4 is super confusing. Why are there arrows going in both directions? I have rarely, if ever, seen a NN diagram that operates like that. The notation used to explain this method is tremendously difficult to read. I *strongly* suggest improving this because this paper is just about unreadable given how many times it takes to go through and properly glean what's going on. \n2. That's not even to mention that it would fail for anyone who isn't looking on an interface w color or to anyone who is color blind to blue/red. You should use color only to highlight, not for something with real purpose and meaning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "YLPd2YoT06T", "reviewer": "Reviewer_EnYB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a debasing approach to better video QA performance that utilizes learnt confounders to discover and fix spurious correlations. The confounders include both data-agnostic item, i.e. a learnable base (Z) and a data-specific item that depends on the video and textual input (Z'). The model jointly learns to output the right answer with the original input (X) and output the same prediction with created unanswerable input (X') using the confounders. ", "summary_of_the_review": "The paper introduces an approach to using confounders to separate out spurious relations in order to debasing video QA. The idea is novel and obtains good results, however with large additional computational cost. ", "strength_and_weaknesses": "Strengths: \n(1) A novel model architecture for debasing video QA \n(2) Separate out the spurious relations with confounders \n(3) Detailed explanations for the algorithms designs, and implementation details\n(4) Good results on various benchmarks\n\nWeakness:\n(1) Nearly Four-times additional computational cost during training due to unanswerable Xq, Xv. in the original network f and the confounder encoder g; Good to include training times \n(2) Good to have ablation study on the two component of Z (visual and textual)separately, currently only have them as a whole.\n(3) Good to have ablation study on the number of the confounders\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2PcHlHjOLX", "reviewer": "Reviewer_H5yq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for debiasing video question answering (VideoQA) with casual intervention. The novelty relies in that the confounders do not need to be know in advance. Instead, the framework uses pairs of \"unanswerable\" questions at training time to find them. Experiments are conducted on three standard VideoQA datasets, achieving better accuracy than previous methods on all of them. Ablation studies and quantitative results are also provided.", "summary_of_the_review": "The paper proposes an interesting solution to the problem of causal intervention in VideoQA without knowing confounders in advance, which is a more realistic setting than using a list of objects as confounders. Experiments are technically sound and show better performance than previous work.", "strength_and_weaknesses": "Strength\n- Casual intervention is gaining popularity as a way to debias vision-and-language models that leverage statistical correlations to make predictions. An important limitation with casual intervention is that the so-called confounders need to be known in advance. Previous work approximates those condounders with, for example, the list of objects in COCO datasets. In contrast, this paper proposes to learn confounders by using unmatching question-video pairs and forcing the model to make predictions based on their statistical correlations.\n- The experiments show that the method achieves high accuracy. Ablation studies shows that all the parts of the proposed framework are relevant.\n\nWeaknesses\n- A question that arises and it is not very clear in the paper is how to ensure that the unmatching question-video pairs are really unrelated. Just choosing random pairs does not ensure that the video and the question are irrelevant. For example, a \"how many people\" question can still be relevant in a different video.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject