Causal Reasoning through Two Cognition Layers for Improving Generalization in Visual Question Answering
Authors: Trang Nguyen
Naoaki Okazaki
Venue: EMNLP 2023 Main (2023)
Keywords: visual question answering
generalization
casual reasoning
human cognition
TL;DR: Improve the generalization ability in Visual Question Answering task via enhancing the causal reasoning, inspired from human cognition.

Abstract: Generalization in Visual Question Answering (VQA) requires models to answer questions about images with contexts beyond the training distribution. Existing attempts primarily refine unimodal aspects, overlooking enhancements in multimodal aspects. Besides, diverse interpretations of the input lead to various modes of answer generation, highlighting the role of causal reasoning between interpreting and answering steps in VQA. Through this lens, we propose Cognitive pathways VQA (CopVQA) improving the multimodal predictions by emphasizing causal reasoning factors. CopVQA first operates a pool of pathways that capture diverse causal reasoning flows through interpreting and answering stages. Mirroring human cognition, we decompose the responsibility of each stage into distinct experts and a cognition-enabled component (CC). The two CCs strategically execute one expert for each stage at a time. Finally, we prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC, thereby emphasizing causal reasoning and supporting generalization. Our experiments on real-life and medical data consistently verify that CopVQA improves VQA performance and generalization across baselines and domains. Notably, CopVQA achieves a new state-of-the-art (SOTA) on the PathVQA dataset and comparable accuracy to the current SOTA on VQA-CPv2, VQAv2, and VQA- RAD, with one-fourth of the model size.

Causal Reasoning through Two Cognition Layers for Improving Generalization in Visual Question Answering
Trang Nguyen
Naoaki Okazaki
Tokyo Institute of Technology nguyen. t. ay@m. titech.te. titech.ac.jp
Tokyo Institute of Technology okazaki@c. ti titech... ac.
Abstract
However, real-life multimodal data diversity poses challenge for VQA models achieve Out-of- Distribution (OOD) gencralization, which involves performing well on data beyond the training distri- bution instead of relying on independent and iden- tically distributed (iid) data [Zhang et al., 2021, Goyal and Bengio, 2022, Kawaguchi et al., 2022]. Recent studies have highlighted risk of OOD gen- eralization in VQA, where models may respond solely based on the question and ignore the input image due to correlations between the question and answer distribution [Niu et al., 2021, Wen et al., 2021] that exist in human knowledge. For example, questions starting with "Is this...?" are typically expected to be yes/no questions (e.g. "Is this a cat?") rather than multiple-choice questions (e.g. "Is this cat dog?"), leading to possible correct answers with simple "yes" or "no".
Generalization Visual Question Answering (VQA) requires models to answer questions about images with contexts beyond the training distribution. Existing attempts primarily refine unimodal aspects, overlooking enhancements multimodal aspects. Besides, diverse inter- pretations of the input lead to various modes of answer generation, highlighting the role of causal reasoning between interpreting and an- swering steps in VQA. Through this lens, propose Cognitive pathways VQA (CopVQA) improving the multimodal predictions by em- phasizing causal reasoning factors. VQA first operates a pool of pathways that capture diverse causal reasoning flows through inter- preting and answering stages. Mirroring hu- man cognition, we decompose the responsibil- ity of each stage into distinct experts and cognition- enabled component (CC), The two CCs strategically execute one expert for each stage time. Finally, prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC, thereby emphasizing causal rea- soning and supporting generalization. Our ex- periments real-life and medical data con- sistently verify that CopVQA improves VQA performance and generalization across base- lines and domains. Notably, CopVQA achieves a new state-of-the-art (SOTA) on the PathVQA dataset and comparable accuracy the cur- rent SOTA VQA-CPv2, VQAv2, and VQA- RAD, with one-fourth of the model size.
There are many attempts to solve this issue, such as (1) reducing the linguistic correlation [Niu al., 2021, Wen et al., 2021] by avoiding answers gen- crated from only the question used as input, (2) strengthening the visual processing [Yang et al., 2020], and (3) balancing the answer distribution by generating new image-question pairs [Chen et al., 2020, Gokhale al., 2020, Si et al., 2022]. How- ever, these approaches tend to overlook the crit- ical aspect of enhancing multimodal predictions, instead focusing on unimodal aspects (either lan- guage or visual) or the data itself. Our assumption is that enhancing the quality of multimodal pre- dictions would be a potential route for improving generalization in VQA.
1 Introduction
The Visual Question Answering (VQA) task in- volves answering questions about images, requir- ing multimodal processing and common sense derstanding [Antol al., 2015J. VQA research has various applications, including autonomous systems [Deruyttere et al., 2019, Zablocki al., 2022], healthcare [Binh D. Nguyen, 2019, Koval- eva et al., 2020], and education [He et al., 2017].
In fact, solving the VQA task requires the in- tegration of multimodal processing and common sense knowledge. Consequently, VQA can be con- ceptualized two-stage process: input inter- preting and answering, which involves generating an interpretation of the multimodal input and an- swering the question by querying the knowledge space. Besides, similarly to how humans tackle the

---

VQA task, the input misunderstanding can harm the answering stage, posing a risk the overall performance of VQA. Therefore, comprehending the causal reasoning behind this two-stage process becomes crucial to improving the VQA task.
sport-related questions by simply providing the an- swer "tennis." To further challenge the generaliza- tion ability of the VQA models, the VQA Changing Priors (VQA-CPv2) dataset [Agrawal et al., 2017] is introduced. This dataset is deliberately designed to feature different answer distributions between its train and test sets. Therefore, the emergence of the VQA-CP dataset has brought about a signifi- cant shift in the VQA landscape, demanding that VQA models go beyond exploiting correlations and overcoming biases from the given training data. Among attempts of capturing language biases, RUBi [Cadene et al., 2019] introduces a notewor- thy approach that computes the answer probability using only the input question, thereby capturing linguistic biases. Then, the question-only branch carrying the biases is used compute a mask that aims to mitigate biased prediction of unimodal in- put from the multimodal prediction. Building upon this line of research, CFVQA [Niu et al., 2021] pioneers a comprehensive causal-effect view of VQA that subtracts the impact of the question-only branch, represented the answer logit, from the overall answer logit (further discussed in Section 3), Similarly, DVQA [Wen et al., 2021J proposes sub- tracting the question-only branch's 's impact from the multimodal branch the hidden layer. By doing so, DVQA aims to mitigate biased predictions be- fore passing the output through the classifier model for final answer prediction.
In this work, propose Cognitive pathways VQA (CopVQA) boost the causal reasoning in VQA to enhance the OOD generalization. CopVQA derives from the findings of knowledge modularity and cognitive pathways from cogni- tive neuroscience [Baars, 2005, Kahneman, 2011 Goyal and Bengio, 2022], which mentions (1) the human brain organizes independent modules to han- dle distinct knowledge pieces and (2) the commu- nication efficacy among these modules supports the generalization. We decompose each interpret- ing and answering stage into a set of experts and a cognition-enabled component (CC) strategically activates one expert for each stage at a time. By this approach, CopVQA disentangles the VQA task into specialized experts connected by pathways through mntepreeinngannseerii quently, we extend the biases in VQA that are also from the monolithic procedure (instead of expert selection) besides the linguistic correlation. Finally, we emphasize answers governed by the disentan- gled stages with multimodal input and disregard other answers, including ones from the monolithic procedure and from unimodal input.
process. Subse-
The contributions of this work are summarized follows: (I) we propose CopVQA to improve OOD generalization by enhancing causal reasoning that is compatible with diverse VQA baselines and domains; (2) to our best knowledge, CopVQA is the first work that formulates VQA as two layers of cognitive pathways, further facilitating research on causal reasoning in VQA; and (3) we achieve the new SOTA for the Path VQA dataset and mark the comparable results the current SOTAs of VQA- CPv2, VQAv2, and VQA-RAD datasets with only one-four of the model sizes.
In this study, delve deeper into capturing biases using separate branches and subsequently eliminate them. However, besides the linguistic cor- relations, we assume another potential restriction in VQA generalization is the monolithic approach across the diverse scenarios of the multimodal in- put. Therefore, we attempt to eliminate answers by monolithic procedures involved in interpreting and answering stages parallelly emphasize answers by disentanglement approach.
Preliminaries: Causality view in VQA
2
Related Work
Introduced in CFVQA [Niu al., 2021], the causal graph of VQA is depicted in Figure la in which the inputs and Q cause an answer A, and a mediator' K represents the knowledge space. We have the direct paths, which are Q - A and V -> A, representing the answers based solely on unimodal
The generalization restriction in VQA arises from biases between questions and answers in human knowledge, where certain question types exhibit strong correlations with predictable answers based on common knowledge. These biases are also reflected in VQA datasets. For example, in the VQAv1 I dataset [Antol et al., 2015]. a VQA model can quickly achieve an accuracy of around 40% on
'To explore the reasons for input effecting on the output, Pearl [2009], Pearl and Mackenzie [2018] 8J mention the term mediator to dissect the effect into direct and indirect effects.

---

work. Consequently, we define non-biased ap- proach as one that commits to integrating the mul- timodal input and strategically considering proper knowledge pieces for interpreting and answering, rather than relying on monolithic procedures.
Cops
Copp
A
K
A
Each intepreting and answering stages involves diverse distinct experts with cognition-enabled component (CC) that activates one expert time. Consequently, we define complete reasoning flow as selecting an appropriate expert pair for the inter- greting-answene process. In contrast, incomplete reasoning flows rely on monolithic procedures or utilize unimodal input. Finally, we attempt to em- phasize the prediction obtained through the full reasoning flow and disregard the ones from incom- pleted reasoning flows.
(a) Conventional VQA (b) Cognitive pathways VQA
Figure The conventional causal-effect view the VQA task and the proposed Cognitive pathways VQA, which enhances the causal reasoning aspect
input (either question or image). In contrast, the indirect path, (V,Q) K A, represents the answer by considering the multimodal input and the interaction in the knowledge space.
With a single mediator K, the idea of CFVQA is to capture the language prior by the Q + A branch, which involves only the question and pro- duces an answer logit Zq. Likewise, the answer produced through K with multimodal input is Zk- Subsequently, they produce the final answer logit as follows, where is parameter for normalization:
4.2 CopVQA from causal-effect view
To establish solid connection between the com- prehensive overview and implementation details of CopVQA, we present the causal-effect view of the proposed architecture in this section.
The causal view of CopVQA presented in Figure 1b, contains the input pair (V, Q) that leads to an answer A, controlled by the two sets of cognitive pathways denoted mediators Cop1 and Cop2. Specifically, we have direct paths: Q A and V A; and indirect paths including Case 1: (V, Q) + Cop > A, Case Q ->> Cop2 > A, and Case 3: (V, V) + Copi - Copp - A. Specifically, any path that does not involve Copi (Q + A, V - A. and Q + Cops + A) are cat- egorized as wnimodal paths, as it does not involve multimodal interpretation. Likewise, indirect paths that bypass Cop2 (e.g. (V,Q) + Cop1 - A) are considered monolithic ones?, it does not in- volve expert selection for both interpreting and answering. Finally, CopVQA emphasizes the effect from completed reasoning flow, which is (V,Q) + Cop1 - Cop2 + A, and eliminates effects of incompleted reasoning flows, including unimodal and monolithic paths.
Z iinl= = logo + Zg)= - logo Zg + c)
Regarding optimization, they utilize Cross-Entropp loss (CELoss) and define L = CELoss(logo (Zx +
Zg), a) CELoss(logo Zg), a).
4 Cognitive pathways VQA CopVQA
this section, we present CopVQA from multiple viewpoints, which are the overview in Section 4.1, the causal-effect view of CopVQA in Section 4.2.
and the VQA implementation in Section 4.3.
4.1 CopVQA overview
The COA, depicted in Figure 2, serves as VQA backbone that emphasizes causal reasoning in multimodal prediction. CopVQA mitigates the negative consequences of disregarding causal rea- soning, yet enhancing generalization. Initially, we explore the knowledge space as (1) multimodal knowledge for interpreting the multimodal input, and (2) commonsense knowledge for answering based on the interpretation obtained. Additionally, assuming that different interpretations of the input lead diverse ways of answering, we perceive the use of monolithic procedures for interpreting and answering across diverse multimodal input as a po- tential bias that hampers generalization, extending beyond the linguistic correlations discussed in prior
4.3 Implementation Details
We design cognitive pathways as a Mixture of Ex- perts [Jacobs et al.. 1991] (MoE), which disentan- gles a task into specialized experts. Mathemati- cally, an MoE setup M contains (1) experts [E, Eg,..., En} with distinct parameters and (2)
²Q - Cop2 + A is also monolithic path as it does not involve both cognition layers. However, for simplicity, we only mention this path as an unimodal path.

---

Cops
Cops
Cops
Copn
Cop
A
Copn
A
Clops
Z11
Z10
Zo1
Zoo
Z
Zo0 10
Zo
Zoo
MA
MA
W
A M
W M
mul
M₁
W I
M
U,
(a) The causal graphs of four different kinds of reasoning flows and the corresponding computational flows in Cop
INTERPRETING
ANSWERING
Biased Multimodal Knowledge
1q
Biased Commomense Knowledge
Zoo
Unimadal interpretation
Monolithic Layer
Monolithic Layer
Disentangled
Disentangled
Multimodal Knewledge
Commonsense Knowledge
u.q
CC
Σ
mul Multimodal interpretation
CC
Σ
Z
Cognitive pathways
Cognitive pathways
(b) Visualization of interpreting answering processes through monolithic and disentangled knowledge spaces in computing Zoo and Z11.
Figure 2: Cop from the causal-effect view aligned to computational flows. CopVQA emphasizes the full causal reasoning flow (green), simultaneously eliminating effects from monolithic (blue) and unimodal (red) flows.
gating model 9 d + RV As described in Equa- tion 1, with input x E Rd, the output y is the sum of results from N experts weighted by g. produced by G. We conduct Gum to achieve 1-hot-lik like probability.
produces an interpretation as a dy-dimensional vec- tor, denoted as E Rd: In the answering stage, CopVQA conducts classifier model that outputs Zjk RRY, where j, k e (0, 1}. Specifically, j is 1 when the interpretation is governed by experts from Copı in interpreting. and 0 otherwise (gov- erned by monolithic procedure). We analogously (1) define E (0, 1} for Cop2 in the answering stage. Obtaining the answer can be done by getting the max value's index of logo (Zjx).
Guumell-mxxgeea
with
Σ #=1
x En(x);
4.3.1 Computational Flow in CopVQA
We describe the computational flow in CopVQA by introducing the notations and 2-stuge process. Given the pre-processed question g de and image Rd., where dg and d, are the shapes of the modalities, the VQA model aims to predict the answer à, which is an index the vocabulary set size of V. the interpreting stage, COpVQA
Notation 1: Denote M, and MA as MoE. se- tups align Cop1 for Interpreting and Cop2 for Answering. respectively. Ni experts in Mi is designed to map Rd, + Rd responding the interpreting stage; similarly, N2 experts in MA map 4 + R responding to the answering stage. The cognition-enabled components align to g(.)

---

in each MoE, which are G R4 + R and GA :R: R for the interpreting and answer- ing stages, respectively.
(9) Model architecture Experts in the Model of Ex- perts (MoE) framework have a common architec- ture but carry distinct parameters. Denote layer 1 Rén -> RRuut with a hidden size h in a particular baseline that is in charge of interpreting or answer- ing stages, design /' R -> RRutt with the hidden size h'. Subsequently, individual experts and the monolithic models in CopVQA share the same architecture as l'. Precisely, h' is fine-tuned to achieve the optimal result. Through experimen- tation, we have discovered that selecting values for h' such that the total number of parameters in all experts and monolithic models does not exceed the number of parameterss in / tends to yield optimal sults. This observation regarding the adjustment of h' aligns with the principles of knowledge modular- ity, which we discuss in more detail in Appendix C. As a result, CopVQA does not increase parameters beyond those present in the baseline model.
Inference: à - argmax(logo (2 (2not)
Notation 2: We denote monolithic procedures Wh R4, + d. and W35 IRd -> IRY for the interpreting and answering stages, respectively. Stage 1- Input interpreting: Let mmli E R& be the multimodal interpretation and ig R4 be the unimodal interpretation, described in Equation 2. Specifically, imuu computed by passing q through Copı and subsequently applying Fusion function (follow the baselines, details in Section 5.3) on e to obtain a multimodal interpretation. Likewise, ig is obtained from the input question q alone by W M-
mmd = Fusion (Cop1 v), 4q = 1g-WMM (1) (2)
Stage 2 Answering: In this step. we compute the pool of outputs from multiple reasoning flows. Let Z11, described in Equation 3, align the full causal reasoning flow governed by both cognition layers. Likewise, Zio, as in Equations 4, represents the monolithic path that involves only Cop1. Fi- nally, Zor and Zoo, formulated in Equations 5 and 6, represent for the output from unimodal paths.
Experiment Setup
We conducted experiments to validate: Hi -Causal reasoning in multimodal prediction benefits VQA performance (Section 6.1), H2 Causal reasoning in multimodal prediction enhances OOD general- ization (Section 6.1), and H6- The disentangled architecture is crucial for reasoning (Section 6.2).
Z11 = Cop2(Emul)
(3)
Z10 Whr (imu))
(4)
Zo1 Cop2(tg)
Zoo = W
(6)
5.1 Datasets
4.3.2 Training and Inference Time Output finalizing As discussed Section 4.2, CopVQA emphasizes the impact of the fully causal reasoning flow and eliminates the impacts of the incompleted reasoning flows. Inspired by the Niu et al. [2021] introduced in Section 3, we design strategies for output finalizing, loss functions, and answer finalizing for the inference time in Equa- tions 7, 8, and 9, respectively, with a is the target answer and the LossFn function is inherited from particular baselines.
To examine Hi, conduct experiments on four datasets in two domains: (1) real-life im- ages: VQA-CPv2 [Agrawal et al., 2017] and VQAv2 [Goyal et al., 2017] and (2) medical data: Path PathVQA [He et al., 2021] and VQA-RAD [Lau et al., 2018J. Questions in VQA-CPv2 and VQAv2 are divided into three types: "Yes/No" (Y/N), Num- ber (Num.), and Other, with 65 categories of ques- tion pre-fix (such as "Is it...""). Vath catego- rizes "Y/N" questions separately from "Free-form" questions, whereas VQA-RAD categorizes ques- tions with limited answer candidates as "Close" and the remaining questions as "Open" type.
fimall = logo + Z1o + Zo1 + Zoo)
- loooo + Zo1 + Zoo) (7)
To examine H2, we investigate results on the VQA-CPv2 dataset, valuable benchmark for sessing OOD generalization in VQA. This dataset features substantial variations in answer distribu- tion per question category between the training and test sets, making it an ideal choice for evaluating (8) the models' ability OOD scenarios.
Ltotal LossFn (Z Zoo
L = LossFn(logr((110).1
Lv (os)Fn((ogo(Zo1),4
+LossFn(logo (Zoo), Zoo),
LCopVQ4 = Liodal + Lm + Lu

---

Test set
VQA22
VQA-CPP2
Method
Y/N
Num.
Other
Overall
Gap
Y/N
Num.
Other
Overall
Gap
RUBi
61.1
68.7
20.3
43.2
47.1
SCRT
78.8
41.6
54.5
62.2
72.4
10.9
48.0
49.5
Mutant
82.1
42.5
53.3
62.6
88.9
49.7
50.7
61.7
CFVQA
81.3
43.4
20.4
50.1
10.1
60.7
90.420.3
21.3-
45.2
20.2
55.0
+CopVQA
81.4
43,8
43,8-002
52.4
42.4
62.2-0.3
62.2
+1.5
91.1
41.6
46,4
57.8 47.8.0.3
+2.8
DVQA
81.7
42.8
42.8-0.4
66.7
64.3
20.3
88.5
88.5
48.7
50.1 F0.1
61.1 =0.1
+CapVQA
82.6 +0.2
45,2-0.3
59.0+0.3
+0.2
+3.2
92.1+0.3
59.4 -0.4
61.4+0.3
67.9-0..3
+6.8
Table I: Accuracy comparison on VQAv2 and VQA-CPy2 datasets. The best scores are bolded.
Test set Method
PathVQA
VQA-RAD
Y/N
Free-form
Overall
Gap
Open
Close
Overall Gap
MMQ
83.6*0.4
13.5
48.6
52.4
66.8 77.1 77.1=0.6 70,2 +0.3 +3.4
75.3*1.
VQA
85.3+0.1
16.6 16.6+0.1
50.9 0.3
+2.3
56.5 +0.9
Table Accuracy comparison on PathVQA and VQA-RAD datasets. The best scores bolded.
5.2 Baselines
Experimental Results and Discussion
We implement and compare CopVQA to baselines that do not emphasize the causal reasoning in mul- timodal prediction. In VQA-CPv2 and VQAv2, we implement CopVQA on CFVQA [Niu et al., 2021] and DVQA [Wen et al., 2021] baselines that attempts to eliminate the language priors. Besides, to fulfiil the comparison, we compare CopVQA to approaches that strengthen visual processing: SCR [Wu and Mooney, 2019]; balancing answers distri- bution: Mutant [Gokhale et al., 2020]². In medi- cal datasets, we conduct CopVQA on Path VQA's SOTA MMQ [Binh D. Nguyen, 2019], which leverage significant features by meta-annotation.
6.1 Quantitative Results
Overall, Cop outperforms all baselines in both iid and OOD. Specifically, Table 1 presents results on VQAv2 and VQA-CPv2, and Table shows results on PathVQA and VQA-RAD. Baselines marked with a "i" are reported results from the orig- inal paper, while others are our reproduced mean and standard error 5 random seeds. The "Gap" shows the improvement of CopVQA from the produced baseline, with a "_" for unreported results. In addition, Table the Appendix compares the sizes and training time between the models.
Proving Hi, CopVQA reveals the consistent improvement in all answer categories regardless of baselines and domains. Notably, in VQAv2, VQp achieves +1.5% and +3.2% higher than CFVQA and DVQA baselines, respectively, in the Overall score. In medical datasets, CopV VQA marks the improved ability in the medical VQA task with a +2.3 and +3.4 points higher MMQ baseline.
5.3 Implementation details
We fine-tune N1 and N2 with each baseline and maintain consistent configurations, including the optimizer, number of epochs, and batch sizes. For the DVQA baseline, the best setting for the (N1, pair is (). In contrast, for the CFVQA and MMQ baselines, the optimal pair (5, 5).
CFVQA employs Fusion as the Block factory fusion from YounYo et al. [2019] and trains by CrossEntropy loss. Likewise, based on the UpDn [Anderson et al., 2017], DVQA designs Fusion as the multiplication of the pre-processed and & and uses BinaryCrossEntropy as LossFn. On the other hand, the baseline MMQ, based on BAN [Kim et al., 2018], designs Fusion as the recurrent multi- plication of the processed v, q. and BAN's result on t and g, and utilizes the BinaryCrossEntropy loss.
Proving H2, regardless of the based models, CopVQA demonstrates the effectiveness of causal reasoning in generalization by outperforming base- lines in VQA-CPv2 by a large margin. Specifically, CopVQA acquires remarkable improvement of +2.8% in CFVQA-based and +6.8% in DVQA- based in the Overall score. Notably, CopVQA shows exceptional performance on the Number type, with +20.3 and +12.7 points increased from the CFVQA and DVQA baselines, respectively. Compared current SOTAs, CopVQA achieves a new SOTA on PathVQA with +2.3 points higher than MMQ, the previous SOTA. Besides, CopVQA
"Wo compare to Mutant that based on UpDn [Anderson et al., 2017] since DVQA and our DVQA-based CopVQA share the common foundation of UpDn.

---

Question:
this real dog or a toy dog?
Correct answer:
Toy
CopVQND)
CopVOAIC)
this t1 weighted t2 weighted or flair image?
71.25
53.85
stuffed animal 13.47
5.85
Correct answer: Flair
kdd+2.11
2.17
kitten + 2.17
CopvQA/M):
Flair
DVDA
CFVQA
MMQ.
No
ves
no neither tay
nelther
2.38
5.52
(a) VQA-CPv2
(b) VQA-RAD
Figure 3: Sample of debiased cases, listing the labels with top probability. The green and red labels the correct and incorrect answers, respectively.
Copa 1
Cop2 2
Cops 1 2 3
1.0
1
0.8
10
Cop
0.5
0
badtrok
naht
Y/N
Num.
Other
(a) CopVQA(D
Witchen
none-tion
Copy
Copy
Copp
Inft
left
12345
12345
12345
Cops UPPWIN
UNDWAN
56000L
Thiin
Tost
CooVOAIC)
DVQA
Vo4VO
CFVOA
Phain
Test
CooYOA(D)
DVQA
CopVoArc)
CPVDA
Y/N
Num.
Other
(b) CopVQA(C)
(a) How many
(b) Which
Cop2
Cop2
Cop2
Figure 4: Answer distributions on VQA-CPv2 and the comparison of debiasing ability. VQA produces the most similar distributions as the test set.
12334
12345
12345
Doooa
Copt
Y/N
Num.
Other
is comparable to MMBS [Si et al., 2022], the cur- rent SOTA of VQA-CPv2 and VQAv2, with only one-fourth of the model size (details in Table 4 in Appendix). Specifically, MMBS gains 68.39% and
(c) CopVQA(M)
Figure 5: The proportion of experts selection from Copi and Cops assignment during the inference time.
69.43% Overall scores on VQA-CPv2 and VQAv2,
respectively, with over 200M parameters, while DVQA+CopVOA marks 67.9% and 67.5% with less than 52M parameters. Lastly, CopVQA marks 1.9 points lower than CLIP [Eslami et al., 2021] current VQA-RAD's SOTA.
by applying softmax on the answer logit. In Fig- ure 3a shows that Cop accurately answers the question by treating it as multiple-choice type, whereas the baselines are trapped by bias toward Yes/No type, resulting in incorrect answers. In ad- dition, CopVQA models list other high-probability answers relevant to the correct one, such "stuffed animal" or "puppy". In Figure 3b, COpVQA gives the correct answer, while MMQ struggles to recog- nize the question's purpose and gives an incorrect one. We provide more samples in Appendix B. Debiased answers distribution Figure 4 de- picts the distribution of answers collected from the entire train set, test and models' prediction during inference. The analysis demonstrates that CopVQA(D) produces the most accurate distribu- tion that resembles the test set by mitigating the biased answers found DVQA in both examples. Likewise, CopVQA(C) exhibits the gemeralizabil-
Compared to other approaches in VQA-CPv2 and VQAv2, Table indicates that Cop is sig- nificantly better than SCR. Specifically, CopVQA achieves comparable accuracy or even higher than Mutant without data augmentation.
6.2 Qualitative Results
We investigate Hg by the underlying performance of CopVQA VQA-CPv2 and VQA-RAD. We denote CopVQA based on DVQA and CFVQA VQA-CPv2 CopVQA(D) and CopVQA(C), and based on MMQ on VQA-RAD as CopVQA(M).
Discussion on the debiased samples Figure compares debiasing results with values obtained

---

CopVOA(D)
CopVQA(C)
CopVOA(M)
Accuracy
50
50
25
25
25
Zmnal
211
2,0
Zu
1220
1Zo
1Zo0
Zina
Z
Zunal
Z11
2
Zu
1Z 1210
1
Z
Zanal
Z11
Z,0
Zu
1210
1
zrana
Zh
Figure 6: Ablation studies to clarity the individual effect of components in CopVQA design. The original design
CopVQA demonstrates the dominant performance over all variants.
ity over CFVQA as it removes the strongly biased answers "2" in Figure 4a and "right" in Figure 4b. Appendix B discusses more on models' success and failure cases in the debiased answer distribution.
N2
N2
N2
2
10
3
3 5
10
50
2
60
40
50
40
40
30
30
10
10
10
20
Discussion on mechanisms selection Figure 5 presents the proportion of expert-pair selection from Cop1 and Cop2 over N1 x N2 combinations. CopVQA consistently assigns the non-overlapping of experts to separate answer categories. For in- stance, in CopVQA(D), Cop1 assigns experts 2, I, and for the Y/N, Number, and Other types, respec- tively, in most of cases. This pattern aligns with the strategy of COA, where experts specialize in distinct contexts of the multimodal input and com- monsense knowledge space. Moreover, we observe that CopVQA allot more than one answering ex- pert for Number type by visible proportion, which adapts the diverse skills required, explaining the significant improvement of CopVQA on this type.
CopVOA(D)
CopVQA(C)
CopVOA(M)
Figure 7: Overall accuracy comparison over various configurations of N₁ and N₂ CopVQA.
rate v in computing Zo1. or designing Z Capa(Fusion (g), r)). Fônal is comparable but does not exceed Final in all models, proving the pivot of the original causal structure design and the balance of monolithic and unimodal paths. We collect scores from the best checkpoint for CopVQA(M) and Zrand and from the best score final over epochs for CopVQA(D) and CopVQA(C).
Number of experts in Cop1 and Cop2 Figure 7 indicates results from multiple (Ni, NN2 pairs. We observe that values in the range of 3 to experts tend yield higher scores, with the pair of dupli- cated values achieving the highest scores, while pairs with values of 2 and 10 result in significantly lower scores. This finding suggests that an ap- propriate balance between experts in two sets of pathways is crucial to acquire high performance.
7 Ablation Studies
Individual effects in CopVQA We conduct abla- tions to observe the effect of components in Equa- tions 7. The analysis confirms the dominance of COA over the modified versions. Figure 6 com- pares the Overvall score of finl to ones from: (1) Zu1 Z10- and Zu (the sum of Zoi and Zoo). Values of Z10 are significantly lower than those of Z fimal, supporting the assumption that disentan- gled architectures benefit VQA.
8 Conclusion
We proposed CopVQA, a novel framework im- prove OOD generalization by leveraging causal rea- soning in VQA. CopVQA emphasizes the answer with a full reasoning flow governed by disentan- gled knowledge space and cognition-enabled com- ponent in both interpreting and answering stages while eliminating answers in incompleted reason- ing flows, which involve unimodal input mono- lithic procedures. CopVQA outperforms baselines across domains in iid and OOD. Notably, CopVQA achieves new SOTA on Path Vat and comparable results with the current SOTAs of other datasets with significantly fewer parameters.
(2) !Zn,, !Z and !oo denote results of fiiol when Z11, Z10. or Zu is excluded from fimml in inference, respectively. It indicates the importance of monolithic and unimodal paths when 10 !Zo1. and IZ are far from Ziina-
(3) Zpand as the score of Ziiaal when experts are randomly selected in inference. Fo drops dramatically, proving the vital role of selecting proper pair of experts to improve accuracy.
(4) / as the score of Z final when we break the causal structure in Figure/ 2 and incorpo-

---

Limitations
Binh X. Nguyen Tuong Do Erman Tjiputra Quang D. Tran Binh D. Nguyen, Thanh- Toan Do. Over- coming data limitation in medical visual question answering. In MICCAI, 2019.
Despite being touted as a robust backbone that en- hances generalization in the VQA task through its emphasis on causal reasoning in multimodal pro- cessing, the CopVQA method exhibits certain limi- tations that should be acknowledged, including:
Remi Cadene, Corentin Dancette, Hedi Ben younes, Matthiuu Cord, and Devi Parikh. Rubi: Re- ducing unimodal biases for visual question answering. In H. Wallach, H. Larochelle. A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors. Advances in Neural In- formation Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: //proceedings. neurips. cc/ pctppppppp/////////// 51d92be1 pdf.
Sensitive in data with limited occurrences such as brand names or country names, which poses challenges for effective debiasing. We further discuss this point in Appendix B.2.
Require careful finetuning for each baseline and dataset, involving tuning hyperparame- ters like N1, N2, and experts' architecture to achieve optimal performance.
Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu. and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question an- swering. In 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 10797-10806, 2020. doi: 10.10094424400000 2020.01081.
A Acknowledgement
These research results were partly obtained from the commissioned research (No. 225) by the Na- tional Institute of Information and Communications Technology (NICT), Japan.
Thierry Deruyttere, Simon Vandenhende, Dusan Gru- jicic, Lue Van Gool, and Marie Francine Moens. Talk2car: Taking control of your self-driving car. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing IJCNLP, pages 2088- 2098, 2019.
References
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question an- swering. 12 2017.
Sedigheh Eslami, Gerard de Melo, and Christoph Meinel. Does CLIP benefit visual question an- swering in the medical domain as much as it does in the general domain? arXiv e-prints, art. 1xXi:2212.119900 December 2021.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image caption- ing and VQA. CoRR, abs/1707.00 http//777.0798.. 2017. URL http://arxivv hotvvoor//aaao orggabs///00 11700//99777.78.
M Garibaldi. Machine learning and deep learning. Journal of Business Research, 134:1-13, 2021.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answer- ing. In International Conference on Computer Vision (ICCV), 2015.
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. MUTANT: A training paradigm for generalization in visual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP). pages 878-892, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 1/663//1//202... eetp00eemmllmmmmmii / //aclanthology, or orooorg////// wttgg//20.eeem wwwp..mneeeeeeee.mmmmmmmm. smnlp-main..e mam.mma.............
Bernard J. Baars. Global workspace theory of consciousness: toward a cognitive neuroscience of human experience. In Steven Laureys, ed- itor, The Boundaries of Consciousness:: Neu- robiology and Neuropathology. volume 150 of Progress in Brain Research, pages 45 53.53, Elsevier, 2005. doi: http:///o..oo/110.001////0000000 50004-9. URL http://s///wwwssccieneneereeee http://iiiee/rtttteee///////////
URL ht
Anirudh Goyal and Y. Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 478 10 2022. doi: 10. 10.1098/ rspa.2021.0m 1,068..
Hedi Ben- Younes, Remi Cadene, Nicolas Thome, and Mathhiu Cord. Block: Bilinear Superdiagonal Fu- sion for Visual Question Answering and Visual Rela- tionship Detection. In The Thir AAAI Con- ference on Artificial Intelligence, 2019. URL http: //remi cadene. com/ com/pdfs wwwppraaaaaaeaaaaaaaaaaaaaaaaaaa 10....p..... pdf.
Yash Goyal, Tejas Khot, Douglas Summers Stmmerr--Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA mat- ter: Elevating the role of image understanding in Visual Question Answering In Conference on Com- puler Vision and Pattern Recognition (CVPR), 2017.

---

Bin He, Meng Xia, Xinguo Yu, Pengpeng Jian, Hao Meng, and Zhanwen Chen. An educational robot system of visual question answering for preschoolers. In 2017 2nd International Conference on Robotics and Automation Engineering (ICRAE), pages 441- 445, 2017. doi: 10.1 (09/1CAA..2017..3)) 1426.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian Saan-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. pages 12695-12705, 06 2021. 2021.01251.
doi: 10.1 199CCP444477
Judea Pearl, Causality: Models, Reasoning and Infer- ence. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. To- wards visual question answering on pathology im- ages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 708-718, Online, August 2021. Association for Computational Linguistics. doi: 10663/v// 80211acl-short.90 URL https://aclantholoogy org/2821.ac1 act.sort.. 90.
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Basic Books, Inc., USA, 1st edition, 2018. ISBN 046509760X.
M R Sayyed and S Kulkarni. Explainable machine learning with prior knowledge: An overview. arXiv preprint arXiv: 10172. 2021.
Qingyi Si, Yuanxin Liu, Fandong Meng. Zheng Lin, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. Towards robust visual question answer- ing: Making the most of biased samples via con- trastive learning, In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6650 (000)-662.22 Abu Dhabi, United Arab Emirates. De- cember 2022. Association for Computational Lin- guistics. URL https: h/tps//atalloogg////oe 202/2022. Findings-emnlp. 495.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991. doi: 10.1 162/neco. (9122/ecco.1911333. 1..793.1.79
Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011. ISBN 978-0- 374-27563-1.
K. Kawaguchi, Y. Bengio, and L. Kaelbling. Gen- cralization in deep learning. In Mathematical Aspects of Deep Learning, pages 112-148. Cam- bridge University Press, dec 2022. doi: 10.1017/ 978 10810000990090003..... URL https://dbi.org//// 00178229771889922550 003.
Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. Debiased visual question answering from feature and sample perspectives. In Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL /ttpps///peerreview. net/ forum?id-Z4rySSPPWgg
Jin-Hwa Kim, Jachyun Jun, and Byoung- Tak Zhang. Bi- linear Attention Networks. In Advances in Neural In- formation Processing Systems 31, pages 1571-1581, 2018.
Jialin Wu and Raymond Mooney. Self-critical reasoning for robust visual question answering.
In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL neurips. cc/ http:/pper_fillsspppere///////// 9//llel// 0 pdf.
Olga Kovaleva, Chaitanya Shivade, Satyananda Kashyap, Karina Kanjaria, Joy Wu, Deddeh Bal- lah, Adam Coy, Alexandros Karargyris, Yufan Guo, David Beymer Beymer, Anna Rumshisky, and Van- dana Mukherjee Mukherjee. Towards visual dia- log for radiology. In Proceedings of the 19th SIG- BioMed Workshop Biomedical Language Pro- cessing. pages 60-69, Online, July 2020. Associ- ation for Computational Linguistics. doi: 10.18653/ v v1/2020.bionlpp 1.6. URL https: httpps///aclantholoog/ org/2020. 000000.boonlp1...
Xiaofeng Yang. Guosheng Lin, Fengmao I.v, and Fayao Liu. Trmet: Tiered relation reasoning for composi- tional visual question answering. In ECCV, 2020.
Éloi Zablocki, Hédi Ben- Younes, Patrick Pérez, and Matthicu Cord. Explainability of deep vision-based
S Kwon, J-H Kim, and D-H Lec. Modular deep learning for predictive maintenance of industrial assets. IEEE Access, 7:160-170. 2019.
autonomous driving systems: Review and challenges. International Journal of Computer Vision, 130( (00)))) 2425 2452, Oct 2022. ISSN 1573-1405. doi: 111.007/11122330222-0161111
Jason Lau, Soumya Gayen, D. L. Demner, and Asma Ben Abacha. Visual question answering in radiology (vqa-rad). In Open Science Framework, 2018.
URL https://doi. org/10. 1171110100771111122-------11
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Ben- jamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generaliza- tion. Commun. ACM, 64(3): 107-115. feb 2021. ISSN 0001-0782. doi: 10.1145/344677. URL http:://doi.org//// 11453447777.
M I Lindauer, K Eggensperger, M Feurer, A Biedenkapp, J Marben, P Müller, and F Hutter. Informed machine learning- a taxonomy and survey. arXiv preprint arXiv:7903 12394, 2019.

---

Y Zhang, Y Zhang, Wang, and Zhang. Modular deep learning for medical diagnosis. JEEEEAcees Access, 174-184, 84, 2020.

---

CopVQA(D)
CopVQA(C) CFVQA
CapVQA(M)
Number experts
DVQA
MMQ
100
Accuracy
60
50-
70
40-
40
30
50
Accuracy
65
25
Epoch
Epoch
Epoch
60
Figure 8: Comparison of validation results during the
55
training process.
50
CopVOA(D) CopVOA(C) CopVΩA(M)
A Further discussion on experiment details
Figure 9: Ablation study on the essential role of the disentangling ability of experts.
A.1 Datasets
The VQA-CPv2 and VQAv2 two popular datasets in the VQA domain. We follow stan- dardized downloading process and input preprocess of CFVQA for CopVQA(C) and from DVQA for CopYQA(D). This ensures consistency and com- parability across different approaches. Similarly, in the case of Path VQA and VQA-RAD from the MMQ baseline, similar guidelines are followed to standardized experimental setup. In addi- tion, Table 3 indicates the training, validation, and test splits of each datasets.
accuracy, with a broader range of errors. This anal- ysis results confirm the essential role of the disen- tangling ability of each expert that learns distinct knowledge to enhance the performance and reserve the robustness of the proposed CopVQA.
B Further discussion on qualitative analysis
B.1 Debiased samples
Figure 10 presents cases where CopVQA correctly answers questions that other baselines fail to an- swer correctly. The primary reason for this success is the improved understanding of the question 's pur- pose by CopVQA. It effectively addresses biases and avoids potential pitfalls that lead to incorrect answers. Additionally, in VQA-CPv2, COp takes into consideration the relationships between related answers, which further enhances its accu- racy and robustness. For instance, the answers "outdoor" and "indoor" have a high probability the third sample, while baselines consider "yes" and "no" as other potential answers. Overall, the qualitative analysis demonstrates the effectiveness of CopVQA in mitigating biases and improving the performance of VQA systems.
A.2 Validation along epochs
In Figure 8, the validation results indicate that the proposed CopVQA outperforms the baseline mod- els DVQA and CFVQA by significant margin. The performance of Cop is noticcably bet- ter, showcasing its effectiveness in visual question answering. Additionally, when compared to the MMQ baseline, CopVQA demonstrates compara- ble performance, with a slight improvement. These findings highlight the superiority of CopVQA in addressing the task of VQA, surpassing existing baseline models and exhibiting promising potential in the field.
Figure shows cases where all models' predic- tions are correct. of these cases, the proposed method Cop gen- erates a higher probability for the correct answer compared to baselines. The higher probabilities as- signed by CopVQA reflect its ability to capture im- portant visual cues, contextual information, and mantic relationships in disentangled architectures, thereby outperforming the baseline models in terms of answer quality. This analysis highlights the supe- rior performance of CopVQA in producing reliable and accurate answers in VQA.
A.3 The role of disentangling ability
observed that in the majority
To verify the role of the disentangling ability of experts, we conduct an ablation that adjusts the number of experts to be activated by & of both Cog- nitive pathways and both training and inference time. Particularly, we modify the G to achieve a k-hot-like instead of I-hot-like probability.
The comparison in Figure 9 demonstrates the dominant performance CopVQA with only one expert activated by each set of pathways. Likewise, increasing 1o to 2 or 3 significantly drops VQA

---

Datasets
Num. Questions
Num. Images
Train
Validation
Test
VQA-CPv2
438K
121K"
409K
219K
VQAr2
1,10KK
204K
443K
214K
453K
PathVQA
32K
4.9K
19K
6K
6K
VQA-RAD
3.5K
315
3K
451
Table 3: Dataset indication. denotes the summary in the train only. "." marks the not reported information. "K" stands for one thousand.
Question:
What signal are these traffic lights? Yellow
Correct answer:
CopVOAAD)
CopVOAICI
yellow
48.92
20.28
The mass is found in which part of the pancreas?
4.59
19.64
and
grean
6.81
8.98
CopVVAAMi::
arswer:
Yes
DVQA
CFVOA
MMQ:
Postertorly
green yellow
yes
yel
17.6
stop
2.56
0.95
Question:
many
to
Is this image?
answer:
CopVQAID)
CoPVQA(C)
Correct
No
68.91
38.64
18.2
12.01
CopVOA(M):
No
MMQ:
Pa
DVOA
CFVOA
34,61
37.47
27.02
27.17
3.96
1.79
Question:
people playing Indoor outdoor satting?
Correct Outdoar
What plane is this?
CopVOA(D)
CapVOA(C)
outdoor
79.0 9o.06ottoorr
75,0
Correct answer: Axial
Indeor
grass 3:17
yard
CоpVQA(M):
Axial
DVDA
CFVQA
MMQ:
Coronal
outdoor
65.46
66,92
22.26
20.53
no 1.07
nolther-
D.76
VQA-CPv2
(b) RAD
Figure 10: Examples of cases effectively debiased by CopVQA over baselines. The green lahels are the correct answers while the red labels are the incorrect answers.
Figure 12 shows cases where all models' pre- dictions are incorrect. It is observed that sig- nificant portion of these cases falls into two cat- egories: complex counting tasks and ambiguous answers. Complex counting tasks often involve intricate arrangements
Test
VQA-CPv2
VQAV
Method
Params
Duration
Params
Duration
CFVQA
48.33M
47.89M
large number of objects, which pose challenges for the models accurately counting and identifying the objects. The inherent complexity of these tasks can lead to errors in the predictions of all models, including CopVQA. Ad- ditionally, cases with ambiguous answers present difficulties for the models. the correct answer may vary depending on the interpretation sub- jective judgment. For example, the second sample can categorized in both complex counting tasks and ambiguous answers where (1) the image in- cluding many people with dark clothes in a dark background, (2) the input is the duplication of 4
+CopVQA
47.20M
0.96x
46.24M
0.91x
DVQA
54.21M
X
12.
+CopVQA
51.09M
0.87x
51.29M
0.93x
Test set
PathVQA
VQA-RAD
Method
Params Duration Params
Duration
MMQ
28.15M
20.07M
+CopVQA
28.06M
0.98x
20.02M
0.94x
Table 4: Comparison of model size and proportion of average training duration over 5 runs. All models are trained on the same single NVIDIA RTX A6000.

---

Question:
What is the color of the scissors?
Correct answer:
Red
organ system
CopVQMID)
CopVOA(C)
50.39
Correct arswer: Brain
pink 12.78
7.06
red
3.76
3.84
CopVQA(M]:
Brain
DVDA
CFVOA
MMQ:
Brain
rod
4.24
red
80.96
red and black
1.87
pink
0.98
Black
red and white
Question:
Is anyone sitting on the bench?
Correct answer:
No
Is the patient lying supine?
CopVOA(D)
CapvOA(C)
88.22
Correct answer:
Yes
0.5.5 1.5
0.93
CopVOA(M):
Yes
unnknwwn-O 0.47
0.55
DVQA
MMQ:
Yes
yes nanc- unionwn
95.91
no
98.39
0-
none
Question:
How many people are the picture?
Correct answer:
Why the pancreas abnormal?
CopVOA(D)
CapvOAICI
91.1
94,03
Correct answer: Enlarged
4.224
D-0055
10 0.13
no 0.21
CopVOA(M):
Enlarged Enlarged
DVQA
CEVQA
0.2
0.44
(a) VQA-CPv2
(b) VQA- RAD
Figure 11; Examples of cases where all models successfully answer the questions. The green labels are the correct answers while the red labels are the incorrect answers.
images, since (3) the target answer likely to only rely image piece. In such instances, all models struggle provide accurate responses, resulting in incorrect predictions across the board. These find- ings emphasize the existing limitations and chal- lenges in VQA systems when it comes to intricate counting tasks and handling ambiguous answers.
the train and test sets. This implies that the models have not effectively generalized from the training data to handle the biases present in the test set, Additionally, the models may struggle with pro- viding accurate answers for categories that have limited occurrences, such as specific brand names that appear only a few times in the dataset. These challenges highlight the need for further improve- ment in handling biases and addressing answer categories enhance the overall performance of VQA models.
B.2 Answers Distribution
Figure 13 indicates the analysis of answer distribu- tion in VQA, across the train, test sets, and predic- tions, where all models successfully overcome the biases. Firstly, it becomes evident that biases exist in the VQA-CPv2 dataset as the answer distribution in the train and test significantly differ. How- ever, despite these challenges, all models demon- strate successful debiasing. Notably, CopVQA out- performs the baseline models by producing a test set distribution that aligns more closely with the ground truth, indicating a notable improvement in its ability to generate likely answers.
C Design principles inspired from Knowledge modularity
In the context machine learning, modulariza- tion refers to the decomposition of complex sys- tems or models into smaller, more manageable modules [Kahneman, 201 1, Lindauer et al., 2019, Zhang et al., 2020, Kwon et al., 2019, Sayyed and Kulkarni, 2021, Garibaldi, 2021, Goyal and Ben- gio, 2022J. the realm of knowledge modularity and machine learning, this principle suggests that breaking down complex tasks or models into mod- ular components can lead to more effective and efficient learning. The advantage of this modular
Figure 14 shows cases where all models fail to perform well and biased answers persist in the pre- dictions, several observations can be made. Firstly, there are strong biases present, indicated by the stark disparity between the answer distributions in

---

Question:
number
CopVQA(D)
CopVQA(C)
23
15.11
23
19:06
What type artifact shown In this Image?
15
8.28
10
10.96
38
6.22
51
Correct answer Melion
DVQA
CPVOA
CopVOKIM::
weighted
MMQ:
weighted
15:34
11:06
11.46
8.38
Question:
How many people are Image?
Are the structures in the pancreas cystic solid?
answer:
CoPVQA((D
CopVQAIC)
100
11.63
lot
23 13
Correct answer:
Cystic
20
7.81
100
18.98
CopVOA(M]:
50
6.52
20
16.97
DVQA
CFVOA
MMQ:
11.96
17.02
6.82
50
16.04
5.18
13.05
Question:
What part of the dog hidden from view? Face
Correct answer:
CopVOA(D)
CopVOAIC)
Which hemisphere of the brain the lesions located in?
eyes
Correct arswer: Right hemisphere
15:54
6.77
CapVGNIM):
Kidneys
DVDA
CFVOA
MMQ: Bilateral
head dog
31.45
legs
13.49
16.52
dog
12:15
cars
neck
Question:
apples same
CopVOADD
CopVOMIC)
What structure depicted?
9.32
no
46,96
yes
0.95
5.75
Correct answer: Brain
not suro
8.08
unknown
5:38
CopVOA/M):
DVDA
CFVQA
MMQ:
Mr weighted
61.88
18:55
14.02
unikownn
2.14
2.57
(a) VQA-CPv2
(b) VQA-RAD
Figure 12: Examples of cases that all models failed to remove biases. The green labels are the correct answers while the red labels are the incorrect answers.
approach is that it allows for the development of specialized modules that can be individually op- timized (leaming independently by Gumbel-max activation function in CopVQA), leading im- proved performance on specific subtasks. It also promotes reusability, as modular components can be shared or combined to address related tasks or domains. Furthermore, modularization in machine learning facilitates interpretability and explainabil- ity. Since each module focuses on specific as- pect of the task, it becomes easier to understand and analyze the contributions of each module to the overall cecision-maaing process. This trans- parency can be particularly valuable domains where interpretability is crucial, such as healthcare or autonomous systems.

---

10
5
6
100
4
no
yes
3
skateboard
surfbarrd
1
skiing
skateboar do connuurfinnuiinn fing
Train
Test
CopVOA(D)
DVQA
CopvOA(O)
CFVQA
Train
Test
CopvoA(D)
DVQA
CopvOA(C)
CFVQA
Train
Test
CopVOA(D)
DVQA
CopvOA(C)
CFVOA
(a) Is this
(b) How many people are in
(c) What is the person
Figure 13: Answers distributions on VQA-CPv2 the that all models successful.
sandici
organize
33
motorola
York
lady
20
motorola
no
lady
15
adidas
motorola la
Intemational
mirror
yes
15
23
13
23
unknown
male
woman
8
8
unknowndell nike
spoon
yes
dell nike
spoon yes
9
dell wilson
samsung
hp
wilson
man
temale
Spoon
10
man
male
10
nike apple wisoowilson
apple
wilson
man
man
2
apple
Train
Test
CopVOA(O)
DVQA
CopVOA(C)
CFVQA
Train
Test
CopVOA(D)
DVQA
CopVQArC)
CFVQA
Train
Test
CopVOA(D)
DVQA
CopVQA(C)
CFVQA
(a) Is there a... ?
(b) What number is ....
(c) What brand
Figure 14: Answers distributions VQA-CPv2 in cases where all models failed.


Reviews:
[{"review_id": "aZCXMmgwKy", "reviewer": "Reviewer_UGPV", "summary": "", "questions": "a. Some details of such model remains unclear: What kind of MoE models are involved in those two versions of CopVQA? \nb. In 6.2 \"Discussion on the debiased samples\", it claims to reduce biases from the texts in question. However, there is no statistics of such phenomenon: How many instances are there in which DVQA/CFVQA/MMQ is misled by \"Is this\" and how many of them are resolved by CopVQA.\nc. The training details are missing: How long does it take to train CopVQA on what kind of machine?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer's time and efforts in reviewing our work. \nWe are glad to see `innovative`, `effectiveness`, `enough case studies`, and `well-organized` in your reasons to accept. \n\n\n\n---\n> a. **Some details of such model remain unclear: What kind of MoE models are involved in those two versions of CopVQA?** \n\nThank you for your questions, and we are so sorry for the confusion on the MoE design.\n\nThe details of the MoE designs are described in Section 4.3 of the paper. We would also highlight them as well as provide some more information as follows:  \n\n\n\n- **Gating Model**: Hard Gating. We design the gating model as an MLP followed by a Gumbel-softmax with hard selection to select  one of the experts to process the given input. We use the Straight-through trick to train this gating model.\n- **Experts with Separate Parameters**: Each expert carries distinct parameters, which are randomly initialized.\n- We tune the **number of experts** for $\\mathcal{M}_I$ and $\\mathcal{M}_A$ for each dataset and baseline and provide the tuning comparison in Section 7.\n- In our main design, the gating model only **activates 1 expert** at a time. We have also analyzed the performance when activating multiple experts and presented the results in Section A.3. The analysis shows that activating one expert is always better, proving the importance of disentanglement in our design. \n- We implement the MoE by ourselves. The MoE architecture design strategy is the same for $\\mathcal{M}_I$ and $\\mathcal{M}_A$. \n\nWe hope the information above could clarify your concerns.\n\n---\n> **b. In 6.2 \"Discussion on the debiased samples\", it claims to reduce biases from the texts in question. However, there are no statistics of such phenomenon: How many instances are there in which DVQA/CFVQA/MMQ is misled by \"Is this?\" and how many of them are resolved by CopVQA.** \n\nThank you for your insightful comment.\n\nThe following tables indicate the number of instances and correctness in the format `# / Acc` in different questions and answers. The `Acc` evaluates the accuracy of these `#` answers,  which would clarify your concerns about, for instance, `how many of them are resolved by CopVQA`. \n\nAccording to the tables, we can see that CopVQA(D) is the model that solves most biased cases within a question category as it acquires the closest `#` compared to ones of the test set and high `Acc`, followed by CopVQA(C). \n\n\n| `Is this` ... | Train | Test | CopVQA(D) | DVQA | CopVQA(C) | CFVQA |\n|----------|----------|----------|----------|----------|----------|----------|\n| **# / Acc** for `yes` | 0/- | 11031/- | 10991 / 0.99 | 11084 / 0.92 | 10866/ 0.99 | 11071 / 0.91 |\n| **# / Acc** for `no` | 11276/- | 0/- | 1 / 0.0 | 165 / 0.0 | 71 / 0.0 | 46 / 0.0 |\n\n\n| `How many` ... | Train | Test | CopVQA(D) | DVQA | CopVQA(C) | CFVQA |\n|----------|----------|----------|----------|----------|----------|----------|\n| **#** / **Acc** for `2` | 15270/- | 0/- | 492 / 0.0 | 2731 / 0.0 | 655 / 0.0  | 13406 / 0.0 |\n| **# / Acc** for `1` | 0/- | 15768/- | 15590 / 0.97 | 13894 / 0.76 | 14577 / 0.91 | 1855 / 0.47 |\n\n\n\n\n\n\n---\n> **c. The training details are missing: How long does it take to train CopVQA on what kind of machine?**\n\nThank you for your question.\n\n\n\n\nIn the paper, we have provided information related to the training details. Particularly, the number of parameters, GPU model, and training time comparisons are in Table 4 in the Appendix and referred to in Section 6.1.\n\n\nIn addition, in the table below, we indicate training **hours** on a single NVIDIA RTX A6000 GPU (same as reported in the paper).\n|  |  CopVQA(D) | DVQA | CopVQA(C) | CFVQA | CopVQA(M)  | MMQ |\n|----------|----------|----------|----------|----------|---------|---------|\n| VQA-CPv2 / VQAv2 |  2.1 / 1.9 | 2.3 / 2.0 | 1.6 / 1.4 | 1.9 / 1.7 |  - | - |\n| PathVQA / VQA-RAD | - | - | - | - | 1.6 / 0.25 | 1.7 / 0.26\n\n\nWe hope the information above could help for your assessment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a. The quantitative results of the newly proposed framework indicate a new SOTA.\nb. Newly designed cognitive pathways are innovative and the corresponding loss function described such reasoning paths well.\nc. The qualitative analyses proves the effectiveness of such two-layer reasoning path way.\nd. Enough case studies explains the advantages of such method. \ne. Paper is well-organized.", "reasons_to_reject": "a. Some details of such model remains unclear.\nb. In 6.2 \"Discussion on the debiased samples\", it claims to reduce biases from the texts in question. However, there is no statistics of such phenomenon.\nc. The training details about training time and hardware requirements are missing.", "questions_to_authors": "a. Some details of such model remains unclear: What kind of MoE models are involved in those two versions of CopVQA? \nb. In 6.2 \"Discussion on the debiased samples\", it claims to reduce biases from the texts in question. However, there is no statistics of such phenomenon: How many instances are there in which DVQA/CFVQA/MMQ is misled by \"Is this\" and how many of them are resolved by CopVQA.\nc. The training details are missing: How long does it take to train CopVQA on what kind of machine?", "paper_topic_and_main_contributions": "This paper tried to improve the multimodal predictions in Visual Question Answering by emphasizing causal reasoning factors through a improved causal graph. The scope of this paper particularly focuses on improving OOD generalizability of current VQA models. \n\nThe main contributions of this paper are two-fold:\n1. Proposed a VQA model that improves both OOD generalization and benchmark performance.\n2. Enhanced causal reasoning factors in VQA with two-layer reasoning pathway, which shows novelty over previous causal VQA researches.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hwqtKU2XZK", "reviewer": "Reviewer_1oDT", "summary": "", "questions": "1. Could you provide the bias distribution of the VQA dataset and specify which biases are addressed, thereby effectively validating the model's effectiveness?\n2. The experiments are not detailed enough, including experimental settings, model parameters, training time, inference time, etc.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer's time and efforts in reviewing our work. \nWe are glad to see `interesting` and `better performance` in your reasons to accept. \n\n\n\n\n\n---\n> **Compared to sota pre-trained methods, there is a significant performance gap with this method. Does it have scaling and generalization ability?**\n\n\n\nThank you for your question.\n\nWe acknowledge the gap between our method and the SOTA baselines. However, in these comparisons, our proposed method takes much fewer parameters, no extra data and human involvement, and less training time. In particular, CopVQA(D) requires just 2 hours of training with a single GPU, whereas MMBS demands 2 days of computation across 2 GPUs. Remarkably, CopVQA(D) achieves an Overall accuracy score on VQA-CPv2 that is only 0.49 points lower than the state-of-the-art MMBS model.\n\nSeveral vital characteristics substantiate our method's capacity for scaling and generalization:\n- Our approach avoids the introduction of additional parameters than the number of parameters of baselines.\n- It can represent the model's selection, including expert assignment explicitly.\n- It exhibits compatibility with a diverse range of baseline models.\n\nConsequently, our method exhibits the potential for scalability to larger datasets and more complex problems while demonstrating generalizability by incorporating causal reasoning considerations.\n\nFinally, we acknowledge room for future work to employ CopVQA on pre-trained baselines to improve the performance further. \n\n\n---\n> **There is doubt about the effectiveness of the complex reasoning module, as the improvement in downstream tasks by the model is limited.**\n\nThank you for your discussion. \n\n\n\n\n\n\nIf we understand correctly, this review is about the `reasoning modules` and `improvement` in the general AI research. We would like to share our opinions as follows:\n- Many of the recent works improve the downstream tasks significantly by the involvement of reasoning modules, such as in dynamic modeling tasks [1,2], biology-related tasks [3,4,5], or in NLP [6,7,8,9]. \n- On the other hand, the downstream task improvements may not be the ultimate goal of complex reasoning modules. They can also enhance models' transferability/reusability [1,2] across tasks, making AI systems more versatile. Therefore, facilitating causality research is a needed step. \n\n\nThese points strongly drive our motivations to propose CopVQA further to facilitate the generalization and explainability via causal reasoning. \n\nThe effectiveness of complex reasoning modules is a topic of ongoing debate within the AI community. We would be happy to discuss further on this point with the reviewer in case we missed some angles.\n\n\n\n---\n> **The method section is too redundant, and the experiments are not detailed enough, including experimental settings, model parameters, training time, inference time, etc.**  \n\nThank you for your comment and question.\n\nIn terms of `The method section is too redundant`, we would greatly value the reviewer's insights into specific areas that appear redundant. Any detailed feedback on this matter would greatly assist us in enhancing the clarity and conciseness of our method section.\n\nIn terms of `the experiments are not detailed enough`, we have provided information related to the training details in the paper. Particularly, the number of parameters, GPU model,  and training time comparisons are in Table 4 in the Appendix and referred to in Section 6.1. \n\nAdditionally, in the table below, we indicate training **hours** on a single NVIDIA RTX A6000 GPU (same as reported in the paper).\n|  |  CopVQA(D) | DVQA | CopVQA(C) | CFVQA | CopVQA(M)  | MMQ |\n|----------|----------|----------|----------|----------|---------|---------| \n| VQA-CPv2 / VQAv2 |  2.1 / 1.9 | 2.3 / 2.0 | 1.6 / 1.4 | 1.9 / 1.7 |  - | - | \n| PathVQA / VQA-RAD | - | - | - | - | 1.6 / 0.25 | 1.7 / 0.26  \n\n\n\n\n\n\nFor more insights on the experiments, we have provided the analyses for observing the hyper-parameter sensitivity as our best efforts in the paper. We would like to list them as follows:  \n\n- Section 7 - Figure 7: Number of experts in each MoE module.\n- Section A.3 - Figure 9: Number of experts to be activated. \n\n\nWe hope this information clarifies your concern and helps for your assessment.  \n\n---\n> **Could you provide the bias distribution of the VQA dataset and specify which biases are addressed, thereby effectively validating the model's effectiveness?**\n\nSure. It is important to note that we are not allowed to provide the answer distribution as Figure 4 from the paper during this rebuttal phase due to constraints on attaching figures or links. Nevertheless, the explanation and analysis below comprehensively fulfill this purpose.\n\nFirst of all, we would like to clarify `which biases are addressed`. The biases that need to be overcome in the VQAv2 dataset (as well as the VQA task) are some answers account for a large proportion compared to other answers' proportions within a question category. The examples in the following part will further visualize the biased answers in the VQAv2 dataset.\n\n\n\nNext, regarding `Could you provide the bias distribution?`, in the following tables, we provide the indications of the number of instances and correctness in the format `# / Acc` in different questions and answers that contain biases. The `Acc` evaluates the accuracy of these `#` answers,  which would clarify your concerns about, for instance, exactly `which biases are addressed` to `validate the model's effectiveness`. \n\n\n\n\n\n\n\n\nLet's analyze this **first example** for questions starting with \"**Is this**\". Please notice that the VQAv2  is biased but not an OOD dataset, as its biases are similar in the train and test sets. \n-  `yes` accounts for 46%,  `no` accounts for 47%, and `inside` takes 0.19% in the training set.\n-  `yes` accounts for 45%,  `no` accounts for 46%, and `inside` takes 0.19% in the test set.\n\n\nIn this case, `yes` and `no` are biased answers. We want to test if CopVQA can overcome them and still predict the answer `inside`.\n- As illustrated in the table below, it is evident that CopVQA can predict low-frequency answers as `inside` with high accuracy,  while the baselines tend to overlook these less common cases.\n\n| `Is this` ... | Train | Test | CopVQA(D) | DVQA | CopVQA(C) | CFVQA | \n|----------|----------|----------|----------|----------|----------|----------| \n| **# / Acc** for `yes` | 7502/- | 3529/- | 3591 / 0.90 | 3623/ 0.82 | 3525 / 0.86 | 3741 / 0.79 |\n| **# / Acc** for `no` | 7656/- | 3620/- | 3632 / 0.89 | 3773/ 0.80 | 3597 / 0.84 | 3963 / 0.72 |\n| **# / Acc** for `inside` | 32/- | 15/- | 12 / 1.0 | 0 / NaN | 4 / 1.0 | 0 / NaN | \n\nMoving to the **second example**, with questions starting with \"**Which**\":\n-  `left` accounts for 16%,  `right` accounts for 18%, and `baseball` takes 1.4% in the training set.\n-  `left` accounts for 15%,  `right` accounts for 16%, and `baseball` takes 1.1% in the test set.\n\nIn this scenario,  `left`  and  `right`  are examples of biased answers. We aim to test whether CopVQA can successfully address this bias and still predict the answer  `baseball`.\n- As demonstrated in the table below, it is clear that CopVQA can generate low-frequency answers as `baseball` with high accuracy, while the baseline models often fail to capture these less common cases. \n\n\n| `Which` ... | Train | Test | CopVQA(D) | DVQA | CopVQA(C) | CFVQA |  \n|----------|----------|----------|----------|----------|----------|----------| \n| **# / Acc** for `left` | 803/- | 385/- | 392 / 0.84 | 443 / 0.32 | 392 / 0.76 | 594 / 0.29 |\n| **# / Acc** for `right` | 859/- | 396/- | 399 / 0.81 | 497/ 0.41 | 390 / 0.74 | 630 / 0.42 |\n| **# / Acc** for `baseball` | 67/- | 28/- | 24 / 0.95 | 1 / 1.0 | 19 / 0.94 | 0 / NaN |   \n\n\nTo conclude, these analyses demonstrate the ability of CopVQA to overcome biased answers. \n\n\nWe hope this information and analysis clarifies your question. \n\n\n\n---\n[1] Neural Production Systems: Learning Rule-Governed Visual Dynamics - Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, Yoshua Bengio\n\n[2] Reusable Slotwise Mechanisms - Trang Nguyen, Amin Mansouri, Kanika Madan, Khuong Nguyen, Kartik Ahuja, Dianbo Liu, Yoshua Bengio\n\n[3] Modeling in systems biology: Causal understanding before prediction? - Szilvia Barsi, Bence Szalai \n\n\n[4] DynGFN: Towards Bayesian Inference of Gene Regulatory Networks with GFlowNets - Lazar Atanackovic, Alexander Tong, Jason Hartford, Leo J. Lee, Bo Wang, Yoshua Bengio \n\n[5] Causal Concepts in Biology: How Pathways Differ from Mechanisms and Why It Matters - Lauren N. Ross \n\n[6] Measuring semantic similarity of clinical trial outcomes using deep pre-trained language representations - Anna Koroleva, Sanjay Kamath, Patrick Paroubek\n\n[7] Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning - Sumedh A Sontakke, Arash Mehrjou, Laurent Itti, Bernhard Schölkopf\n\n[8]  Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias - Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, Stuart Shieber\n\n[9] Counterfactual Generative Networks - Axel Sauer, Andreas Geiger", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of CopVQA is interesting，which formulates VQA as two layers of cognitive pathway to  boost the causal reasoning.\n2. CopVQA achieves the better performance on several VQA datasets.", "reasons_to_reject": "1. There is doubt about the effectiveness of the complex reasoning module, as the improvement in downstream tasks by the model is limited.\n2. Compared to sota pre-trained methods, there is a significant performance gap with this method. Does it have scaling and generalization ability?\n3. The method section is too redundant, and the experiments are not detailed enough, including experimental settings, model parameters, training time, inference time, etc.", "questions_to_authors": "1. Could you provide the bias distribution of the VQA dataset and specify which biases are addressed, thereby effectively validating the model's effectiveness?\n2. The experiments are not detailed enough, including experimental settings, model parameters, training time, inference time, etc.", "paper_topic_and_main_contributions": "The paper proposes Cognitive pathways VQA (CopVQA) to boost the causal reasoning in VQA to enhance the OOD generalization.  The paper validates the model's effectiveness on several VQA datasets.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Iwb5e9uAcL", "reviewer": "Reviewer_ecEB", "summary": "", "questions": "See above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer's time and efforts in reviewing our work. \nWe are glad to see `interesting`, `timely`, `strong results`, and `well-written` in your reasons to accept.\n\n\n\n\n\n\n\n---\n> **I like the analysis in B.2, but was curious if authors have identified any possible data contaminations in the test set of the results they have reported in the paper?** \n\nWe appreciate your attention to this crucial aspect.\n\nIn this work, we adhere to the pipeline established by prior studies, focusing on four datasets: VQA-CPv2, VQAv2, PathVQA, and VQA-RAD. It is essential to emphasize that, within the AI research community, there is no known history of reports regarding data contamination within these datasets.  \n \nWe hope this information clarifies your concerns. \n\n\n\n---\n> **In NLP, Use of 'instructions' have strongly improved generalization.** Wondering authors would like to discuss if they have considered using instructions in their setup? Some of the issues authors see in sec 6.2 and 7 may be solved via instructions.  \n\nThank you for your interesting suggestion. \n\nFirst of all, we would like to clarify that we interpret the \"Instructions\" as the \"Instructions Tuning\" [1] technique.\n\nNext, after careful consideration of the concept of \"Instructions Tuning\", we would like to offer the following insights:\n- We are thrilled to observe that the prevalent issue of generalization is receiving substantial attention within the research community.\n- It is important to note that our methodology does **not** incorporate LLMs,  domain generalization, and especially explicit task definitions. Consequently, we do not immediately perceive a strong correlation between our approach and the concept of \"instructions\".  \n- On an opposite note, we think that integrating the Cop concept into \"Instructions\" training could yield promising benefits based on the sets of experts and the reasoning flows. This integration might enhance generative capabilities within a specific task and facilitate the reusable modules (experts) across various tasks.\n\n\n\n\nFinally, we would really appreciate if the reviewer could help us figure out possible approaches in case we missed some angles.\n\n---\n> **How sensitive is the model to hyper-parameters?** I see a mention in the limitation section 'Require careful fine-tuning', but was looking for more details.\n\nThank you for your question.\n\nIn the paper, we provide the analyses for observing the hyper-parameter sensitivity as our best efforts. We would like to list them as follows:  \n\n- Section 7 - Figure 7: Number of experts in each MoE module.\n- Section A.3 - Figure 9: Number of experts to be activated. \n\nWe would really appreciate the reviewer suggesting other analyses in case we missed some angles.\n\n[1] Instruction Tuning with FLAN - Finetuned Language Models are Zero-Shot Learners - Google Research", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting and timely problem statement\n- strong results on medical data\n- I like cognitive pathways angle brought in to VQA\n- well-written paper and interesting analysis", "reasons_to_reject": "These are not 'reasons to reject', but some questions I have:\n- I like the analysis in B.2, but was curious if authors have identified any possible data contaminations in the test set of the results they have reported in the paper?\n- In NLP, Use of 'instructions' have strongly improved generalization (https://aclanthology.org/2022.acl-long.244.pdf, and https://openreview.net/forum?id=gEZrGCozdqR). Wondering authors would like to discuss if they have considering using instructions in their setup? Some of the issues authors see in sec 6.2 and 7 may be solved via instructions.\n- How sensitive is the model to hyper-parameters? I see a mention in the limitation section 'Require careful fine-tuning', but was looking for more details.", "questions_to_authors": "See above", "paper_topic_and_main_contributions": "This paper studies an important problem statement involving generalization in Visual Question Answering (VQA) that requires models to answer questions about images with contexts beyond the training distribution. The authors propose Cognitive pathways VQA (CopVQA) improving the multimodal predictions by emphasizing causal reasoning factors. CopVQA first operates a pool of pathways that capture diverse causal reasoning flows through interpreting} and answering stages. Next, they decompose the responsibility of each stage into distinct experts and a cognition-enabled component (CC). The two CCs strategically execute one expert for each stage at a time. Then they prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC, thereby emphasizing causal reasoning and supporting generalization. They experiment on medical data and find that CopVQA improves VQA performance and generalization across baselines and domains.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uDIpIJY37I", "reviewer": "Reviewer_YXHL", "summary": "", "questions": "Can you explain why do you add the L_M, L_U, and L_total as the final loss?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer's time and efforts in reviewing our work. \nWe are glad to see `interesting`, `first proposes`, and `achieves SOTA` in your reasons to accept.  \n\n---\n> **The implementation part is like an increment of [1]. And for the experiment part, I am willing to see more results of other baselines on PathVQA and VQA-RAD**\n\n\nThank you for your observation.\n\n\n\nFirst of all, we would like to clarify that our implementation does not seem `like an increment of [1]`.\n\nWe provide more details  as follows:\n- We extend the main ideas (`mediator` and `subtraction of flows`) from [1].\n- However, the implementation is based on each particular baseline. That means we do not solidly improve/implement based on the implementation of [1], but  implement our method multiple times on different baselines' code base for a fair performance comparison. \n\nRegarding the baselines of PathVQA and VQA-RAD, to our best knowledge, before our submission, PathVQA has 1 baseline, which is MMQ;  and VQA-RAD has 2 baselines, which are MMQ and CLIP. We have compared our performance to these baselines in the main text. \n\nWe hope this information could clarify your confusion.\n\n--- \n> **Can you explain why do you add the L_M, L_U, and L_total as the final loss?** \n\nSure. The main reason is that we follow the ideas from [1]. The following insights would explain the loss functions design:\n- Notice that we do not include $Z_{final}$ in any loss function.  We expect each component in $Z_{final}$  to capture the desired biases.   \n- By that reason, we train $Z_U$, $Z_M$, and $Z_{total}$  separately.\n- If we do not train them separately but, for instance, through a combined answer as $Z_{total}$, the individual component is not ensured to receive feedback (from the target answer) fairly and, thus, would not capture the desired biases well. \n- Although trained separately, these branches do not achieve good scores (as it is to capture biases and designed with limited capabilities) as visualized in Section 7 (Ablations study).\n\n\nWe hope this information could clarify your confusion.\n\n---\n[1] Niu, Yulei, et al. \"Counterfactual VQA: A Cause-Effect Look at Language Bias.\" 2021 CVPR", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The intuition from cognitive neuroscience about knowledge modularity and cognitive pathways is interesting\n- This work first proposes a causal reasoning method, CopVQA  that formulates VQA as two layers of cognitive pathways (i.e. interpreting and answering)\n- The proposed model CopVQA achieves SOTA for PathVQA dataset", "reasons_to_reject": "I am not an expert in causality, but I try my best to read this work, the implementation part is like an increment of [1]. And for the experiment part, I am willing to see more results of other baselines on PathVQA and VQA-RAD\n\n[1] Niu, Yulei, et al. “Counterfactual VQA: A Cause-Effect Look at Language Bias.” 2021 CVPR\n", "questions_to_authors": "Can you explain why do you add the L_M, L_U, and L_total as the final loss?", "paper_topic_and_main_contributions": "- The paper proposes CopVQA to improve OOD generation by enhancing causal reasoning that is compatible with diverse VQA baselines and domains\n- This work first formulates VQA as two layers of cognitive pathways\n- This paper achieve SOTA for the PathVQA dataset", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to enhance out-of-distribution (OOD) generalization in Visual Question Answering (VQA) by emphasizing causal reasoning through a novel construct, the Cognitive pathways VQA (CopVQA). This model promotes multimodal predictions and primarily works in two layers of cognitive pathways. The paper has garnered attention for its state-of-the-art (SOTA) performance on the PathVQA dataset and its innovative approach to introducing cognitive pathways into VQA. The novel two-layer reasoning pathway provides a fresh perspective on causal VQA research, especially among tons of LLM-related papers. This  work also provides a comprehensive qualitative analysis supporting the effectiveness of the two-layer reasoning pathway. Reviewers unanimously rated this work positively, but authors are encouraged to further improve the paper based on the suggestions given in the reviews.
Decision: Accept-Main