Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering
Authors: Jie Ma
Min Hu
Pinghui Wang
Wangchun Sun
Lingyun Song
Hongbin Pei
Jun Liu
Youtian Du
Venue: NeurIPS 2024 poster (2024)
Keywords: audio-visual question answering
bias elimination
debiasing
multimodality learning
Primary area: natural_language_processing
TL;DR: We systematically explore the bias in the AVQA task from the persepctive of model designs and model evaluations.

Abstract: Audio-Visual Question Answering (AVQA) is a complex multi-modal reasoning task, demanding intelligent systems to accurately respond to natural language queries based on audio-video input pairs. Nevertheless, prevalent AVQA approaches are prone to overlearning dataset biases, resulting in poor robustness. Furthermore, current datasets may not provide a precise diagnostic for these methods. To tackle these challenges, firstly, we propose a novel dataset, *MUSIC-AVQA-R*, crafted in two steps: rephrasing questions within the test split of a public dataset (*MUSIC-AVQA*) and subsequently introducing distribution shifts to split questions. The former leads to a large, diverse test space, while the latter results in a comprehensive robustness evaluation on rare, frequent, and overall questions. Secondly, we propose a robust architecture that utilizes a multifaceted cycle collaborative debiasing strategy to overcome bias learning. Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9.32\%. Extensive ablation experiments are conducted on the two datasets mentioned to analyze the component effectiveness within the debiasing strategy. Additionally, we highlight the limited robustness of existing multi-modal QA methods through the evaluation on our dataset. We also conduct experiments combining various baselines with our proposed strategy on two datasets to verify its plug-and-play capability. Our dataset and code are available at <https://github.com/reml-group/MUSIC-AVQA-R>.

Look, Listen, and Answer: Overcoming Biases for Audio Visual Question Answering
Jie Mat Min Hu Pinghui Wang', Wangchun Sun', Lingyun Song',
Hongbin Peil, Jun Liu", Youtian Du
MOE KLINNS Lab, Xi an Jiaotong University
2 China Mobile System Integration Co.
School of Computer Science and Technology, Xi'an Jiaotong University
School of Computer Science, Northwestern Polytechnical University
Equal Contribution
Corresponding Author
Abstract
Audio- Audio-Visual Visual Question Answering (AVQA) is complex multi-modal reasoning task, demanding intelligent systems to accurately respond to natural language queries based on audio-video input pairs. Nevertheless, prevalent AVQA ap- proaches are prone to overlearning dataset biases, resulting in poor robustness. Fur- thermore, current datasets may not provide precise diagnostic for these methods. To tackle these challenges, firstly, we propose a novel dataset, MUSIC-AVQA-R, crafted in two steps: rephrasing questions within the test split of public dataset (MUSIC-AVQA) and subsequently introducing distribution shifts to split questions. The former leads to large, diverse test space, while the latter results in a compre- hensive robustness evaluation on rare, frequent, and overall questions. Secondly, we propose a robust architecture that utilizes a multifaceted cycle collaborative debiasing strategy to overcome bias learning. Experimental results show that this architecture achieves state-of-the-are performance on MUSIC-AVQA-R, notably obtaining significant improvement of 9.32%. Extensive ablation experiments are conducted on the two datasets mentioned to analyze the component effectiveness within the debiasing strategy. Additionally, we highlight the limited robustness existing multi-modal QA methods through the evaluation on our dataset. We also conduct experiments combining various baselines with our proposed strategy on two datasets to verify its plug-and sagg-add-payy capability. Our dataset and code are available https: // /////gttaa github. com/reml www.rremlooggeee group/MUSIC-AAA
Introduction
Humans possess the extraordinary capacity to seamlessly integrate auditory and visual cues, effec- tively establishing a cohesive relationship between visual and auditory stimuli [1]. Audio-Visun Question Answering (AVQA) [2-5, 3] seeks to enable intelligent systems acquire this capability and produce answers based on provided natural language questions. It requires the system to learn high-order interaction representations of the concepts encompassed with audio, video, and language modalities. As is known to us [6-8], the high-level reasoning ability of the system mainly relies on large-scale data that does not contain harmful biases or statistical regularities.
Nevertheless, completely avoiding the negative bias in datasets seems challenging. Previous studies [9-12] in visual and extractive QA have investigated the bias from the perspective of changing answer distributions and human-in-the-loop adversarial attacks. Drawing inspiration from these works,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

STG
Yes
Train
Yes
Is the ukulele in the video always playing?
O urs
Predefined Question Template: the <Object> in the video always playing?
STG
X
Yes
Test
No
the flute the video always playing?
Ours
Figure I:
The question in current AVQA datasets is generated by a limited set of predefined templates, which may not be in line with the real-world scenario. Our findings indicate that existing methods [5, 1] such as STG [4] are not robust, which may be attributed to excessive bias learning, such as memorizing statistical regularities between critical question words and answers.
several open questions are proposed for the AVQA task, concerning model evaluations and model designs.
Question 1: have existing datasets comprehensively measured model robustness? The questions in the current AVQA dataset [13, 5, 14, 3] are generated by limited set of predefined templates, such as the 33 templates in the MUSIC-AVQA dataset [4]. Fig. 1 shows the samples the training and test split, which produced using predefined template. The observed difference mainly stems from a single word, leading to a limited vocabulary size of only 93 words. This has the potential to deviate from real-world scenarios. Moreover, current datasets cannot reflect the performance or less common samples, which is an important indicator for evaluating model robustness [15, 16].
Question 2: have existing methods overcome the data bias? We found that existing methods [5, 1, 17, 18] such as STG [4] are brittle for the question with rare answers. This may attributed to memorizing the statistical regularity between critical question words and answers, such as the connection between "Is", "Playing", and "Yes". Specifically, the experimental result [4] shows that STG achieves accuracy of 54.09% on the test split of MUSIC-AVQA only given questions.
In this paper, we present the development of a novel dataset called MUSIC-AVQA-R, which aims to address the first question precisely. The dataset complements MUSIC-AVQA [4] and provides a more refined diagnostic for current AVQA methods. To preserve the inherent bias, we maintain the original training and validation splits of the MUSIC-AVQA dataset. In contrast, we employ collaboration mechanism to rephrase the question in the test split. This ensures diverse and natural question forms while remarkably expanding the number of questions from 9,129 211 211,572. We introduce distribution shift based on answer distributions specific question types. This allows us to measure performance on both frequent (in-distribution) and rare couttof-distribution)) data simultancously.
To tackle the second question, we propose robust framework that applies a Multifaceted Cycle Collaborative Debiasing (MCCD) strategy. Specifically, the strategy introduces a novel optimization objective, which enlarges the distribution difference between uni-modal (question, audio, and video) and multi-modal logit. By doing so, our model becomes less prone to learning biases from individual modalities. Intuitively, we cannot choose the correct answer based on only one modality. Hence, logit distribution of each modality, thereby promoting
MCCD employs cycle guidance to constrain the similarity of uni-modal logit distribution. The experimental results demonstrate that our framework yields significant improvements on both datasets, with a particularly notable enhancement of 9.32% observed the MUSIC-AVQA-E dataset.

---

In summary, our contributions are fourfold: (1) We propose novel dataset MUSIC-AVQA-R and a set of respective evaluation metrics. This enables us to thoroughly evaluate the reasoning behavior of AVQA models and characterize their generalization capabilities in in- and out-of-distribution scenarios. (2) We present an AVQA architecture that incorporates the MCCD strategy to overcome training biases. To the best of our knowledge, this is the first work to systematically explore biases in the AVQA task from model evaluations as well as model designs. (3) We conduct extensive experiments MUSIC-AVQA and MUSIC-AVQA-R to verify the effectiveness and superiority of our proposed architecture and debiasing strategy. (4) We evaluate 13 recent multimodal QA methods on the proposed dataset and show their limited ability generalize not only in-distribution scenarios but also in out-of out-of-distribute distribution situations.
2 Related Work
2.1 Model Robustness Evaluation
Despite the notable achievements of QA datasets [19-23, 3J, they suffer from biases, resulting in incomplete evaluations. In recent years, numerous studies have tackled this issue from various perspectives [24-27].
One avenue of research [12, 28, 29] reorganizes existing datasets. thereby making the distribution between training and testing splits significantly different or even reversed. The reorganized datasets reflect the performance in the off-odi-itriiuttinn situation but lack measurement in the in-distribution scenario. To this end, GQA-OOD [30] introduces the distribution shift in both the validation and test splits to assess visual QA models in both scenarios simultaneously. Nevertheless, the number of questions in the GQA-OOD test split only 2,796, which may not reflect the real generalization ability of visual QA models due to the presence of limited number of testing samples [31]. Inspired by the adversarial attack, another line of works [32, 33] regard the dataset construction as a game played by two parties: a human annotator and well-trained model. Only samples generated by humans that successfully attack the model are incorporated into the dataset. In addition, there exists another line of work [14] that complements videos and questions to obtain balanced training data.
Different from the mentioned works, our dataset, MUSIC-AVQA-R, not only prioritizes question diversity but also considers the volume of test samples. This enhances the precision and comprehen- siveness of robustness evaluation. Moreover, we recognize the formidable challenge of obtaining completely pure training data. As such, we opt to retain the inherent bias present in both the training and validation splits. Our primary objective to inspire the community to enhance model robustness through the implementation of debiasing strategies, rather than striving for balanced training data. Remarkably, to the best of our knowledge, our dataset is the first AVQA dataset explicitly designed for robustness evaluation.
2.2 Bias Dependency Elimination
A variety of debiasing QA methods [34-37] have been proposed to overcome bias memorization. These methods can be divided into four classes [24]: ensemble learning, data augmentation, con- trastive learning, and answer re- re-ranking.
Ensemble learning methods [38, 28, 39, 6, 40, 41] typically leverage combination of a bias learner and a vanilla QA model to comprehensively predict answers. Data augmentation methods [42-45] generate additional question-answer pairs to balance the data distribution. Based on the positive and negative sample generation, contrastive learning- based methods [46-48] strive to learn an embedding space where similar sample pairs are closely clustered while disparate ones are distinctly separated. Consequently, the vanilla QA method is optimized jointly through contrastive and QA losses. Answer re-ranking methods [34, 49-52] primarily focus on reordering the answers predicted by the vanilla QA model to enhance context comprehension, such as vision grounding.
To the best of our knowledge, COCA [53] is the only work to mitigate the bias learning in the AVQA task, which first employs causal regularization to intervene bias-irrelevant causal effects and then introspects predictions. Unlike the mentioned works, which only consider language biases, our method considers audio, vision, language biases, and their collaboration. The proposed MCCD strategy features plug-an aareg-and-pp capability, enhancing the debiasing potential of baseline methods.
3

---

3 Dataset Creation and Analysis
We introduce the first dataset, MUSIC-AVQA-R, to evaluate the robustness of AVQA models. The construction of this dataset involves two key processes: rephrasing and splitting. The former involves the rephrasing of questions in the test split of MUSIC-AVQA, and the latter is dedicated to the categorization of questions into frequent (head) and rare (tail) subsets.
3.1 Rephrasing
The questions within the existing dataset [5, 4] are formulated using a restricted collection of pre- defined templates. To augment diversity and reality, we employ rephrasing tool' to rephrase each question 25 times. To ensure the rephrasing quality, three annotators participate in a verification process where their consensus through voting is required. They are all senior students in the field of information science, with one specializing in computer science and the other two in automation. Their extensive professional background equips them with the ability to assess whether the above rephrasing fulfills the requirement. Rephrasings are incorporated into the dataset only when two or more individuals validate the quality of the modifications. According the statistics, 92.4% of rephrasings pass this validation, and the Fleiss Kappa value used to measure vote consistency is 0.839. Please see details in Table 4 of Appendix B. These results strongly suggest exceptionally high quality of the rephrasing efforts. Fig. 2 illustrates the distribution of rephrased questions based on their initial three words. We see that rephrasing questions have various formats, and the comparison between the two datasets is shown in Fig. 6 of Appendix B. The vocabulary size of dataset is 465, which is 5x larger than MUSIC-AVQA AVQ.. These results indicate that dataset is more in line with the real-world scenario. Furthermore, an expansion of the question count within the test split has been implemented, escalating from 9,129 211,572 questions. This substantial increase in the volume of test samples enhances the precision of evaluations for AVQA methods.
3.2 Splitting
To provide a precise diagnostic for AVQA models, we introduce a distribution shift based on answer distributions of specific question types, following [30]. Guided by this distribution, we categorize rephrased questions into head and tail, enabling the assessment of in-distribution and out- of-distribution performance, respectively. We also utilize the overall performance to assess the model effectiveness on the entire test split.
iooud
thie
50
Specifically. to characterize the distribution shift, we first utilize the annotation for question types, including "Ex- istential", "Location", "Counting", 'Comparative", and "Temporal", to group questions. Fig. 3(a) illustrates the answer distribution of the "Temporal" questions within the AVQA task. We see that the answer presents long-tailed distribution. The distribution of other types is given in Appendix B. It is essential to note that MUSIC-AVQA en- compasses three tasks: audio QA, visual QA, and AVQA.
the
does
how minny
Figure 2: Distribution of rephrasing questions based on the first three words.
Next, we characterize the answer balance using Shannon entropy, expressed H (A)
E p(a:) p(a.), where ((A) is the entropy of answer set for a certain question type, N is the number of answer classes, and p(a,) represents the probability of answer class i. It is important to note that the entropy depends on the number of answer classes, which exhibits significant variability across different question groups. To facilitate meaningful comparisons, we normalize ((A) of each group by log(N): H(A) = H(A)
[og[N]> with
log(N) representing the entropy of a uniform distribution of size N. Refer to Appendix C for detailed
proof. Thus, the normalized entropy H (A) indicates the proximity of the distribution H (AA) to

---

3000
30003
2000
Namberr
1000
Numbers
20003
12
10029
500
9020
0
d
Coontig
Comparate
Cooning
Locaion
Bawerddd
Looptin
Churtin
Corpanahi
Tempera
Answers
Question Typee
(a) Answer distributions of "Temporal" questions in the AVQA task.
(b) Statistics of head and tail samples.
Statistics visualization for MUSIC-AVQA-R, μl(a) is the average number of answers in group. The dark color on the right denotes the number of head samples, while the light-colored area denotes that of tail samples.
Figure 3:
uniform distribution. We preserve the group with a normalized entropy below a threshold of 0.9, which aims at selecting imbalanced groups.
Finally, we categorize the samples into head and tail classes. We define the tail class class with J0:1 S 1.2)((a) following [30], where la;] represents the number of samples belonging to answer class and p(aa) denotes the average sample count for group. Consequently, the tail samples are rare, while the head samples are more prevalent within group. Fig. 3(b) illustrates the statistics of head and tail samples across various groups within each task.
4 Method
To mitigate bias learning, we propose robust AVQA architecture that integrates a multifaceted cycle collaborative debiasing strategy. Fig. 4 illlsstraes an overview of our proposed architecture. It first learns the uni-modal and multimodal representations using a pre-trained model. Then, the architecture utilizes distinct bias learners to capture uni-modal biases. Finally, collaborative debiasing strategy leveraged to magnify the disparity between fusion logit and bias logit, obtained based on multi- modality and uni uni-modality representations, respectively. Meanwhile, cycle guidance mechanism is employed to maintain the similarity between bias logit. The aforementioned procedure is only carried in the test split of MUSIC-AVQA. Consequently, our proposed dataset, MUSIC-AVQA-R, allows for a more precise and comprehensive evaluation of models handling data biases.
Uni-modal Embedding. Given AVQA sample comprising video sequence and corresponding question, we initially partition the sequence, consisting of visual and audio tracks, into non- overlapping pairs of visual and audio segments, denoted as { (V, At)E where each segment spans one second. Subsequently, a distinct embedding layer is employed to acquire uni-modal embeddings. Specifically, employ a pre-trained VGGish model [54] with fixed parameters, which is VGG-like audio processing network, to obtain an audio embedding vector. For video embedding vectors, employ pre-trained ResNet-18 with fixed parameters the frames. The VisualBert model [55] is applied to obtain a word-level question embedding vector. To ensure dimension matching, distinct linear layers are applied to the aforementioned vectors, resulting in uni-modal embeddings A Ve E R x76 and Q E R4x768 where is the question length.
Uni- and Multi-modal Representation. We leverage VisualBert to obtain both uni-modal and multi- modal representations, represented as A,, Vū, Q M E R 768 In the case of uni-modal learning, we exclusively input the aforementioned uni-modal embeddings to VisualBert. For multi-modal learning, we treat question embeddings as queries, concatenate video and audio embeddings as context, and leverage VisualBert perform multi-modal interaction. Then, we apply linear projection the representation to obtain the mudti-modality logit y. E R-42 where 42 denotes the number of possible answers.
Uni-modal Bias Learning. AVQA may involve various harmful uni-modal biases, encompassing biases associated with audio, video, and language, respectively. To capture these uni-modal biases,
5

---

Input
Embodding
Un(Multinodal Reprosentation
Learning
Collsborative Debiasing
Video
y
y
Audio logit
Video
logh
fom
Audio Bles
logit
Audic Emoedding
Aucio Ecodder
Learmerr
Qhan ion BIDG
y
Levirne!
Enargg dissimilenty
the tuto video always playing?
[Cnsummys
Mulifaroted dabiocing, guidance, AVQA
Quection Emoodding
Quection Enooddr
M
Figure 4: Robust AVQA architecture to overcome bias learning, Our MCCD strategy is plug-and-play allowing seamless integration with other AVQA methods.
we utilize a bias learner that takes only one of the three modalities as input. Specifically, distinet non-linear multi-layer perceptron layers serve as the learners, producing the corresponding logit y..y y E 1R on the answer space. should be noted that these bias learners are removed during the testing stage.
Collaborative Debiasing. To eliminate bias learning, we propose a mulifaaceed cycle collaborative debiasing (MCCD) strategy. It first reduces the bias impact from multiple views by enlarging the dissimilarity between uni-modal and multi modal logit. This discrepancy enlargement La implemented by the joint inverse distance:
K
1
La
(I)
3K
1=1
where K is the batch size, is used to balance optimization, di denotes the Euclidean distance between audio logit and multi-modality logit, dj represents the distance between video logit and multi-modalig logit, de is the distance between question logit and multi-modality logit, and 1e-5 added to the denominator to avoid division by zero.
Intuitively, relying solely on one modality for answer prediction may result in similar logit distribu- tions. Therefore, MCCD employs cycle guidance to constrain the distribution of uni-modal logit. This guidance Le is implemented by the Kullback-Leibler divergence:
3 (Lau Lav flvg),
Le =
where / is the factor to control weight, Lag = K Σx= 89 (log 8 log ) denotes the relative entropy between the question y and audio logit 5 Lav = t 2i-1 8: (log 8 yo) is the relative entropy between the audio y and video logit y., and liv K ΣE PY (log 9 represents the relative entropy between the video y.¹ and question logit y.
Finally, utilize the summation of La. Le and La to optimize the parameters of our method. C. - オे Eie yf log yf is the loss of answer prediction that is regarded as multi-clasifiiationn problem, where y.. yf denote the one-hot answer label and logit multi-modality fusion, respectively. The training details are shown in Appendix A.
5 Experiments
5.1 Dataset and Evaluation
MUSIC-AVQA [4], which contains training, validation, and testing splits with 31,927, 4,568, and 9. 191 QA pairs, is developed by gathering questions for 9,288 musical performances. The questions are produced by limited set of pre-defined templates. The videos, sourced from YouTube,, include solo performances, ensembles of the same instruments, and ensembles of different instruments. This datasct consists of three tasks: audio QA, visual QA, and AVQA. The standard accuracy is used to

---

evaluate model performance on the mentioned tasks. To comprehensively assess model robustness, we conduct rephrasing and splitting on the test split, expanding the question count from 9,129 to 211,572. Owing to the introduction of distribution shift, our proposed dataset provides three metrics: head accuracy, tail accuracy, and overall accuracy, to evaluate models precisely. The test split comparison between MUSIC-AVQA and MUSIC-AVQA-R is shown in Appendix D.1. We can see that the latter exhibits a larger test sample space.
Table 1: Experimental results (%) the MUSIC-AVQA test split. EXIST, LOC, CNT, COMP, and TEMP, which are question types, denote "Existential", "Location", "Counting", "Comparative", and "Temporal", respectively, Avg. denotes the average accuracy.
Method
MCCD
Andio QA
Vicual QA
AVQA
All
CNT
COMP
Avg.
CNT
LOC
Avg.
EXIST
LOC
CNT
COMP
TEMP
Avg.
Avg.
FUNLSTM
70.45
66. 22
68.58
63.89
46.74
55.21
82,01
46.28
59.34
62.15
47.33
60.06
60.34
70.99
66.50
69.34
66.08
59.02
62.51
83,50
57.17
60.47
61.58
57.54
64.11
64.61
CONVLSTM
72.76
74.07
69.53
68.89
71.57
72.15
69.59
67.47
58.12
54.56
63.79
60.94
82.91
82.69
56.09
50.81
63.03
62.13
62.03
60.27
55,11
51.58
62.24
63.87
63.65
65,21
BiLSTM Altn
70.35
47.92
62.05
64.64
64.33
64.48
78. 39
45.85
56.91
53,09
49.76
57.10
59.92
68.24
$4.88
63.31
61,65
55.92
58.75
79. 15
41.9%
55.02
49,41
49.15
55.18
$7.56
HCAIt
V
69.52
53.37
54.91
63.56
63.99
65-47
66.37
64.74
78. 64
47.28
61.11
59.97
S5.86
55.72
60.03
62.30
61.90
MCAN
77.50
55.24
69.25
71,56
70.93
71.24
80,40
54.48
61.91
57,22
47.57
61.58
65.49
56.57
70.27
71,93
71.18
65.77
65.74
GRU
72.21
66.89
70.24
67.72
70.11
68. 9.3
81.71
59, 44
62.64
61,88
60.07
65. 18
67.07
73.35
66. 16
70.70
67.25
71.43
69.36
81,98
60.11
63.08
62.76
61.19
65.84
67.63
HCRN
X
68.59
50.92
62.05
64.39
61.81
63.08
54.47
41.53
53.38
52.11
47.69
50.26
55.73
72.17
65
69.40
67,42
60)
64,08
79 66
48.70
65,14
61,22
55,72
60.40
64,20
HMB
70.61
80.30
66.45
72.96
62.29
69.03
68.76
69.31
69.03
80.77
52.61
62,92
63.03
60.71
64.19
66.33
PSAC
75.64
66.06
72.09
68.64
69.79
69.22
77.59
55.02
63.42
61.17
59.47
63.52
66.54
75.02
66.66
71.57
69.
53.04
61.98
61.13
57.66
62.85
65.15
AVSD
72.07
72.41
61,90
63.977
68.52
69.09
67.42
67.39
74.553
74.19
70.83
71.02
81.61
59.13
58.79
63.08
63.89
61,52
62.49
63.50
61.41
65.49
65.82
67.44
67.77
LAVIT
74.36
64.56
70.73
69.39
75.65
72.56
81.21
59.33
64.91
64,22
63,23
66.64
68.93
75.12
65.49
71.57
70.43
76.73
73.62
81.38
60.33
65.30
62.49
62.29
66.42
69.24
78.18
67.05
71.56
76.38
70,80
71.52
COCA
X
79.35
66.50
74.61
72.35
76.08
74.24
83.50
64.02
70.99
63.40
64.48
69.47
71.64
Ours
83.87
71.04
79.14
79.78
76.73
78.24
80.87
51.63
71.46
64.67
64.60
67.13
72.20
LAViSH
81.32
74.67
79.20
80.57
79.
72.96
66.
70.57
73.76
80.33
63.80
74.24
78.86
81.31
80.10
73.91
65.22
73.91
64,31
66.55
70,80
73.87
5.2 Implementation Details
During the data pre-processing stage, audio and video are sampled rates of 16 kHz and I fps, respectively. In the uni-modal embedding module, we employ ResNet-18 18 and VGGish to obtain 512-dimensional and 128 728-dimensional embeddings of the visual and audio segments, respectively. In the uni-modal bias learning module, the hidden layer size of the bias learnr is set to 768. In the collaborative debiasing module, we set the factors a and equilibrium, respectively. During the training stage, the initial learning rate is set to 3e-5, decaying by 0.5 every 20 epochs. The maximum epoch and batch size are set to 150 and 64, respectively. We use the Adam optimizer to train our architecture and save the model that achieves the best performance on the validation split. The experiments for all methods, except LAVisH, are conducted using single NVIDIA Tesla V100 GPU. The experiment for LAVisH is run on two NVIDIA Tesla A100 GPUs. The other details are shown in Appendix D.2.
le-2 and 3e-1 for optimization
5.3 Baselines
We select 14 previous state-of-the-art multi- modal QA methods as baselines to verify the effectiveness
of the proposed architecture and investigate the robustness of these methods. Audio QA methods: FCNLSTM [56], and CONVLSTM [56]. Visual QA methods: GRU [57]. BiLSTM Attn [58], HCAttn
[59], and MCAN [60]. Video QA methods: HME [61], PSAC [62]. and HCRN [63]. AVQA methods:
AVSD [13], LAVIT [5], STG [4], COCA [53] and LAViSH [1]. We abstain from reassessing COCA on the MUSIC-AVQA- dataset due to its lack of publicly available code. The baseline introductions are shown in Appendix D.3. Due to the particularly slow computation speed of STG, we do not
conduct experiments with MCCD Due to computing power limitations, we reevaluate LAVisH with batch size of 2.
7

---

Table 2: Experimental results (%) on the MUSIC-AVQA-R test split. The question types, such as CNT and COMP, are introduced in Table H and TT denote the head and tail accuracy. There is no publicly available code for COCA.
QA
QA
AYQA
MCCD
CNT
COMP
CNT
LOC
FAIST
LOC
CNT
COMP
TEMP
Ang,
11
T
1E
T
T
11
T
14
T
11
T
11
71.67
62.51
34.44
61.13
5.66
92.40
00.09
44.55
54.55
CONVLSTM
68.38
30.22
41.14
41.58
67.550
05.39
52.00
52.03
61.45
62.11
9.17
9.50
53.64
56.17
49.65
50.53
59.61
50.66
84.52
83.31
55.29
46.45
56.52
59.90
56.33
59.13
9.18
7.82
45.51
45.29
12.5
72. 7M
41.56
41.36
45.12
45.40
55.20
55.74
BUSSTM Ano
33.08
46-12
91:51
73.96
0.00
42:01
87.51
21:14
15.15
41:25
60.84
17.98
49.15
HCAR
61.67
62.50
41.63
41.45
$5.59
47.42
50.65
56.52
5.55
67.31
$2.00
53.16
66.57
56.52
59.57
38.25
42.53
59.36
12.42
57.59
48.91
52.01
32.54
59.55
$3.39
51.50
CRU
68.00 02
69.04
46.63
46.00
$0.31
56.20
63:77
59.61
06.24
64.37
13.30
11:3
57.85
63.55
57.59
57.56
83.00
76.90
64.76
54.16
41.05
43.36
43:61
63.36
60.71
57.39
18.05
18.30
57.19
53.93
57.58
03
30.53
11.00
40.67
35.25
55.70
$5.21
60.16
64.55
61.81
64.76
50.57
HORN
55.53
51.96
53,31
40.21
43.42
47.37
32.44
41.13
41.87
23.55
20.71
$7.70
39.00
51.77
50.99
46.38
41.51
05.45
58.40
55.05
3662
06.33
42.72
54.58
$4.39
20.00
19.57
33.33
34.50
36.87
15.14
40.15
40.47
44.15
44.00
42.57
43.92
63.60
53.95
54:07
9.229
50.95
58.00
65.0
65.49
06.3
63.18
17.18
50.57
33.05
51.65
PSAC
13.01
35.14
56.08
$2.26
64.70
44.45
05.55
52.34
22.15
76.40
72.06
00.50
60.70
58.97
59.56
52.35
41.18
41.9%
52.30
49.7%
53.25
18.85
42.60
39.13
$4.53
58.92
26.06
25.81
45.58
$1.31
50.45
AVSD
X
54.10
$5.87
47.84
40.18
65.41
62.61
47.79
K1 01:32
00.34
10,07
7.41
74.75
61.43
66.28
61.56
70.40
31.85
33.00
20.35
54.55
46.21
33:31
3.06
$1.99
99.06
56.00
67.88
40.14
44.08
41.52
52.35
$3,89
LAWI
45.05
45.04
6862
17.00
6661
57.57
5392
54.06
15.35
45:91
STG
56.40
41.48
02.38
57.59
5936
12.44
64.31
16.00
13.35
71.36
35.35
00:49
08.31
1.41
53.30
62.4 64
40.25
56.15
$3.80
LAVIS
61.73
74.00
07
43.99
64.15
65.06
66.36
51.15
63.53
71 96
40.50
11.13
18.04
70.21
64.73
16.00
83 17
71.83
65.68
79.66
41.76
90.11
61:40
41.20
09:85
61.77
76.51
14.87
59.25
16.31
65.0 40
55.0 10
41.84
59.19
41.85
46.25
$7.65
59:25
5.4 Comparison on MUSIC-AVQA
We conduct experiments on the MUSIC-AVQA test split to validate the effectiveness of the proposed architecture. The results are presented in Table 1. All methods, except LAVisH, utilize ResNet-18 to encode visual features, whereas LAVisH employs stronger models such as VIT [64] or Swin [65] to acquire visual representations. We first analyze the comparison under the same visual encoding conditions. Notably, compared with COCA, our architecture obtains significant improvements of 4.53% and 4% in the audio and visual QA tasks, respectively. It also achieves the best performance in architecture obtains a new state-of-the-ar result of 72.20% on the
the AVQA task. Furthermore, whole question. It is worth mentioning that visual QA methods, like GRU, and video QA methods, such as HME, also exhibit competitive results in the AVQA task, despite lacking one modality input. We also observe that LAVisH. proposed based on STG, introduces trainable parameters into robust visual encoders, thereby achieving superior results compared to methods employing weaker visual encoders. The results on this dataset can demonstrate the efficacy of these methods to some extent. However, MUSIC-AVQA lacks refined and precise evaluations due its inherent shortcomings that are introduced in Section 1. Consequently, it is insufficient to evaluate these methods only on this dataset.
5.5 Robustness Evaluation
We conduct experiments on the MUSIC-AVQA-R test split to explore the robustness of the afore- mentioned methods. Their released codes are employed to conduct this experiment. The results are presented in Table 2. Several crucial insights arise when combining the results from this table with those from Table 1. Firstly, audio QA methods, such as CONVLSTM, showcase competi- tive robustness and even achieve the highest tail accuracy on EXIST questions within the AVQA task. Secondly, the visual QA method MCAN demonstrates noteworthy robustness by obtaining the second-best performance on the test split. Thirdly, the video QA baseline experiences a relatively significant performance degradation, with PSAC, for instance, declining by 16.45%. Notably, HCRN exhibits the lowest performance on both datasets, indicating its poor robustness. Furthermore, the results of AVQA baselines lag behind other types of QA methods like HME, suggesting that their strong performance on the MUSIC-AVQA dataset may rely on memorizing statistical regularities between input modalities and answers. Ultimately, our architecture outperforms others on the test split. Benefiting from the MCCD strategy, it attains the highest head (in-distribution setting) and tail (out-of-distry dintr---dittiiuuionn setting) results across various question types, providing further evidence of its superior robustness. We also show the overall accuracy of these methods on each type of question. Please see the details Appendix D.4. It can be seen that our architecture achieves the best overall accuracy on each type of question.
8

---

To validate the Boug-and-laa capability of MCCD, we conduct extensive experiments using the baselines+MMCCD on the aforementioned datasets. The results are presented in Tables and 2. We observe that MCCD consistently improves performance across most methods on MUSIC-AVQA (9 out of 13) and MUSIC-AVQA-R (11 out of 13), respectively. This underscores the robust debiasing capability of MCCD in plug-and-play manner.
5.6 Ablation Study
To verify the debiasing effectiveness of MCCD, we conduct extensive ex- periments on both the test split of MUSIC-AVQA and MUSIC-AVQA-
Table 3: Ablation results (%) on the test split of MUSIC-
AVQA and our dataset. AQA and VQA denote audio QA, and visual QA, respectively. df#)) is the distance between the (#) logit and the multi- logit. MD: multifaceted debiasing. CG: cycle guidance.
R. The results are shown in Table 3. Firstly, we validate the contribu- tion of the component within multi- faceted debiasing. It can be seen that removing the component will lead to an overall performance improvement in some aspects of MUSIC-AVQU while resulting in a significant de- crease in our dataset. This observa-
Method
MUSIC-AVQA
MUSIC-AVQA-R
AQA
VQA
AVQA
All
AQA
VQJ
AVQA
H
All
Ours
79.15
78.24
67.13
72.20
74.76
72.76
61.34
72.24
59.39
66.95
w/o 6
79.64
77.62
67.52
72.34
74.89
72.26
59:53
71.72
57.47
63.86
d
79.27
78.61
67:23
71.29
70.18
58.08
08.34
57.43
63.25
77.22
77.50
67.31
16.76
70.92
66.05
61.75
MD
77.54
66.3
13.97
70.09
Weo
78.77
78.65
67.50
72.45
75.42
71.72
59.68
71.40
57.98
68.87
tion strongly supports the debiasing efficacy of these components. Secondly, we verify the overall contribution of the multifaceted debiasing. It can be seen that the performance decrease of 0.72% and 2.15% occurs in both datasets, respectively. Finally, we validate the contribution of cycle guidance. We see that this model variant obtains the best performance there was a noticeable performance degradation in our proposed dataset. In summary, each component plays a distinctive role in the debiasing process, which is further demonstrated by the performance degradation on the head and tail samples.
the MUSIC-AVQA dataset. However,
5.7 Sensitivity and Qualitative Analysis
We employ the control variable method to perform a sensitivity analysis on the weight-controllingg factors of the MCCD strategy. The results are presented in the left part of Fig. 5. Our findings indicate stable optimization across various settings, except for the case with a = 0.008 and B = 0.3. Upon conducting further experimental analysis. we identify the issue as originating from the model's failure to converge. Moreover, we visualize the attention weight on the uniformly sampled audio and video frames to qualitatively analyze the debiasing capability of our method. The visualization, displayed in the right part of Fig. 5, reveals that crucial audio and video frames for QA consistently receive significant attention, both in in- and out-of-distributie settings. This further demonstrates that our method predicts answers through the grounding capabilitics of audio and vision rather than relying on bias learning. More cases are shown in Appendix D.5.
6 Conclusion and Limitation
We are the first to investigate bias learning the AVQA task from model evaluation and design aspects. On the one hand, we construct new dataset, MUSIC-AVQA-R, which evaluates the performance on the head, tail, and overall samples, providing a precise measure of model robustness. On the other hand, we introduce a robust architecture employing the MCCD strategy to mitigate bias learning. Extensive experiments demonstrate the effectiveness of our architecture and the plug-and- play debiasing capability of MCCD. Furthermore, we reevaluate previous multi-modal QA methods on our proposed dataset, revealing their poor robustness.
Due to constraints imposed by MUSIC-AVQA, the answer space of our dataset is limited, comprising only 42 classes, and answer lengths are typically confined to a single word. This deviation from real-world scenarios is noteworthy. Concerning model designs, for fair comparison with baselines, we do not select large generative models to be backbones. However, compared with the answer classification, it may be more useful to generate answers for the AVQA task.

---

0.82
12:15
26-12
3.79
1.25
1.74
2.77
3.51
1,07
8.27
2.92
17-12
9.13
5.06
6:18
4.14
4.37
11.60
6.95
3.42
Which iestument may be seen the left the electric bass?
Saxcophione
(Head)
3.81
7:30
5.79
2.62
4.22
20 10
3:19
18.61
8.40
0L00 200E
OCCO
25.78
0.79
7.00
1.83
1.77
2.31
2.55
2.20
5,02
6.71
being played,
being
Acoustic
Figure 5: Sensitivity and qualitative analysis. a and / the weight-controlling factors the MCCD strategy. We visualize attention weights on the uniformly sampled audio and video frames.
7 Acknowledgements
This work was supported by the National Key Research and Development Program of China 2021YFB17156000 the National Natural Science Foundation of China (U22B2019, 62477037, 62450005, 62437002, 62306229, 62293553), the Natural Science Basic Research Program of Shaanxi (2023-JC- YB)593), the Youth Innovation Team of Shaanxi Universities "Multi-modal Data Mining and Fusion", the Shaanxi Undergraduate and Higher Education Teaching Reform Research Program (195,, the Youth Talent Support Program of Shaanxi Science and Technology Association (202401 13), the Xi'an Jiaotong University- Mobile Communications Group Co., Ltd. Digital Government Joint Institute, and the China Postdoctoral Science Foundation (2024M752585),
References
[1] Y.-B. Lin, Y.L. Sung, J. Lei, M. Bansal, and G. Bertasius, "Vision transformers parfmeter-efficient audio-visual learners," in CVPR, 2023, pp. 2299-2309. [Online]. Available: https://doi.or//10.
[2] H. Alamri, V. Cartillier, A. Das, J. Wang, A. Cherian, I. Essa, D. Batra, T. K. Marks, C. Hori. Anderson al., "Audio visual acene-aware dialog." CVPR, 2019, pp. 7558-7561 [Online]. Available: http://openaccess.o http://pnnnccess.tec....oooomm whwccv...om/comtentt CVPR 20 2019html/Alamri Audio Visual
Scene-Aware www.wweeeaaeaaaaaaeeeeee CVPR CVPR_20199000000000
2019 1appee_ppppphmll
[3] P. Yang. X. Wang. X. Duan, H. Chen, R. Hou, C. Jin, and W. Zhu, "AVQA: A dataset for audio-visual question answering on videos," in ACM MM, 2022, pp. 3480-3491. [Online]. Available: https://doi.org// http://o.orrrr..1114//5333311111 155111115//5511135555
[4] G. Li, Y. Wei, Y. Tian, C. Xu, J.-R. Wen, and D. Hu, "Learning to answer questions in dynamic audio-visual scenarios." " in CVPR, 2022, pp. 19 108-19 118. [Online]. Available: http::/doi.orr/// 1ttp////0.0001//2208000002222220
[5] H. Yun, Yu, W. Yang, K. Lee, and G. Kim, "Pano-AVQA: Grounded audio-visual question answering on 360 videos." in CVPR, 2021, pp. 2031-2041. [Online]. Available:
[6] Z. Wen, G. Xu, M. Tan, Q. Wu, and Q. Wu, "Debiased visual question answering from feature and sample perspectives," " in NewrIPS, 2021, pp. 3784-3796. [Online]. Available: https:// proceedings.neurips. wwwpuripss.cppape/22222222 httper//////hhhh/// 16: 1933d2babee6.9333aaaabbb Bataalbbbbbce--bstrtttt..t. Abstract.html
[ M. Vatsa, A. Jain, and R. Singh, "Adventures of trustworthy vision-language models: survey," in AAAI, 2024, pp. 2265-22266 658. [Online]. Available: https:
htt.:ogg/10.6099/aai..3333
10

---

[8] S. M. Hall, F. Gonçalves Abrantes, H. Zhu, G. Sodunke, A. Shtedritski, and H. R. Kirk, "Visogender: dataset for benchmarking gender bias image-text pronoun resolution,"
http://aaprr.nnssccc/pppp hwtpscccppeeefeee/pppepepp files/paper/2023/hahh in NeurIPS, 2024. [Online]. Available: Abstract-Dattsets Benchmarks.htm))
[9] Y. Li, B. Hu, Zhang, Y. Yu, J. Liu, Y. Chen, and J. Xu, "A multi-modal debiasing model with dynamical constraint for robust visual question answering," in Findings of ACL, 2023, Pp. 5032-5045 [Online]. Available: https://doi.org/10..180. 10.1100.1.///// hann2...inings-ac....
[10] A. Ravichander, J. Stacey, and M. Rei, "When and why does bias mitigation work?" in Findings of EMNLP, 2023, pp.
[Online]. Available: https:
findings- //cckmthology.orr/2022 .indinnn///..iinnnnng indingg-emmll.6619 emnlp.619
[11] Miller. K. Krauth, B. Recht, and L. Schmidt, "The effect of natural distribution shift on question answering models," in ICML, 2020, pp. 6905-6916. [Online]. Available: http://proceedings.. press// 19/miller20a.htm
[12] A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi, "Don't just assume; look and answer: Overcoming priors for visual question answering," in CVPR, 2018, pp. 4971-4980. [Online]. Available: http://openaccess. http://ppenaccesss.eetcccoooooop thhccc.comm/oontett_cycccc 8/html/Agrawal _Dnt________ Just Aussssssss_eeeee 2018_paper.html
[13] I.
Schwartz, A. G. Schwing, and T. Hazan, "A simple baseline for audio- visual scene-aware dialog," in CVPR, 2019, pp. 12548-12558. [Online]. Avail-
able: http://openacces.theeeye ctthecf.com/cconeettcccoo. CVPR 2019/htm///hhwarrrr_ooo Lttp///hwaatrtSSSpleeeeee Aoriio Visual Scene-Aware Dialog 2019 pa1pPp_hh
[14] X.
Liu, Z. Dong, and P. Zhang, "Tackling data bias in music-avqa: Crafting balanced dataset for unbiased question-answering: in WACV, 2024, pp. 1478-4487. [Online). Available: hups://doi.org//II 1000/ACV57000.22200.0000
[15] X. Zhang. F. Zhang, and C. Xu, "Next-ood: Overcoming dual multiple-choiee VQA biases." " IEEE TPAMI, vol. 46, no. 4, pp. 1913-1931, 2024. [Online]. Available:
https://oi.oor ort.:/..///11/11/11111
[16] B. Zhu, K. Tang, Q. Sun, and H. Zhang, "Generalized logit adjustment: Cal- ibrating fine-tuned models by removing label bias foundation models," in NeurlPS, 2023. [Online]. Available: http://aapers.iisscc/aaap httpccccaaapceeeseae////2//aaaa 10423 104231cch99 Dastgg-bssttaaCoonneae
[17] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and Misra, "ImageBind: One embedding space to bind them all," in CVPR, 2023, pp. 15 180-15 190. [Online]. Available: http:://lii.rr/11..11100 10t.:///.........0000000220..... 2T1.:52222...22.14557
[18] H. Zhang, X. Li, and L. Bing, "Video-LLaMA: An instruction-tunet audio-visura language model for video understanding," in EMNLP (Demos), 2023, pp. 543-553, [Online]. Available: https//aclanthoolgyy wtt.eoor//22...mmlpmddeeem .ew.lp-deoo..9999
[19] Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. "Squad: 100,000+ questions for machine comprehension of text," in EMNLP, 2016, pp. 2383-2392. [Online]. Available: https:/doi.org//
18653/v 1/d16-18 1264
[20] Y. Goyal, T. Khot, A. Agrawal, D. Summers-Stay, D. Batra, and D. Parikh, "Making the v in VQA matter: Elevating the role of image understanding in visual question answering," IJCV, vol. 127, pp. 398-414, 2019. [Online]. Available: https://do..rr//1..o 1007/s 1263-018-1116-0
[21] D. A. Hudson and C. Manning, "GQA: reasoning and compositional question answering." in CVPR, 2019, pp. 6700-6709. [Online]. Available: http://openaccesstheee wwwccf.ccom/ooneenntt CVPR CCVPR_2019/hun 2019/hnml/Hudson GQA_A_Neww Dataset for Real- World. Visual Reasoning and Compositional CVPR 2019 2019_paper.htm)
new dataset for real-world visual

---

[22] A. Yang, A. Miech, J. Sivic. I. Laptev, and C. Schmid, "Zero-shot video ques tion answering
frozen bidirectional language models," in NeurIPS, S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., 2022, pp. 124-141. [Online]. Available: http//ppaper.nnps..oo/ 00d (0a1100hhh7144011177777000777783 htt.h//e/t/eeee//ssseaatattaaae Ceatract-Coffeenceehhe ccchaper_ilee/pnper/2222220
[23] Y. Li, W. Li, and Nie, "Mmcoqa: Conversational question answering over text, tables, and images," in ACL, 2022, pp. 4220-4231. [Online]. Available: hitps:
//doi.org/10. //bb.org/10.11865//1/222222 16153/////022..cllloooo
[24] J. Ma, P. Wang. D. Kong, Z. Wang, J. Liu, H. Pei, and J. Zhao, "Robust visual question answering: Datasets, methods, and future challenges." IEEE TPAMI, vol. 46, no. 8, pp. 5575-5594, 2024. [Online]. Available:
[25] C. Kervadec, C. Wolf, G. Antipov, M. Baccouche, and M. Nadri, "Supervis- ing the transfer of reasoning patterns in vqa." in NeurIPS, vol. 34, 2021, pp. 18256-18261 [Online]. Available: http://occccins..uuupsscccpppppp
[26] S. Ramakrishnan, A. Agrawal, and S. Lee, "Overcoming language priors in visual question answering with adversarial regularization," NeurlPS, 2018, pp. 1548-1558. [Online]. Available: http::/proceedings.eeee www.nnss....uarrr/////e 67d96d458abdef21 2tt.9/aaeee222222222688992222222 Aas
[27] C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang, "Large language models are not robust multiple choice selectors," in ICLR, 2023. [Online]. Available: https:
//bi.org/10.488550arxxiiii 200.4448//...i.233..... 008822
[28] M. Ko, J. Lee, H. Kim, G.
Kim, and J. Kang, "Look at the first sentence: Position bias in question answering," in EMNLP, 2020, pp. 1109-1121. [Online]. Available: htps://doi.org/100 10t///....../////.
[29] C. Dancette, R. Cadene, D. Teney, and M. Cord, "Beyond question- based biases: Assessing multimodal shorteut learning in visual question answering," ICCV, 2021, pp. 1574-1583 [Online]. Available: https://doi.org/1 10. 1099/CCC4499222222200000
[30] C. Kervadec, G. Antipov, M. Baccouche, and C. Wolf, "Roses are red, violets are blue... but should VQA expect them to?" in CVPR, 2021, pp. 2776-2788 [Online]. Available: https://openacces. www.tte.ecccmcmce Wwcoootten/C//222////mlmmlm/mmm Violets Are Blue... but Las___ut_____a__S Should VQA Expect Expect_CVPR CVPR CVPR_2021 1021__pper.html
Roses_ Are Red_
[31] E. Harrell Jr, K. L. Lee, and D. B. Mark, "Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors." Statisties in medicine, vol. 15, no. pp. 361-387, 1996. [On- line]. Available: https://onlineiiraayy..
[32] S. Sheng, A. Singh, V. Goswami, J. Magana, T. Thrush, W. Galuba, D. Parikh, and D. Kiela, "Huuaan-adversrril visual question answering." in NeurIPS, 2021, pp. 20 24344-20 359. [Online]. Available: https//mooedddngggenrrpccccpppep a47755488447414477
Abstract.tmm
[33] L. Li, J. Lei, Z. Gan, and J. Liu, "Adversarial VQA: A new benchmark for evaluating the robustness of VQA models," in ICCV, 2021, pp. 2042-2055 [Online]. Available: http:///ii.rrr//..../o//ccvvvv 109/1CCV48922.2000 200.000
[34] J. Wu and R. J. Mooney, "Self-critical reasoning for robust visual question answering." in wwwp:ccccinnn.curppscccpaapapppp 140099991111----1 1--sstrtct..)mmmmmmm
NewrIPS, 2019, pp. 8614. [Online]. Available:
[35] W. Xu, Q. Liu, S. Wu, and I.. Wang, "Counterfactual debiasing for fact verification." in ACI. 2023, pp. 6777-6789. [Online]. Available: http:://oi..gg//...6653///122200 1/2.o3633551111222aall
12

---

[36] C. Tsirigotis, J. Monteiro, P. Rodríguez, D. Vázquez, and A. C. Courville, "Group robust classification without any group information." in NeurIPS, 2023. [Online]. Available: http:/papers.nnpp niwps......nn.seeee....... wwtppperrssiisscccpppeeeeeee b0d9ceb3d1 d01 dd11ee55da221d22220755.. Awstrract--ooeeeeeeeee
[37] D. Esiobu, X. Tan, S. Hosseini, M. Ung. Zhang, Fernandes, J. Dwivedi- Yu, E. Presani, A. Williams, and E. Smith, "Robbie: Robust bias evaluation of large generative language models," in EMNLP, 2023, pp. 3764-3814. [Online]. Available: http:///oii.or//II 10001
[38] R. Cadene, C. Dancette, H. Ben-younes, M. Cord, and D. Parikh, "RUBi: Reducing unimodal biases for visual question answering," in NeurlPS, 2019, pp. 841-852. [Online]. Available: https///rooceeings.memm 51d92be 1c60d dbl d2255e700dda55522- Abstract.html
[39] Y. Niu and H. Zhang, "Introspective distillation for robust question answering," in NeurIPS, 1/hash/
2021, pp. 1629221222 16 304. [Online]. Available:
[40] Cho, D. Kim, H. Ryu, and S. Kweon, "Generative bias for robust visual question answering," in CVPR, 2023, pp. 681-1111 690. [Online]. Available: https: //bb..or//00.1009//PPR3322 202.3.01222
[41] Ma, Wang, Z. Wang, D. Kong, M. Hu, T. Han, and J. Liu, "Adaptive loose optimization for robust question answering," arXiv preprint arXiv:2305.03971 2023. [Online]. Available: http://4xxivvorrp////205503311
[42] G. Kv and A. Mittal, "Reducing language biases in visual question answering with visually-groundee question encoder," in ECCV, 2020, pp. 18-34. [Online]. Available: http:///ooi.org/10 wtt.://///...1/09970111000
[43] E. Abbasnejad, D. Teney, A. Parvaneh, J. Shi, and A. d. Hengel, "Counterfactual vision and language learning," " in CVPR, 2020, pp. 10044-10054, [Online]. Avail- able: https://openaccess.u www.tttf..oc/ooontteee Cepnntent___PRR._e CVPR CLP___CCP_____0 2020/tml/Abbaasnejad. hatpr//abaaaaaaaaCCCooeetttllll Counterfactual httppppnncccess.eeef.ooo///////e
Vision aadea_an______ Languagge_aaaaniiinnng 200Ppee2o_____________2 papeeppperrhhmml
[44] L. Chen, Y. Zheng, and J. Xiao, "Rethinking data augmentation for robust visual question answering," in ECCV, 2022, pp. 95- [Online]. Available: http::/doi.orr/10. 1007/ 978-3-031-20059-555
[45] D. Tency,
Abbasnejad, and A. van den Hengel, "Unshuffling data for improved generalization in visual question answering," in ICCV, 2021, pp. 1417-1427. [Online]. Available:
htts://doi.or/11..o http://coo.r//...11//ccccv22222 109/1CCV4892222221 100145
[46] Z. Liang, W. Jiang, H. Hu, and Zhu, "Learning to contrast the counterfactual samples for robust visual question answering," in EMNLP, 2020, pp. 3285 3292- [Online]. Available: https://oi.org org.:/....rr/..... 10. 18653/v 1/2020. emnlp- main.265
[47] X. Zhu, Z. Mao, C. Liu, Zhang, B. Wang, and Y. Zhang, "Overcoming language priors with sule-sep learning for visual question answering." in IJCAI, 2021, pp. 1083-1089. [Online]. Available: https://doi.orgg http://do..org10..4966666 200024/3iicc
[48] Q. Si, Y. Liu, F. Meng. Z. Lin, P. Fu, Y. Cao, W. Wang, and J. Zhou, "Towards robust visual question answering: Making the most of biased samples via contrastive learning," " Findings of EMNLP, 2022, pp. 6650-6662. [Online]. Available: https://doi.or/100 10...18663//v1 http3////222..ndiimg m.0go...nnings-eel...995 emnlp..99
[49] C. Jing, Y. WI, X. Zhang, Y. Jia, and Q. Wu, "Overcoming language priors in VQA via decomposed linguistic representations." in AAAI, 2020, pp. 11 181-11 188. [Online]. Available: https://doi.org/ hrt.://oooooo//100 10. 1609/aaai. 334107..677
[50] R. Shrestha, K. Kafle, and C. Kanan, "A negative case analysis of visual grounding methods for VQA," ACI. 2020, pp. 8172-8181. [Online]. Available: https:
13

---

[51] I. Gat, I. Schwartz, A. Schwing, and T. Hazan, "Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies." in NeurIPS, 2020,
wtw.eeeddnnnnneuuisuccc//pp/ppee pp. 3197-3208. [Online]. Available: Abstract.html Db0449ccoS44/aaacc00e44444444464
[52] Q. Si, Z. Lin, M. yu Zheng, P. Fu, and W. Wang, "Check again: Progressive visual question answering via visual entailment," in ACL, 2021, pp. 4101-4110. [Online]. Available: http://doi..rr//10 /6553v1/20222 acc- longlong.317
[53] M. Lao, N. Pu, Y. Liu, K. He, E. M. Bakker, and M. S. Lew, "COCA: Collaborative causal regularization for audio-visual question answering," in AAAI, 2023, pp. 99995-13 003. 1t09///ai.v377111...
[Online]. Available: https://doi.org/Ih
[54] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "Audio set: An ontology and human-labeled dataset for audio events," in ICASSP, 2017, pp. 776-780. [Online]. Available: https://doi.orge org/10. L09ICASSP22111.99555550
[55] L. H. Li, M. Yatskar, D. Yin, C.-. Hsieh, and K.-W. Chang, "VisualBert: A simple and performant baseline for vision and language," arXiv preprint arXiv: 2019. [Online]. Available: http://arxiv.org/bbs. /0tps///////00/ 005557557
[56] H. M. Fayek and and J. Johnson, "Temporal reasoning via audio question answering," TASLP, vol. 28, pp. 2283-2294, 2020. [Online]. Available: hups://doi.org//// 10. http:///....00011122000000000000
[57] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, "VQA: Visual question answering," in ICCV, 2015, pp. 2425-2433. [Online]. Available: hops://doi.org/100 110010111///CC22015...77
[58] P. Zhou, W. Shi, J. Tian, Z. Qi. B. Li, H. Hao, and B. Xu, "Attention-based bidirectional long short-term memory networks for relation classification," " in ACL, 2016, pp. 207-212. [Online]. Available: https://oii.org/ org/O0. 10. 8653/v wt.gg//..11.////// 1/p16-2034
[59] J. Lu, J. Yang. D. Batra, and D. Parikh, "Hierarchical question-image co-attention for visual question answering." in NeurlPS, 2016, pp. 289-297. [Online]. Available: https:/procceedings hatea/aaaaad---tttttttttha
[60] Z. Yu, J. Yu, Y. Cui, D. Tao, and Q- Tian, "Deep modular co-attention networks for visual question answering," in CVPR, 2019, pp. 6281-6290. [On- line]. Available: Co- -et-Antttoo_Nttteeete Networks for Visual
wwwp:/acccess.ccc....ooo//////oe Leppontet__CCPPP CVPR CCRC_CVVR111111 2019/ttl/Yuu Deep__Dee_____e Deep_Modular, Modular__ 2PFR_2019_paaae
[61] C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang, "Heterogeneous memory enhanced multimodal attention model for video question answering," " in CVPR, 2019, pp- 1999-2007. [Online]. Available: http://ppenaceess.o www.eaaccccsss.ccc.cc.cc Meemory_Enhanced
CCVP 2019/htm//Fan_ Heterogeneous _Mamoryeeerrrrra Enhanced Muldd_odalllll Multimodal_Attentionn, Aatention_Model for for__odeoo Video_ Question_ Answering_CVPP CLBR_CVRR201110 2019_ 2VVPR_2019_papprhhml
[62] X. Li, J. Song, L. Gao, X. Liu, W. Huang, X. He, and C. Gan, "Beyond RNNs: Positional
self-attention with co-attention for video question answering," in AAAI, 2019, pp. 8658-8665. [Online]. Available: https://doi.org/ 10. 1609/666i..ivv. 00009/aaiiv33301.3.... 333.0..33.11..15..
[63] T. M. Le, V. Le, S. Venkatesh, and T. Tran, "Hierarchical conditional relation networks for video question answering," " in CVPR, 2020, pp. 9972-9981. [Online]. Avail- able: http://pwnacceesseteece//o/////p CVVR CWwp__220/hmmmlll/ce H _Cnniitiona__ Relation_Networks_fton
Vudde__uuesioon_nsseeriiii wttp:/eeeeinn_cc________ 2V2R_2020_ppp__p_pe papper.hmm
[64] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x
words: Transformers for image recognition scale," in ICLR, 2020. [Online]. Available: https///peeeeeeeeeeeefoonnddddod
[65] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong et al., "Swin transformer v2: Scaling up capacity and resolution," in CVPR, 2022, pp. 12 019-12019. [Online]. Available: https://doi.org/ 10000
14

---

A Model Training and Testing
The details of model training are shown in Algorithm where L denotes the number of training samples, and nh is the batch size. In the test stage, the bias learner is removed.
Algorithm 1: Model Training
Input: D {(A, Vi, Qú y)))
Output: Robust AVQA model. Initialize model parameters 0;
2 Initialize Adam optimizer;
Set learning rate Y):
Set the number of training epochs M;
for epoch + 1 to M do
for each batch in D do
Learn uni-modal and multi-modal representations: A +- VGGish (A), v +
ResNet (aaaNett8(V)) (8V). A &- VisualBert(A9)) Ve < VisualBert(WV) (V)),
Q VisualBert(Q)), M + VisualBert([AA IV[] Q"):
Capture uni-modal biases: ya + BiasLearner, (A A. y f BiasLearner, (V"),
99 BiasLearner, (Q"):
Obtain answer predictions: 8m +- Classifier(M");
Compute the QA loss: La <- 1
10
nh
Mitigate biases by MCCD: fa, f. MCCD(F*, $", "m); Compute the joint loss: t La + +a
12
13
Backward pass: V0 gr
14
Update model parameters: 0 + Optimize(0,, v0, n)):
15 return AVQA model.
B Dataset Analysis
We make statistics on the rephrasings, as depicted in Table 4. Any rephrasing that garners fewer than vote will be disregarded. It is evident that the overwhelming majority of the rephrased questions received three favorable votes. Fig. 6 shows the distribution of questions in MÚSIC-AVQA and
Table 4: Statistics of rephrasing consistency. Positive and Negative denote whether the annotator agrees with the rephrasing or not.
Positive
Negative
Total
3
0
164, 219
2
1
47,353
1
2
7,481
0
3
9,172
MUSIC-AVQA-R based on the first three words. It is evident that there are notably more entries within each circle in the left figure compared to the right figure. This suggests that the diversity of dataset is higher than MUSIC-AVA.
We visualize the answer distribution of specific types of questions. Fig. 7, 8, and show the visualization of the AVQA, audio QA and visual QA tasks, respectively. We can see that all of the answer distributions are long-tail. This further demonstrates the necessity of head and tail sample splitting. Specifically, for the "Location" type in the AVQA task, we can see that the number of "congas" is far less than that of "yes". possible answers, we classify questions with lower frequencies as tail samples and those with higher frequencies as head samples. For instance, for the "Fxistential" questions the AVQA task, we consider the questions with the answer "no" tail samples.
noteworthy that, for question types featuring only two
15

---

Sound
there
the
this
the
Whiin
think
Where
kind
where
musical
What
the
does
how
dlfferen
Uhee
what
many
HOK
Rist
the
sund
video
the
mt
(a) MUSIC-AVQA-R.
(b) MUSIC-AVQA.
Figure 6: Distribution visualization of questions based on the first three words.
12000
12000
8000
10000
10000
Numbbers
6000
Numbers
8000
Numbers
8000
4000
1.2ule)
4000
4000
2000
2000
2000
two
zero
tivoo
four
sise
eight
nine
0
more than
yob
60
yes
Answers
Answers
Answers
(a) Existential
(b) Counting
(c) Comparative
3500
3000
2500
Numbers
2000
1500
1.2ula)
500
108
left
Right
valnn
middlo
piana
acoustic gutar
cella
Eocortion
Basapponn
cannel
ukuaoio
Buta
giatheng
electri basa
tumpat
Piaa eruu
Enspine sjlaghens
suona
congoo
Answers
(d) Location
Figure 7: Answer distributions of specific types of questions in the AVQA task. p(a) is the average number of answers in group.
16

---

12000
7000
6000
8000
5000
Numbers
Numbers
4000
4000
1.2u(a)
2000
2000
1000
y00
tvppe
five
more than ton
six
eight
envon
60
yes
Answers
Answers
(a) Comparative
(b) Counting
Figure 8: Answer distributions of specific types of questions in the audio QA task. (uaa is the average number of answers in group.
10000
6000
0000
8000
Nambers
1.2ule)
2000
C of
Indbag
congen
Answers
Answers
(a) Counting
(b) Location
Figure 9: Answer distributions of specific types of questions in the visual QA task. u(a) is the average number of answers group.
B.1 Ethics Statement
MUSIC-AVQA [4] has undergone meticulous pre-processing tailored for academic research purposes. ensure the absence of any information that discloses names or uniquely identifies individuals, as well avoiding any offensive content. We only perform rephrasing and splitting for the questions within the dataset to develop MUSIC-AVQA-R, which preserves its inherent characteristics.
B.2 Question Comparison
We select questions from both the head and tail splits to demonstrate the diversity of our dataset, as illustrated Figures and 11. Due to the use of pre-defined templates, the questions in train and test splits of MUSIC-AVQA differ by only single word. In contrast, the questions in our dataset exhibit a variety of formats, which better reflect cal-world scenarios. Additionally, our dataset encompasses larger test space compared to MUSIC-AVOA.
17

---

Predefined Question Template in MUSIC-AVQA: Which is the musical instrument that sounds the same time as the <Object>?
Question in MUSIC-AVQA: Which is the musical instrument that sounds at the same time as the pipa?
Rephrased Question in MUSIC-AVQA-R (Head Split):
Which is the musical instrument that sounds at the same time as the pipa?
V
Which musical instrument plays simultaneously with the pipa?
What musical instrument is playing in unison with the pipa?
What musical piece is playing in the background the pipa is being played?
What musical instrument is playing concurrently with the pipa?
What musical instrument being played when the pipa is being played?
What musical piece is being played simultaneously with the pipa?
When the pipa is being played, what musical composition is being performed?
What musical work is being performed when the pipa being played?
The pipa is being played, but what musical composition is that?
When playing the pipa what musical composition is being performed?
What instrument playing simultaneously with the pipa?
What other instrument is playing at the same time the pipa?
What musical instrument plays simultaneously with the pipa?
What the musical instrument that produces sound in tandem with the pipa?
What musical instrument produces sound simultaneously with the pipa?
Question Comparison between MUSIC-AVQA and MUSIC-AVQA-R. The rephrased question comes from the head split of our dataset. The questions in our dataset feature diverse formats, which more accurately reflect real-world scenarios.
Figure 10:
C Proof
Given a probability density function (1x that conforms uniform distribution with size N, its entropy H(X) can be calculated as follows:
N
H(X)= = - p(xs) logg p(r;),
(3)
1=1
p(x;)= ((xs,
where p(z:) is the probability of X Fi.
In uniform distribution, each probability p(x;)) is the same, ie., p(x;)) = f(x;) = 1
N so
N
H(X) = Σ
logg
1
(5)
N
N
Then, bring the term outside the summation:
N
1
1
H(X)
log2
(6)
N
Thirdly, move the negative sign inside the logarithm:
1 H(X)=
log, (N),
(7)
N
Finally, combine N with the summation:
H(X) log, (N).
(8)
18

---

Predefined Question Template in MUSIC-AVQA:
What is the instrument on the <LR> of <Object>?
Question in MUSIC-AVQA:
What is the instrument on the right of piano?
Rephrased Question in MUSIC-AVQA-R (Tall Split)
What is the instrument on the right of piano?
Which instrument is the right of the piano?
The piano right instrument is what Kind of instrument?
What kind of instrument is on the right of piano?
What kind instrument is that to the piano right?
What type of instrument can you see the right of the piano?
Which instrument may be seen to the right of the piano?
Which instrument is that on the right side of the piano?
What is the instrument to the right of the piano?
Which instrument can you make out to the right of the piano?
What is the instrument on the right the piano called?
What's the name of the instrument to the right of the piano?
Which equipment is situated to the right of the piano?
What instrument is located on the right of the piano?
What is the right side instrument of the piano?
What is the instrument located on the right of the piano?
Figure II: Question Comparison between MUSIC-AVQA and MUSIC-AVQA-R. The rephrased
question comes from the tail split of our dataset. The questions in dataset feature diverse formats, which more accurately reflect real-world scenarios.
D Experiments
D.1 Dataset Comparison
The test split comparison between MUSIC-AVQA and MUSIC-AVQA-R is shown in Table 5. We can see that our proposed dataset exhibits larger test sample space. This can provide a more precise evaluation for the model robustness.
Table 5: Test split comparison between MUSIC-AVQA and MUSIC-AVQA-R. EXIST, LOC, CNT, COMP, and TEMP, which are question types, denote "Existential". "Location", "Counting". "Com- parative", and "Temporal", respectively.
Audio QA
Visual QA
AVQA
Dataset
CNT
COMP
CNT
LOC
EXIST
LOC
CNT
COMP
TEMP
MUSIC-AVQA
1.017
594
1.197
1,225
988
920
1,265
1,101
MUSIC-AVQA-R
23,107
13,506
27,867
3,3049
25,049
21,546
26,565
23,121
17,762
D.2 Implementation Details
The number of trainable parameters of our model is 117M. In the default settings, our model training takes about 20 hours. We initialize the seed for both Numpy and Torch to 42. The other details can be found in our uploaded code.
D.3 Baselines
The audio QA baselines are follows.
19

---

FCNLSTM employs a fully convolutional network and LSTM to initially learn the rep- resentations of audio and questions separately. Subsequently, projects the concatenated features of both into the answer space.
CONVLSTM a variant of FCNLSTM, incorporating five convolutional blocks identical to VGGNet for acquiring a variable-sized representation of audio.
The visual QA baselines are as follows.
GRU (dubbed "deeper LSTM + Norm I" in the published paper) is simple baseline that first uses VGGNet and LSTM to encode the images and questions and then maps the concatenated features of them into the answer space.
BiLSTM Attn is an attention-based bi-directional LSTM network, which was often used in
previous relation classification.
HCAttn is hierarchical co-attention method that employs question- and image image-guided guided attention reason on images and questions, respectively.
MCAN is deep modular co-attention network comprised of cascaded modular co-attention layers, where attention is implemented through multi-head attention Transformers.
The video QA baselines are as follows.
HME* a heterogeneous memory-enhancee multimodal attention model that can effectively learn global context information from appearance and motion features.
PSAC employs positional self-attention block to model the dependency between question words and video frames, respectively. It utilizes co-attention mechanism to perform multi-modal interaction.
HCRN is hicrarchical conditional relation network that embeds video input at various granularities, encompassing frames, short clips, and entire video levels.
The AVQA baselines follows.
AVSD simple but effective audio-visual dialog method. initially encodes the input modalities separately and subsequently feeds their fused features into LSTM to generate answers.
LAViT" is spatial AVQA framework that utilizes three distinct Transformer blocks to perform interaction between input modalities.
ITG associates particular visual locations with audio to conduct spatial grounding. Based this, audio and visual features of key timestamps are further emphasized through question queries for temporal grounding.
"A based STG, incorporates trainable parameters into powerful visual encoders such as ViT and Swin.
D.4 Reevaluation on MUSIC-AVQA-R
Table 6 illustrates the overall accuracy for specific types of questions. Notably, our architecture surpasses all baselines across every question type. Importantly, our architecture achieves the highest performance in all three tasks, underscoring the idea that the additional modality serves valuable complement. For instance, the audio modality may enhance AVQA models in the video QA task.
hetpp://ggttabbbooo//LLL////e/g VideoQQ
hharp://ggtthbabcccom///xxgggggg
20

---

Table 6: Experimental results on the MUSIC-AVQA-R test split. The numerical values represent the overall accuracy for specific types of questions. EXIST, LOC, CNT, COMP, and TEMP are question types, representing "Existential", "Location", "Counting", "Comparative", and "Temporal", respectively.
Audio
Type
Method
CNT
COMP
Avg.
CNT
LOC
Avg.
EXIST
L.CC
CNT
COMP
TEMP
Avg.
Avg.
Audio
FCNLSTM
60,98
58.74
60.15
56.17
55.41
52,93
54,07
71.21
48.67
44.76
57.21
43.08
34.34
52.21
55.20
54,12
BiLSTM Atin
68.85
46.36
60.56
57.08
47.87
52.08
56.41
36.79
43.23
50.68
23.49
43.34
48.84
Visual
HCAtt
58.13
53.79
56.53
51.21
59.98
55.97
64.02
38.08
44.32
54,38
36.
48.24
51.90
MCAN
72:40
54.99
65.98
60.33
64.33
62.50
58.75
49.04
51.99
51.47
44.71
$1.69
57.27
HCRN
55.38
41.28
50.18
39.89
45.33
42.84
53.07
38.00
42.80
35,03
42.32
37.94
43.92
Video QA
PSAC
53.29
52.89
48.
35.09
50:45
HME
61.07
56.44
39.55
47.08
65.81
57.24
66.09
36.11
48.31
56.84
37.23
49.91
53.66
AVQA
AVSD)
52.91
54.70
68.01
61.9/2
33.
58
40.37
46.
52.35
STG
53.77
60.20
56.14
54.59
59.07
$7.02
75.18
36.33
35.41
$7.81
39.35
49.47
52,80
Ours
$1.30
63.57
74.76
72.09
73.32
72.76
75.36
$7.37
60.32
59.74
49.97
61.34
66.95
D.5 Case Study
4.50
2.43
1.71
2.23
3,90
4.98
4,00
2.86
4.17
6.84
0.00
12.98
9.89
10.23
8.98
How many kinds of musical Instruments are heard in the video?
(Head)
1.45
5.95
8.49
5.07
2.65
1.63
2.77
2.10
4.69
3.31
8.56
8.58
3.37
5.39
2.29
16.38 18.38
1.51
10.34
0.72
7.98
is number Instruments
the video?
Three
(Tail)
1.36
23.46
15.40
12.69
22.14
0.72
7.50
2.86
8.43
6.23
10.701
10.51
10.30
2.27
1.18
1.25
1.26
1.10
1.11
1.06
Whatis the guzheng that gives off the sound?
Simultaneously.
(Head)
2,00
3.33
3.51
6.86
1.99
0.80
0.34
0.73
0.92
2.45
3.58
3.41
5.70
2.41
4.70
7.01
11.75
14.79
11.75
5.02
In the video, how many bagpipe can heard playing?
(Tail)
Figure 12: Attention weight visualization on the uniformly sampled audio and video frames.
21

---

We visualize attention weight on extra uniformly sampled audio and video frames to qualitatively analyze the debiasing capability. In Fig. 12, the visualization is presented for more extensive range of head and tail samples. We can see that our method can focus on the key audio and video frames for QA simultaneously in both in- and out-of-distribuution settings. For instance, in the upper head and tail case, our method demonstrates high attention to various instruments, leading to accurate answers for "counting" questions. This serves as additional evidence supporting the debiasing effectiveness of our proposed MCCD strategy, highlighting its substantial contribution to improving model robustness.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made clear claims about our contribution and scope in Section Abstract and 1.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our work not only from the model design but also from the model evaluation in Section 6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In Section 3.2, we claim that log N is the entropy of a uniform distribution. We have provided detailed proof in Appendix C.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the implementation and training details in Section 5.1, 5.2, Appendix A, B, and D.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have uploaded our code and dataset in the supplemental material. The detailed "readme" file is also uploaded.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described the training and details in Section 5.1, 5.2, Appendix A, B, and D.2.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We re-evaluate 13 multi-modal QA methods, conducting the statistical signifi- cance experiment may be computationally and timely expensive. For example, running one epoch for STG takes almost 12 hours, and the maximum number of epochs is set 80. To the best our knowledge, all methods adopt seed fixing, which undoubtedly enhances the reproducibility of the experiments and the credibility of the results.
23

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We have provided the detailed description in Section 5.2, Appendix D.2, and
D.3.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http//eurriippeee httprr/pp.p/pppb.tiaicaaieeieeee
Answer: [Yes]
Justification: Our work conforms to the NeurlPS Code of Ethics. We have provided the reason in Appendix B.I.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided analyses in Appendix B.1.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model is not generative.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have adhered the mentioned licenses and terms of for all the assets used in the paper, and the original creators or owners have been properly credited or mentioned Section 5.3, and Appendix D.3.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have uploaded detailed "readme" file in the supplementary material.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have provided crowdsourcing descriptions in Section 3.1 and Appendix B. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer: [NA]
Justification: Our paper does not contain such risks.
25


Reviews:
[{"review_id": "UQVH958XB0", "reviewer": "Reviewer_Zk4k", "summary": "This paper first analyzes the bias in the existing benchmark (MUSIC-AVQA) on Audio-Visual Question Answering (AVQA). They find MUSIC-AVQA testing set is created using templates and only has 9k QA pairs with limited vocabulary, leading to the potential correlation between words in questions and answers. Therefore, previous methods optimized on MUSIC-AVQA could \"memorize\" the regularity between questions and answers and may not be proper for AVQA question in real world, which include diverse QA pairs. To evaluate the robustness of the AVQA methods, they use a rephraser to rephrase questions in MUSIC-AVQA testing set, thus create the MUSIC-AVQA-R test set. In addition to the rephrased testing set, they design a model trained with proposed multifaceted cycle collaborative debiasing (MCCD) strategy that decreases the logit between any single modality and multimodality logit. MCCD also includes the loss function to maintain the similarity between bias logits. In experiments, they report results on both MUSIC-AVQA and MUSIC-AVQA-R testing sets. Comparing with previous methods, their proposed approach gets good performances in both MUSIC-AVQA and MUSIC-AVQA-R.", "questions": "See weakness", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(1) The proposed MCCD strategy is model-agnostic. The experiments in Table 1 show the consistent improvements for previous methods when MCCD is applied on MUSIC-AVQA. \n(2) This paper also contribute a new testing set for the research society to evaluate the robustness of AVQA models. \n(3) The proposed model and MCCD achieves good performances in AVQA, obtaining comparable results on MUSIC-AVQA comparing with previous AVQA methods. It also outperforms previous methods in MUSIC-AVQA-R, showing the robustness of the proposed model.", "weaknesses": "(1) In ablation study, have the authors tried to only finetune the linear classifier using multimodal features? Without training with MCCD, I would like to see the performances on MUSIC-AVQA-R and MUSIC-AVQA. \n(2) VisualBERT is originally trained with only image and text, but in the paper it seems VisualBERT is used to encode the audio information as well. Do the authors finetune VisualBERT during training? \n(3) Even though the paper is easy to follow, the presentation can still be improved. In Figure 4, \"Unimodal Bias Learning\" has \"Audio Bias Learner\" but the input is actually video features. Some highlighted regions such as the purple region is not explained well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable and constructive comments.\n\n* **Reply for Q1.** We fine-tune the VisualBERT and linear classifier simultaneously in the ablation study, consistent with the main experiment. The experimental results without training MCCD are shown in the following table. We observe that VisualBERT achieves competitive results on two datasets. This demonstrates that pre-trained or large multimodal models are more robust than smaller models, which aligns with the findings in [1] and [2]. We will add this experimental result and analysis to the revised manuscript.\n\n|           Method          | MUSIC-AVQA |           |        |        | MUSIC-AVQA-R |           |        |        |        |        |\n|:-------------------------:|:----------:|:---------:|:------:|:------:|:------------:|:---------:|:------:|:------:|:------:|:------:|\n|                           |  AQA  | VQA |  AVQA  |   All  |   AQA   | VQA |  AVQA  |    H   |    T   |   All  |\n| Ours              | 79.14   | 78.24   | 67.13    | 72.20   | 74.76   | **72.76**   | **61.34**   | **72.24**   | **59.39**   | **66.95**   |\n| w/o $d_i^{\\mathrm{q}}$  | **79.64**   | 77.62   | **67.52**    | 72.34   | 74.89   | 72.26   | 59.53    | 71.72   | 57.47   | 65.86   |\n| w/o $d_i^{\\mathrm{v}}$  | 79.27   | 78.61   | 67.23    | 72.37   | 71.29   | 70.18   | 58.08    | 68.34   | 57.43   | 63.85   |\n| w/o $d_i^{\\mathrm{a}}$  | 77.22   | 77.50   | 67.31    | 71.76   | 70.92   | 67.82   | 55.57    | 66.05   | 55.61   | 61.75   |\n| w/o MD            | 78.46   | 77.54   | 66.39    | 71.48   | 73.97   | 70.41   | 58.87    | 70.09   | 57.25   | 64.80   |\n| w/o CG            | 78.77   | 78.65   | 67.50   | **72.45**   | **75.42**   | 71.72   | 59.68    | 71.40   | 57.98   | 65.87   |\n| w/o MCCD          | 77.85   | **78.81**   | 66.43    | 71.73   | 67.36   | 65.46   | 55.12    | 63.23   | 54.54   | 60.22   |\n\n\n[1] Wen, Z., Xu, G., Tan, M., Wu, Q., & Wu, Q. (2021). Debiased visual question answering from feature and sample perspectives. Advances in Neural Information Processing Systems, 34, 3784-3796.\n\n[2] Ma, J., Wang, P., Kong, D., Wang, Z., Liu, J., Pei, H., & Zhao, J. (2024). Robust visual question answering: Datasets, methods, and future challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n\n* **Reply for Q2.** We finetune VisualBERT during training. In Figure 1, we first employ ResNet-18 and VGGish to obtain 512-dimensional and 128-dimensional embeddings of the video and audio segments, respectively. Then, we utilize linear layers to ensure dimension matching for the input embeddings. Subsequently, we concatenate the video and audio embeddings to form context representations which are then input into the \"visual_embeds\" of VisualBERT.\n\n* **Reply for Q3.** We have re-painted this figure shown in Figure 1 of \"response.pdf\". We replace the purple region with a dashed box. The video and audio embeddings are concatenated to form context representations in multimodal learning, while they are input into VisualBERT to learn the single modality representation in uni-modal learning, respectively. This updated figure will be included in the revised manuscript. Thank you again for your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7lzKtTwW5p", "reviewer": "Reviewer_b8Yd", "summary": "Prevalent Audio-Visual Question Answering (AVQA) methods often learn biases from the dataset, which can lead to reduced robustness. Additionally, current datasets may not accurately assess these methods. To address these issues, the authors introduce MUSIC-AVQA-R, a dataset created in two steps. First, they rephrase questions in the test split of a public dataset (MUSIC-AVQA) and then introduce distribution shifts to the split questions. Rephrasing the questions allows for a diverse test space, while the distribution shifts enable a comprehensive evaluation of robustness for rare, frequent, and overall questions. The authors also propose a robust architecture that uses a multifaceted cycle collaborative debiasing strategy to overcome biased learning.", "questions": "Please refer to the questions in the weakness part.", "limitations": "I can’t find a negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The authors raised a new problem, the bias problem in audio-visual question answering.\n- The authors proposed a new dataset.", "weaknesses": "- First, I have a concern about the quality of the new dataset, it is not verified enough whether the dataset is properly collected. Similar to the case of the VQA-CP dataset, the authors should claim that the proposed dataset is a useful testbed to measure the distribution shift out of distribution robustness by showing the distribution difference between the train and test datasets. However, the authors only show the distribution of the rephrased dataset without showing the distribution difference between the train and test datasets. In short, there is no way to see whether the collected data after paraphrasing and distribution shifting is a valid one or not. In other words, how does the resulting distribution differ from the previous distribution?\nThe proposed solution also lacks novelty. Extensive work has been done to mitigate modality bias in the VQA-CP dataset, such as [1,2,3]. Therefore, in order to study the modality bias problem in AVQA, the authors should also devise a novel solution specialized for the AVQA problem by comparing it with the existing debiasing methods for VQA. \n\n[1] Counterfactual Samples Synthesizing for Robust Visual Question Answering. CVPR 2020.\n\n[2] Greedy Gradient Ensemble for Robust Visual Question Answering. ICCV 2021.\n\n[3] Generative Bias for Robust Visual Question Answering. CVPR 2023.\n\n- In addition, the authors should discuss why the proposed method is better than existing solutions theoretically. However, there is no explanation for why simply increasing the logit distance among the modalities in Equation 1 reduces the bias. Is there any reference from which the authors adopted the ideas? What I especially don't understand is whether there is any theoretical evidence as to why it is necessary to add the inverse of distance. How about minus? More theoretical discussions are required.\n- In addition, the proposed solution itself also doesn’t make sense. Equation 3, the KL divergence between logits, and equation 2, the Euclidean distance, are the same in that they measure the distance between the logits. In theory, when the temperature term in the softmax function is large enough, the kl divergence loss becomes equivalent to the Euclidean distance. It is unclear what is the meaning of putting these two similar losses in opposite directions? The authors should further explain the theoretical motivation for using those loss functions.\n- The performance in Tables 1 and 2 is also questionable. If other existing debiasing methods, such as [1,2,3], were applied to the same baseline, could the proposed method still outperform them? The proposed solution will be meaningful only if the proposed method sufficiently outperforms the existing methods.\nI don't understand how a simple method in this paper (ours in Table 1) achieves a favorable performance in MUSIC-AVQA with such a simple baseline architecture. What is the baseline model performance without adding the proposed loss functions? In Table 3, how does the model without any proposed loss function already show a favorable performance that outperforms the state-of-the-art methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and valuable suggestions.\n\n* **Reply for Q1**. Please see the above author's rebuttal attached with a \"response.pdf\".\n* **Reply for Q2**. Please see the above author's rebuttal attached with a \"response.pdf\".\n* **Reply for Q3**. We will explain the MCCD from theoretical and empirical views. **Theoretical explanation:** (1) different roles and purposes of the loss functions. **In Equation (1), we use the Euclidean distance to enlarge the dissimilarity between *uni-modal and multi-modal logits*.** Increasing these distances reduces the model's reliance on single-modal information, thereby mitigating bias. **In Equation (2), we use KL divergence to constrain the distribution of *uni-modal* logits.** By minimizing the KL divergence between uni-modal logits, we enforce a closer distribution, establishing a collaborative relationship among them. It may be obvious that relying solely on one modality for answer prediction may result in similar logit distribution because, for a multimodal task, we cannot choose a correct answer only given a single modality. Therefore, we employ KL divergence to constrain the distribution of uni-modal logits including $KL(\\hat{\\mathbf{y}} _ q, \\hat{\\mathbf{y}} _ a), KL(\\hat{\\mathbf{y}} _ a, \\hat{\\mathbf{y}} _ v), KL(\\hat{\\mathbf{y}} _ v, \\hat{\\mathbf{y}} _ q)$, where $\\hat{\\mathbf{y}} _ q$, $\\hat{\\mathbf{y}} _ a$, and $\\hat{\\mathbf{y}} _ v$ denote question, audio, and video bias logits respectively. These constraints drive the mentioned uni-modal logits to be similar. **In other words, Equation (2) only constrains the distribution of uni-modal logits, without involving the multimodal logits.** Therefore, Equation (1) and (2) are different and they do not cause opposite directions. **Experimental validation.** We demonstrate the effectiveness of the combined use of these two loss functions through ablation experiments. The results show a performance drop when either $\\mathcal{L} _ {\\mathrm{d}}$ or $\\mathcal{L} _ {\\mathrm{c}}$ is used alone, demonstrating the importance and necessity of their combined use. **In summary**, by combining $\\mathcal{L} _ {\\mathrm{d}}$ and $\\mathcal{L} _ {\\mathrm{c}}$, we can more effectively reduce bias and enhance collaboration between uni-modal logits, improving the model robustness. We hope the above explanation clarifies your concerns.\n* **Reply for Q4.** **First**, the goal of this manuscript is to provide a robust evaluation dataset and a bias elimination method for the AVQA task. Please see the four-fold contributions of this manuscript. We have clarified the differences between bias mitigation methods for VQA and AVQA tasks (see R1). **Second**, our model is based on the pre-trained model VisualBERT, which has been demonstrated higher robustness compared to smaller models, as shown in Table 7 of [3]. Specifically, LXMERT+GenB outperforms the VQA baseline SAN+GenB by 14.44%. The model performance without MCCD training on two datasets is detailed in our response to Reviewer Zk4k. **Additionally**, we have provided the dataset, code, and a detailed \"readme\" file. We believe these resources will address your concerns.\n\n[4] Anderson, Peter, et al. \"Bottom-up and top-down attention for image captioning and visual question answering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[5] Cadene, Remi, et al. \"Rubi: Reducing unimodal biases for visual question answering.\" Advances in neural information processing systems 32 (2019).\n\n[6] Li, Guangyao, et al. \"Learning to answer questions in dynamic audio-visual scenarios.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[7] Lao, Mingrui, et al. \"Coca: Collaborative causal regularization for audio-visual question answering.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n\n[8] Ragonesi R, Volpi R, Cavazza J, et al. Learning unbiased representations via mutual information backpropagation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 2729-2738.\n\n[9] Zhu W, Zheng H, Liao H, et al. Learning bias-invariant representation by cross-sample mutual information minimization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 15002-15012.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "msRgtxRmLx", "reviewer": "Reviewer_npgc", "summary": "The authors have made a series of contributions to the field of Audio-Visual Question Answering (AVQA). Firstly, the introduction of the innovative dataset MUSIC-AVQA-R, along with its associated evaluation metrics, stands out. This dataset and metrics are crucial for a comprehensive evaluation of AVQA models' reasoning capabilities and their ability to generalize across different scenarios. Secondly, their novel AVQA architecture, which incorporates the MCCD strategy to mitigate training biases, is interesting. This marks the first concerted effort to systematically tackle biases in AVQA tasks, examining both model evaluations and designs. Additionally, the authors have undertaken extensive experiments with both MUSIC-AVQA and MUSIC-AVQA-R, providing robust evidence of the proposed architecture and debiasing strategy's effectiveness and superiority. Finally, their evaluation of 13 multimodal QA methods on the new dataset exposes a significant limitation: these methods struggle to generalize effectively, whether within the distribution or in out-of-distribution contexts.", "questions": "(1) Please see the weaknesses.\n\n(2) I notice that the answer of MUSIC-AVQA is only single word. I mean that the answer of MUSIC-AVQA-R is still a single word. However, I think this may not be in line with the real-world scenario. Because I think the machine may generate a few words rather than a single word.  Which perspectives do you think should be taken to solve it?\n\n(3) There is no more fine-grained split of bias types in the dataset. How do you think we should deal with it?", "limitations": "N/A.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "(1) The robustness of multimodal models is worth exploring in depth. The authors are inspired by robust VQA and propose the first AVQA dataset and corresponding evaluation metrics for robust AVQA. This dataset has a large number of test samples and can provide fine-grained evaluations for both head and tail samples, which is very interesting. The authors also conduct a detailed analysis for this dataset. I think the quality of this dataset would be acknowledged by the community.\n\n(2) The authors suppose that completely avoiding the negative bias in the dataset is challenging. The development of this dataset encourages the community to enhance the model's robustness through the implementation of debiasing strategies. To this end, the authors proposed an AVQA architecture that incorporates the MCCD strategy to overcome the bias. The authors not only explore biases in the AVQA task from the perspective of model evaluations but also from model designs.\n\n(3) The authors carried out extensive experiments on MUSIC-AVQA and MUSIC-AVQA-R to demonstrate the effectiveness and superiority of their proposed architecture and the MCCD strategy. The experimental results in Table 1 and 2 demonstrate the effectiveness of the architecture. In addition, the results also demonstrate the plug-and-play capability of the proposed MCCD strategy. The results in Table 3 demonstrate the effectiveness of the component within the MCCD strategy. I think the experimental results can strongly support their claim for the architecture and MCCD strategy.\n\n(4) The authors evaluate 13 recent multimodal QA methods on the proposed dataset and show their limited ability to generalize not only in-distribution scenarios but also in out-of-distribution situations.  These experiments further strengthen the solidity of this work.\n\n(5) The authors provide detailed descriptions of the dataset and the implementation settings of the architecture. Notably, the authors provide a detailed readme.md file for their uploaded dataset and code.", "weaknesses": "(1) In the dataset developing stage, the authors perform rephrasing for the original question 25 times. For example, given a question in the Appendix B.2 \"Which is the musical instrument that sounds at the same time as the pipa?, the authors may obtain 25 new questions in the MUSIC-AVQA-R. I think, in the test stage, the evaluation metrics would be better considering how much a model can make correct predictions for the mentioned group. For example, if a AVQA model can only provide 5 accurate predictions for the 25 questions, you should devise a more fine-grained evaluation metric to evaluate the robustness of AVQA models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reply for Q1.** For a truly robust AVQA model, it should accurately answer questions that have the same meaning but are presented in different forms. In future work, we will devise more fine-grained evaluation metrics. The metric is as follows:\n$$\nACC=\\frac{\\sum_{i=1}^{n_g} \\operatorname{sgn}(i) }{n_g}, \\\\\\\\\n\\operatorname{sgn}(i) = \n\\begin{cases} \n1 & \\text{if } c_i \\geq \\frac{n_{i}}{2}, \\\\\\\\ \n0 & \\text{else } c_i < \\frac{n_{i}}{2},\n\\end{cases}\n$$\n\nwhere $n_g$ is the number of question groups, n_i denotes the number of rephrased questions in the $i$-th group, $c_i$ is the number of questions accurately answered by AVQA models. In this equation, we consider an AVQA model to be robust if it correctly answers at least half of the total questions in a group.\n\n**Reply for Q2.** We believe that there are two potential solutions to address this issue: dataset construction and model design. First, we can build a small instruction-tuning dataset to guide AVQA models in generating accurate answers. Second, we can choose a generative model, such as AnyGPT [1], as the backbone. This backbone can be trained on the instruction dataset, making it seamlessly applicable to the AVQA task. To mitigate training bias, we can input the uni-modal token representation into MCCD. We will explore the issue you kindly pointed in future work.\n\n**Reply for Q3.** We believe that the bias in datasets can be explicitly collected. For example, we can feed only the audio modality into an AVQA model to predict answers and collect samples that are answered accurately. These samples can then serve as a test bed for audio bias mitigation. This process can also be applied to obtain video and question bias data. By doing so, we can evaluate AVQA models in a more fine-grained manner. We will explore this approach in future work.\n\n[1] Zhan, Jun, et al. \"Anygpt: Unified multimodal llm with discrete sequence modeling.\" arXiv preprint arXiv:2402.12226 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
* **Reply for dataset quality and bias difference between AVQA and VQA.** **First**, we believe the rephrasing quality of MUSIC-AVQA-R can be guaranteed from three aspects: (1) professional annotators, (2) a strict screening mechanism, and (3) strong support from statistical indicators. For example, as mentioned in Section 3.1 and Appendix B, the Fleiss Kappa value used to measure consistency is 0.839, demonstrating high rephrasing quality. **Secondly**, we think the suggestion for comparing (train vs. test_head) and (train vs. test_tail) to be very valuable. The results, shown in "response.pdf," provide insightful analysis. In comparing the training and head splits, we observe that the distributions are similar, except for the "comparative" and "existential" questions. However, when comparing the training and tail splits, the distributions differ significantly. Therefore, performance on the head and tail splits can accurately reflect model robustness in both in- and out-of-distribution scenarios. We will also include this comparison result in the revised version of the manuscript. **Next**, we thank you for highlighting the three debiasing methods for VQA. Specifically, **[1]** generates various counterfactual training samples by masking critical objects in images or words in questions, assigning different ground-truth answers, thereby forcing VQA models to focus on all critical objects and words. **[2]** employs a greedy gradient ensemble strategy to eliminate language biases, including distribution and shortcut biases. Similarly, **[3]** uses a generative adversarial network and knowledge distillation loss to capture both dataset distribution bias and target model bias. To verify the effectiveness of these methods, integrations with non-debiasing VQA methods like UpDn [4] and debiasing methods such as RUBi [5] are conducted. However, we note that these methods are primarily designed for VQA, specifically addressing language bias, rather than for AVQA. The training bias in AVQA contains language, audio, and video bias. Taking STG [6] as an example, the experimental results on the MUSIC-AVQA test set are as follows. We see that this method can achieve 54% overall accuracy only depending on question inputs. It can be seen that relying solely on uni-modal bias can achieve good results. In addition, we can see that only depending on two modalities inputs obtain significant improvement, although AVQA is a task that involves three modalities. These results show that the mentioned training bias plays a key role in predicting answers.  **Therefore, we believe our method should be compared with both non-debiasing and debiasing AVQA methods.** Unfortunately, research on AVQA is still in its infancy, **with COCA being the only existing bias elimination method.** For this reason, our comparison is limited to non-debiasing AVQA methods. Following COCA [7], we compare our method with audio, visual, video, and AVQA baselines. Due to the lack of public code for COCA, we only compare our method with COCA on the MUSIC-AVQA dataset. Additionally, we think MCCD is specifically designed to mitigate three biases in the AVQA task. This strategy captures the audio, video, and language biases respectively, and then reduces the biases by enlarging the distance between the uni-modal logit and fusion logit, constrained by cycle guidance. To verify MCCD's effectiveness and plug-and-play capability, we conduct extensive experiments on two public datasets. We will discuss the aforementioned three works in the related work section. 

| Input Modality | A Question | V Question | AV Question | All |
|:--------:|:-----:|:----------:|:-----:|:----------:|
| Q | 65.19 | 44.42 | 55.15 | 54.09 |
| A+Q | 67.78 | 62.75 | 63.86 | 64.26 |
| V+Q | 68.76 | 67.28 | 63.23 | 65.28 |

* **Reply for theoretical analysis of Equation (1).** **First**,  we focus on the relationship between distance and bias. In the context of logit magnitudes being constant, an increase in the distance between the uni-modal logit and the fusion logit generally implies a greater difference in their respective probability distributions. For example, in Figure 2 of "response.pdf", the distance between $A$ and $B'$ is larger than the distance between $A$ and $B$, which also makes the distribution difference (i.e. KL divergence) greater. Generally, considering the fusion logit distribution $X$ and a uni-modal logit distribution $Y$, the KL divergence $KL(Y||X)$ tends to increase as the distance between the logits grows. Furthermore, if we consider the mutual information $ I(Y;X) = H(X) + H(Y) - H(X,Y) $ between $X$ and $Y$, and utilize the definition of KL divergence, we get $ I(Y;X) = H(X) - KL(Y||X) $. This numerically indicates that, while keeping the fusion logit distribution fixed, a distance increase results in a mutual information decrease. This leads to a reduction in correlation between the uni-modal and fusion logit. **Several works [8], and [9] have discussed the relationship between mutual information and debiasing, supporting this observation.** **Next**, we will address why we opted to use the reciprocal of distance in MCCD. Our optimization goal is to maximize the separation between the fusion logit and uni-modal logit. Using $\frac{1}{d}$ as a metric allows for heightened sensitivity to small distance variations when the distance is small. Conversely, when a significant distance is established, further variations in distance have a relatively diminished impact. By employing $\frac{1}{d}$, the loss function becomes more effective in encouraging greater separation when logits are close to each other, thus ensuring that small distances are penalized more heavily. Additionally, $-d$ may be unsuitable: (1) may not handle the mentioned scenario, (2) fails to ensure nonnegativity of the total loss, and (3) may reduce the total loss and gradient. We hope this explanation clarifies the rationale behind MCCD and adequately addresses your concerns.

Meta-review: This paper investigates bias in Audio-Visual Question Answering, introducing a new dataset and a method to mitigate the bias. All questions raised were addressed during the post-rebuttal discussion, leading to a consensus in favor of accepting the submission. The AC finds no significant concerns and agrees with this assessment.
Decision: Accept (poster)