Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

Authors: jie  ma
min  hu
pinghui  wang
wangchun  sun
lingyun  song
hongbin  pei
jun  liu
youtian  du
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 05:37:29

Abstract: Audio-Visual Question Answering (AVQA) is a complex multi-modal reasoning task, demanding intelligent systems to accurately respond to natural language queries based on audio-video input pairs. Nevertheless, prevalent AVQA approaches are prone to overlearning dataset biases, resulting in poor robustness. Furthermore, current datasets may not provide a precise diagnostic for these methods. To tackle these challenges, firstly, we propose a novel dataset, *MUSIC-AVQA-R*, crafted in two steps: rephrasing questions within the test split of a public dataset (*MUSIC-AVQA*) and subsequently introducing distribution shifts to split questions. The former leads to a large, diverse test space, while the latter results in a comprehensive robustness evaluation on rare, frequent, and overall questions. Secondly, we propose a robust architecture that utilizes a multifaceted cycle collaborative debiasing strategy to overcome bias learning. Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9.32\%. Extensive ablation experiments are conducted on the two datasets mentioned to analyze the component effectiveness within the debiasing strategy. Additionally, we highlight the limited robustness of existing multi-modal QA methods through the evaluation on our dataset. We also conduct experiments combining various baselines with our proposed strategy on two datasets to verify its plug-and-play capability. Our dataset and code are available at <https://github.com/reml-group/MUSIC-AVQA-R>.

Main paper: 
 introduction: 
 Humans possess the extraordinary capacity to seamlessly integrate auditory and visual cues, effectively establishing a cohesive relationship between visual and auditory stimuli [1] . Audio-Visual Question Answering (AVQA) [2] [3] [4] [5] 3] seeks to enable intelligent systems to acquire this capability and produce answers based on provided natural language questions. It requires the system to learn high-order interaction representations of the concepts encompassed with audio, video, and language modalities. As is known to us [6] [7] [8] , the high-level reasoning ability of the system mainly relies on large-scale data that does not contain harmful biases or statistical regularities.Nevertheless, completely avoiding the negative bias in datasets seems challenging. Previous studies [9] [10] [11] [12] in visual and extractive QA have investigated the bias from the perspective of changing answer distributions and human-in-the-loop adversarial attacks. Drawing inspiration from these works, Figure 1 : The question in current AVQA datasets is generated by a limited set of predefined templates, which may not be in line with the real-world scenario. Our findings indicate that existing methods [5, 1] such as STG [4] are not robust, which may be attributed to excessive bias learning, such as memorizing statistical regularities between critical question words and answers. several open questions are proposed for the AVQA task, concerning model evaluations and model designs.Question 1: have existing datasets comprehensively measured model robustness? The questions in the current AVQA dataset [13, 5, 14, 3] are generated by a limited set of predefined templates, such as the 33 templates in the MUSIC-AVQA dataset [4] . Fig. 1 shows the samples in the training and test split, which are produced using a predefined template. The observed difference mainly stems from a single word, leading to a limited vocabulary size of only 93 words. This has the potential to deviate from real-world scenarios. Moreover, current datasets cannot reflect the performance on rare or less common samples, which is an important indicator for evaluating model robustness [15, 16] .Question 2: have existing methods overcome the data bias? We found that existing methods [5, 1, 17, 18] such as STG [4] are brittle for the question with rare answers. This may be attributed to memorizing the statistical regularity between critical question words and answers, such as the connection between "Is", "Playing", and "Yes". Specifically, the experimental result [4] shows that STG achieves an accuracy of 54.09% on the test split of MUSIC-AVQA only given questions.In this paper, we present the development of a novel dataset called MUSIC-AVQA-R, which aims to address the first question precisely. The dataset complements MUSIC-AVQA [4] and provides a more refined diagnostic for current AVQA methods. To preserve the inherent bias, we maintain the original training and validation splits of the MUSIC-AVQA dataset. In contrast, we employ a human-machine collaboration mechanism to rephrase the question in the test split. This ensures diverse and natural question forms while remarkably expanding the number of questions from 9,129 to 211,572. We introduce a distribution shift based on answer distributions of specific question types. This allows us to measure performance on both frequent (in-distribution) and rare (out-of-distribution) data simultaneously.To tackle the second question, we propose a robust framework that applies a Multifaceted Cycle Collaborative Debiasing (MCCD) strategy. Specifically, the strategy introduces a novel optimization objective, which enlarges the distribution difference between uni-modal (question, audio, and video) and multi-modal logit. By doing so, our model becomes less prone to learning biases from individual modalities. Intuitively, we cannot choose the correct answer based on only one modality. Hence, MCCD employs cycle guidance to constrain the logit distribution of each modality, thereby promoting the similarity of uni-modal logit distribution. The experimental results demonstrate that our framework yields significant improvements on both datasets, with a particularly notable enhancement of 9.32% observed on the MUSIC-AVQA-R dataset.In summary, our contributions are fourfold: (1) We propose a novel dataset MUSIC-AVQA-R and a set of respective evaluation metrics. This enables us to thoroughly evaluate the reasoning behavior of AVQA models and characterize their generalization capabilities in in-and out-of-distribution scenarios. (2) We present an AVQA architecture that incorporates the MCCD strategy to overcome training biases. To the best of our knowledge, this is the first work to systematically explore biases in the AVQA task from model evaluations as well as model designs. (3) We conduct extensive experiments on MUSIC-AVQA and MUSIC-AVQA-R to verify the effectiveness and superiority of our proposed architecture and debiasing strategy. (4) We evaluate 13 recent multimodal QA methods on the proposed dataset and show their limited ability to generalize not only in-distribution scenarios but also in out-of-distribution situations. 
model robustness evaluation: 
 Despite the notable achievements of QA datasets [19] [20] [21] [22] [23] 3] , they suffer from biases, resulting in incomplete evaluations. In recent years, numerous studies have tackled this issue from various perspectives [24] [25] [26] [27] .One avenue of research [12, 28, 29] reorganizes existing datasets, thereby making the distribution between training and testing splits significantly different or even reversed. The reorganized datasets reflect the performance in the out-of-distribution situation but lack measurement in the in-distribution scenario. To this end, GQA-OOD [30] introduces the distribution shift in both the validation and test splits to assess visual QA models in both scenarios simultaneously. Nevertheless, the number of questions in the GQA-OOD test split is only 2,796, which may not reflect the real generalization ability of visual QA models due to the presence of a limited number of testing samples [31] . Inspired by the adversarial attack, another line of works [32, 33] regard the dataset construction as a game played by two parties: a human annotator and a well-trained model. Only samples generated by humans that successfully attack the model are incorporated into the dataset. In addition, there exists another line of work [14] that complements videos and questions to obtain balanced training data.Different from the mentioned works, our dataset, MUSIC-AVQA-R, not only prioritizes question diversity but also considers the volume of test samples. This enhances the precision and comprehensiveness of robustness evaluation. Moreover, we recognize the formidable challenge of obtaining completely pure training data. As such, we opt to retain the inherent bias present in both the training and validation splits. Our primary objective is to inspire the community to enhance model robustness through the implementation of debiasing strategies, rather than striving for balanced training data. Remarkably, to the best of our knowledge, our dataset is the first AVQA dataset explicitly designed for robustness evaluation.A variety of debiasing QA methods [34] [35] [36] [37] have been proposed to overcome bias memorization. These methods can be divided into four classes [24] : ensemble learning, data augmentation, contrastive learning, and answer re-ranking.Ensemble learning methods [38, 28, 39, 6, 40, 41] typically leverage a combination of a bias learner and a vanilla QA model to comprehensively predict answers. Data augmentation methods [42] [43] [44] [45] generate additional question-answer pairs to balance the data distribution. Based on the positive and negative sample generation, contrastive learning-based methods [46] [47] [48] strive to learn an embedding space where similar sample pairs are closely clustered while disparate ones are distinctly separated. Consequently, the vanilla QA method is optimized jointly through contrastive and QA losses. Answer re-ranking methods [34, [49] [50] [51] [52] primarily focus on reordering the answers predicted by the vanilla QA model to enhance context comprehension, such as vision grounding.To the best of our knowledge, COCA [53] is the only work to mitigate the bias learning in the AVQA task, which first employs causal regularization to intervene bias-irrelevant causal effects and then introspects predictions. Unlike the mentioned works, which only consider language biases, our method considers audio, vision, language biases, and their collaboration. The proposed MCCD strategy features plug-and-play capability, enhancing the debiasing potential of baseline methods. 
dataset creation and analysis: 
 We introduce the first dataset, MUSIC-AVQA-R, to evaluate the robustness of AVQA models. The construction of this dataset involves two key processes: rephrasing and splitting. The former involves the rephrasing of questions in the test split of MUSIC-AVQA, and the latter is dedicated to the categorization of questions into frequent (head) and rare (tail) subsets.The questions within the existing dataset [5, 4] are formulated using a restricted collection of predefined templates. To augment diversity and reality, we employ a rephrasing tool1 to rephrase each question 25 times. To ensure the rephrasing quality, three annotators participate in a verification process where their consensus through voting is required. They are all senior students in the field of information science, with one specializing in computer science and the other two in automation. Their extensive professional background equips them with the ability to assess whether the above rephrasing fulfills the requirement. Rephrasings are incorporated into the dataset only when two or more individuals validate the quality of the modifications. According to the statistics, 92.4% of rephrasings pass this validation, and the Fleiss Kappa value used to measure vote consistency is 0.839. Please see details in Table 4 of Appendix B. These results strongly suggest an exceptionally high quality of the rephrasing efforts. Fig. 2 illustrates the distribution of rephrased questions based on their initial three words. We see that our rephrasing questions have various formats, and the comparison between the two datasets is shown in Fig. 6 of Appendix B. The vocabulary size of our dataset is 465, which is 5x larger than MUSIC-AVQA. These results indicate that our dataset is more in line with the real-world scenario. Furthermore, an expansion of the question count within the test split has been implemented, escalating from 9,129 to 211,572 questions. This substantial increase in the volume of test samples enhances the precision of evaluations for AVQA methods. To provide a precise diagnostic for AVQA models, we introduce a distribution shift based on answer distributions of specific question types, following [30] . Guided by this distribution, we categorize rephrased questions into head and tail, enabling the assessment of in-distribution and outof-distribution performance, respectively. We also utilize the overall performance to assess the model effectiveness on the entire test split. Specifically, to characterize the distribution shift, we first utilize the annotation for question types, including "Existential", "Location", "Counting", "Comparative", and "Temporal", to group questions. Fig. 3(a ) illustrates the answer distribution of the "Temporal" questions within the AVQA task. We see that the answer presents a long-tailed distribution. The distribution of other types is given in Appendix B. It is essential to note that MUSIC-AVQA encompasses three tasks: audio QA, visual QA, and AVQA.Next, we characterize the answer balance using Shannon entropy, expressed as H(A) = -N i=1 p(a i ) log p(a i ), where H(A) is the entropy of an answer set A for a certain question type, N is the number of answer classes, and p(a i ) represents the probability of answer class i. It is important to note that the entropy depends on the number of answer classes, which exhibits significant variability across different question groups. To facilitate meaningful comparisons, we normalize H(A) of each group by log(N ): H(A) = H(A) log(N ) , with log(N ) representing the entropy of a uniform distribution of size N . Refer to Appendix C for detailed proof. Thus, the normalized entropy H(A) indicates the proximity of the distribution H(A) to a (a) Answer distributions of "Temporal" questions in the AVQA task.Audio-Visual (b) Statistics of head and tail samples. uniform distribution. We preserve the group with a normalized entropy below a threshold of 0.9, which aims at selecting imbalanced groups.Finally, we categorize the samples into head and tail classes. We define the tail class as class i with |a i | ≤ 1.2µ(a) following [30] , where |a i | represents the number of samples belonging to answer class i, and µ(a) denotes the average sample count for a group. Consequently, the tail samples are rare, while the head samples are more prevalent within a group. Fig. 3 (b) illustrates the statistics of head and tail samples across various groups within each task. 
method: 
 To mitigate bias learning, we propose a robust AVQA architecture that integrates a multifaceted cycle collaborative debiasing strategy. Fig. 4 illustrates an overview of our proposed architecture. It first learns the uni-modal and multimodal representations using a pre-trained model. Then, the architecture utilizes distinct bias learners to capture uni-modal biases. Finally, a collaborative debiasing strategy is leveraged to magnify the disparity between fusion logit and bias logit, obtained based on multimodality and uni-modality representations, respectively. Meanwhile, a cycle guidance mechanism is employed to maintain the similarity between bias logit. The aforementioned procedure is only carried out in the test split of MUSIC-AVQA. Consequently, our proposed dataset, MUSIC-AVQA-R, allows for a more precise and comprehensive evaluation of models handling data biases.Uni-modal Embedding. Given an AVQA sample comprising a video sequence and a corresponding question, we initially partition the sequence, consisting of visual and audio tracks, into T nonoverlapping pairs of visual and audio segments, denoted as {V t , A t } T t=1 , where each segment spans one second. Subsequently, a distinct embedding layer is employed to acquire uni-modal embeddings. Specifically, we employ a pre-trained VGGish model [54] with fixed parameters, which is a VGG-like audio processing network, to obtain an audio embedding vector. For video embedding vectors, we employ a pre-trained ResNet-18 with fixed parameters on the frames. The VisualBert model [55] is applied to obtain a word-level question embedding vector. To ensure dimension matching, distinct linear layers are applied to the aforementioned vectors, resulting in uni-modal embeddingsA e i , V e i ∈ R T ×768 and Q e i ∈ R l×768, where l is the question length. Uni-and Multi-modal Representation. We leverage VisualBert to obtain both uni-modal and multimodal representations, represented asA c i , V c i , Q c i , M c i ∈ R 768 .In the case of uni-modal learning, we exclusively input the aforementioned uni-modal embeddings to VisualBert. For multi-modal learning, we treat question embeddings as queries, concatenate video and audio embeddings as context, and leverage VisualBert to perform multi-modal interaction. Then, we apply a linear projection on the representation to obtain the multi-modality logit ŷm i ∈ R 42 , where 42 denotes the number of possible answers.Uni-modal Bias Learning. AVQA may involve various harmful uni-modal biases, encompassing biases associated with audio, video, and language, respectively. To capture these uni-modal biases, we utilize a bias learner that takes only one of the three modalities as input. Specifically, distinct non-linear multi-layer perceptron layers serve as the learners, producing the corresponding logit ŷa i , ŷv i , ŷq i ∈ R 42 on the answer space. It should be noted that these bias learners are removed during the testing stage.V i A i Q e i V e i A e i Q c i V c i A c i Q c i M Linear ClassifierCollaborative Debiasing. To eliminate bias learning, we propose a multifaceted cycle collaborative debiasing (MCCD) strategy. It first reduces the bias impact from multiple views by enlarging the dissimilarity between uni-modal and multi-modal logit. This discrepancy enlargement L d is implemented by the joint inverse distance:EQUATIONwhere K is the batch size, α is used to balance optimization, d a i denotes the Euclidean distance between audio logit and multi-modality logit, d v i represents the distance between video logit and multi-modality logit, d q i is the distance between question logit and multi-modality logit, and ϵ = 1e-5 is added to the denominator to avoid division by zero.Intuitively, relying solely on one modality for answer prediction may result in similar logit distributions. Therefore, MCCD employs cycle guidance to constrain the distribution of uni-modal logit. This guidance L c is implemented by the Kullback-Leibler divergence:EQUATIONwhere β is the factor to control weight, L qa = 1 K K i=1 ŷq i (log ŷq i -log ŷa i ) denotes the relative entropy between the question ŷq i and audio logit ŷa i ,L av = 1 K K i=1 ŷa i (log ŷa i -log ŷv i )is the relative entropy between the audio ŷa i and video logit ŷv i , andL vq = 1 K K i=1 ŷv i (log ŷv i -log ŷq i )represents the relative entropy between the video ŷv i and question logit ŷq i . Finally, we utilize the summation of L d , L c and L a to optimize the parameters of our method.L a = -1 K K i=1 y f i log ŷf i isthe loss of answer prediction that is regarded as a multi-classification problem, where y f i , ŷf i denote the one-hot answer label and logit of multi-modality fusion, respectively. The training details are shown in Appendix A. 
dataset and evaluation: 
 MUSIC-AVQA [4] , which contains training, validation, and testing splits with 31,927, 4,568, and 9,129 QA pairs, is developed by gathering questions for 9,288 musical performances. The questions are produced by a limited set of pre-defined templates. The videos, sourced from YouTube, include solo performances, ensembles of the same instruments, and ensembles of different instruments. This dataset consists of three tasks: audio QA, visual QA, and AVQA. The standard accuracy is used to evaluate model performance on the mentioned tasks. To comprehensively assess model robustness, we conduct rephrasing and splitting on the test split, expanding the question count from 9,129 to 211,572. Owing to the introduction of distribution shift, our proposed dataset provides three metrics: head accuracy, tail accuracy, and overall accuracy, to evaluate models precisely. The test split comparison between MUSIC-AVQA and MUSIC-AVQA-R is shown in Appendix D.1. We can see that the latter exhibits a larger test sample space.Table 1 : Experimental results (%) on the MUSIC-AVQA test split. EXIST, LOC, CNT, COMP, and TEMP, which are question types, denote "Existential", "Location", "Counting", "Comparative", and "Temporal", respectively. Avg. denotes the average accuracy. During the data pre-processing stage, audio and video are sampled at rates of 16 kHz and 1 fps, respectively. In the uni-modal embedding module, we employ ResNet-18 and VGGish to obtain 512-dimensional and 128-dimensional embeddings of the visual and audio segments, respectively. In the uni-modal bias learning module, the hidden layer size of the bias learner is set to 768. In the collaborative debiasing module, we set the factors α and β to 1e-2 and 3e-1 for optimization equilibrium, respectively. During the training stage, the initial learning rate is set to 3e-5, decaying by 0.5 every 20 epochs. The maximum epoch and batch size are set to 150 and 64, respectively. We use the Adam optimizer to train our architecture and save the model that achieves the best performance on the validation split. The experiments for all methods, except LAVisH, are conducted using a single NVIDIA Tesla V100 GPU. The experiment for LAVisH is run on two NVIDIA Tesla A100 GPUs. The other details are shown in Appendix D.2.We select 14 previous state-of-the-art multi-modal QA methods as baselines to verify the effectiveness of the proposed architecture and investigate the robustness of these methods. Audio QA methods: FCNLSTM [56] , and CONVLSTM [56] . Visual QA methods: GRU [57] , BiLSTM Attn [58] , HCAttn [59] , and MCAN [60] . Video QA methods: HME [61] , PSAC [62] , and HCRN [63] . AVQA methods: AVSD [13] , LAViT [5] , STG [4] , COCA [53] and LAVisH [1] . We abstain from reassessing COCA on the MUSIC-AVQA-R dataset due to its lack of publicly available code. The baseline introductions are shown in Appendix D.3. Due to the particularly slow computation speed of STG, we do not conduct experiments with STG+MCCD. Due to computing power limitations, we reevaluate LAVisH with a batch size of 2. We conduct experiments on the MUSIC-AVQA test split to validate the effectiveness of the proposed architecture. The results are presented in Table 1 . All methods, except LAVisH, utilize ResNet-18 to encode visual features, whereas LAVisH employs stronger models such as ViT [64] or Swin [65] to acquire visual representations. We first analyze the comparison under the same visual encoding conditions. Notably, compared with COCA, our architecture obtains significant improvements of 4.53% and 4% in the audio and visual QA tasks, respectively. It also achieves the best performance in the AVQA task. Furthermore, our architecture obtains a new state-of-the-art result of 72.20% on the whole question. It is worth mentioning that visual QA methods, like GRU, and video QA methods, such as HME, also exhibit competitive results in the AVQA task, despite lacking one modality as input.We also observe that LAVisH, proposed based on STG, introduces trainable parameters into robust visual encoders, thereby achieving superior results compared to methods employing weaker visual encoders. The results on this dataset can demonstrate the efficacy of these methods to some extent. However, MUSIC-AVQA lacks refined and precise evaluations due to its inherent shortcomings that are introduced in Section 1. Consequently, it is insufficient to evaluate these methods only on this dataset.We conduct experiments on the MUSIC-AVQA-R test split to explore the robustness of the aforementioned methods. Their released codes are employed to conduct this experiment. The results are presented in Table 2 . Several crucial insights arise when combining the results from this table with those from Table 1 . Firstly, audio QA methods, such as CONVLSTM, showcase competitive robustness and even achieve the highest tail accuracy on EXIST questions within the AVQA task. Secondly, the visual QA method MCAN demonstrates noteworthy robustness by obtaining the second-best performance on the test split. Thirdly, the video QA baseline experiences a relatively significant performance degradation, with PSAC, for instance, declining by 16.45%. Notably, HCRN exhibits the lowest performance on both datasets, indicating its poor robustness. Furthermore, the results of AVQA baselines lag behind other types of QA methods like HME, suggesting that their strong performance on the MUSIC-AVQA dataset may rely on memorizing statistical regularities between input modalities and answers. Ultimately, our architecture outperforms others on the test split. Benefiting from the MCCD strategy, it attains the highest head (in-distribution setting) and tail (out-of-distribution setting) results across various question types, providing further evidence of its superior robustness. We also show the overall accuracy of these methods on each type of question. Please see the details in Appendix D.4. It can be seen that our architecture achieves the best overall accuracy on each type of question.To validate the plug-and-play capability of MCCD, we conduct extensive experiments using the baselines+MCCD on the aforementioned datasets. The results are presented in Tables 1 and 2 . We observe that MCCD consistently improves performance across most methods on MUSIC-AVQA (9 out of 13) and MUSIC-AVQA-R (11 out of 13), respectively. This underscores the robust debiasing capability of MCCD in a plug-and-play manner. To verify the debiasing effectiveness of MCCD, we conduct extensive experiments on both the test split of MUSIC-AVQA and MUSIC-AVQA-R. The results are shown in Table 3 . Firstly, we validate the contribution of the component within multifaceted debiasing. It can be seen that removing the component will lead to an overall performance improvement in some aspects of MUSIC-AVQA while resulting in a significant decrease in our dataset. This observation strongly supports the debiasing efficacy of these components. Secondly, we verify the overall contribution of the multifaceted debiasing. It can be seen that the performance decrease of 0.72% and 2.15% occurs in both datasets, respectively. Finally, we validate the contribution of cycle guidance. We see that this model variant obtains the best performance on the MUSIC-AVQA dataset. However, there was a noticeable performance degradation in our proposed dataset. In summary, each component plays a distinctive role in the debiasing process, which is further demonstrated by the performance degradation on the head and tail samples.We employ the control variable method to perform a sensitivity analysis on the weight-controlling factors of the MCCD strategy. The results are presented in the left part of Fig. 5 . Our findings indicate stable optimization across various settings, except for the case with α = 0.008 and β = 0.3. Upon conducting further experimental analysis, we identify the issue as originating from the model's failure to converge. Moreover, we visualize the attention weight on the uniformly sampled audio and video frames to qualitatively analyze the debiasing capability of our method. The visualization, displayed in the right part of Fig. 5 , reveals that crucial audio and video frames for QA consistently receive significant attention, both in in-and out-of-distribution settings. This further demonstrates that our method predicts answers through the grounding capabilities of audio and vision rather than relying on bias learning. More cases are shown in Appendix D.5. 
conclusion and limitation: 
 We are the first to investigate bias learning in the AVQA task from model evaluation and design aspects. On the one hand, we construct a new dataset, MUSIC-AVQA-R, which evaluates the performance on the head, tail, and overall samples, providing a precise measure of model robustness. On the other hand, we introduce a robust architecture employing the MCCD strategy to mitigate bias learning. Extensive experiments demonstrate the effectiveness of our architecture and the plug-andplay debiasing capability of MCCD. Furthermore, we reevaluate previous multi-modal QA methods on our proposed dataset, revealing their poor robustness.Due to constraints imposed by MUSIC-AVQA, the answer space of our dataset is limited, comprising only 42 classes, and answer lengths are typically confined to a single word. This deviation from real-world scenarios is noteworthy. Concerning model designs, for a fair comparison with baselines, we do not select large generative models to be backbones. However, compared with the answer classification, it may be more useful to generate answers for the AVQA task.  
 
 Appendix: MUSIC-AVQA [4] has undergone meticulous pre-processing tailored for academic research purposes. We ensure the absence of any information that discloses the names or uniquely identifies individuals, as well as avoiding any offensive content. We only perform rephrasing and splitting for the questions within the dataset to develop MUSIC-AVQA-R, which preserves its inherent characteristics.We select questions from both the head and tail splits to demonstrate the diversity of our dataset, as illustrated in Figures 10 and 11 . Due to the use of pre-defined templates, the questions in the train and test splits of MUSIC-AVQA differ by only a single word. In contrast, the questions in our dataset exhibit a variety of formats, which better reflect real-world scenarios. Additionally, our dataset encompasses a larger test space compared to MUSIC-AVQA. Given a probability density function f (x) that conforms to a uniform distribution with size N , its entropy H(X) can be calculated as follows:EQUATIONEQUATIONwhere p(x i ) is the probability of X = x i .In a uniform distribution, each probability p(x i ) is the same, i.e., p(EQUATIONThen, bring the 1 N term outside the summation:EQUATIONThirdly, move the negative sign inside the logarithm:EQUATIONFinally, combine 1 N with the summation: EQUATIONThe test split comparison between MUSIC-AVQA and MUSIC-AVQA-R is shown in Table 5 . We can see that our proposed dataset exhibits a larger test sample space. This can provide a more precise evaluation for the model robustness. The number of trainable parameters of our model is 117M. In the default settings, our model training takes about 20 hours. We initialize the seed for both Numpy and Torch to 42. The other details can be found in our uploaded code.The audio QA baselines are as follows.• FCNLSTM 2 employs a fully convolutional network and LSTM to initially learn the representations of audio and questions separately. Subsequently, it projects the concatenated features of both into the answer space. • CONVLSTM is a variant of FCNLSTM, incorporating five convolutional blocks identical to VGGNet for acquiring a variable-sized representation of audio.The visual QA baselines are as follows.• GRU (dubbed "deeper LSTM + Norm I" in the published paper) is a simple baseline that first uses VGGNet and LSTM to encode the images and questions and then maps the concatenated features of them into the answer space. • BiLSTM Attn is an attention-based bi-directional LSTM network, which was often used in previous relation classification. • HCAttn 3 is a hierarchical co-attention method that employs question-and image-guided attention to reason on images and questions, respectively. • MCAN 4 is a deep modular co-attention network comprised of cascaded modular co-attention layers, where attention is implemented through multi-head attention in Transformers.The video QA baselines are as follows.• HME 5 is a heterogeneous memory-enhanced multimodal attention model that can effectively learn global context information from appearance and motion features. • PSAC 6 employs positional self-attention block to model the dependency between question words and video frames, respectively. It utilizes a co-attention mechanism to perform multi-modal interaction. • HCRN 7 is a hierarchical conditional relation network that embeds video input at various granularities, encompassing frames, short clips, and entire video levels.The AVQA baselines are as follows.• AVSD 8 is a simple but effective audio-visual dialog method. It initially encodes the input modalities separately and subsequently feeds their fused features into a LSTM to generate answers. • LAViT 9 is a spatial AVQA framework that utilizes three distinct Transformer blocks to perform interaction between input modalities. • STG 10 associates particular visual locations with audio to conduct spatial grounding. Based on this, audio and visual features of key timestamps are further emphasized through question queries for temporal grounding. • LAVisH 11 , based on STG, incorporates trainable parameters into powerful visual encoders such as ViT and Swin.Table 6 illustrates the overall accuracy for specific types of questions. Notably, our architecture surpasses all baselines across every question type. Importantly, our architecture achieves the highest performance in all three tasks, underscoring the idea that the additional modality serves as a valuable complement. For instance, the audio modality may enhance AVQA models in the video QA task. In the video, how many bagpipe can be heard playing? We visualize attention weight on extra uniformly sampled audio and video frames to qualitatively analyze the debiasing capability. In Fig. 12 , the visualization is presented for a more extensive range of head and tail samples. We can see that our method can focus on the key audio and video frames for QA simultaneously in both in-and out-of-distribution settings. For instance, in the upper head and tail case, our method demonstrates high attention to various instruments, leading to accurate answers for "counting" questions. This serves as additional evidence supporting the debiasing effectiveness of our proposed MCCD strategy, highlighting its substantial contribution to improving model robustness.https://quillbot.com/paraphrasing-tool 


Decision: This paper investigates bias in Audio-Visual Question Answering, introducing a new dataset and a method to mitigate the bias. All questions raised were addressed during the post-rebuttal discussion, leading to a consensus in favor of accepting the submission. The AC finds no significant concerns and agrees with this assessment.

Reviews: review: paper_summary: paper_summary: This paper first analyzes the bias in the existing benchmark (MUSIC-AVQA) on Audio-Visual Question Answering (AVQA). They find MUSIC-AVQA testing set is created using templates and only has 9k QA pairs with limited vocabulary, leading to the potential correlation between words in questions and answers. Therefore, previous methods optimized on MUSIC-AVQA could "memorize" the regularity between questions and answers and may not be proper for AVQA question in real world, which include diverse QA pairs. To evaluate the robustness of the AVQA methods, they use a rephraser to rephrase questions in MUSIC-AVQA testing set, thus create the MUSIC-AVQA-R test set. In addition to the rephrased testing set, they design a model trained with proposed multifaceted cycle collaborative debiasing (MCCD) strategy that decreases the logit between any single modality and multimodality logit. MCCD also includes the loss function to maintain the similarity between bias logits. In experiments, they report results on both MUSIC-AVQA and MUSIC-AVQA-R testing sets. Comparing with previous methods, their proposed approach gets good performances in both MUSIC-AVQA and MUSIC-AVQA-R.
strength_weakness: strength_weakness: (1) In ablation study, have the authors tried to only finetune the linear classifier using multimodal features? Without training with MCCD, I would like to see the performances on MUSIC-AVQA-R and MUSIC-AVQA. 
(2) VisualBERT is originally trained with only image and text, but in the paper it seems VisualBERT is used to encode the audio information as well. Do the authors finetune VisualBERT during training? 
(3) Even though the paper is easy to follow, the presentation can still be improved. In Figure 4, "Unimodal Bias Learning" has "Audio Bias Learner" but the input is actually video features. Some highlighted regions such as the purple region is not explained well.
questions: questions: See weakness
limitations: limitations: yes
paper_summary: paper_summary: Prevalent Audio-Visual Question Answering (AVQA) methods often learn biases from the dataset, which can lead to reduced robustness. Additionally, current datasets may not accurately assess these methods. To address these issues, the authors introduce MUSIC-AVQA-R, a dataset created in two steps. First, they rephrase questions in the test split of a public dataset (MUSIC-AVQA) and then introduce distribution shifts to the split questions. Rephrasing the questions allows for a diverse test space, while the distribution shifts enable a comprehensive evaluation of robustness for rare, frequent, and overall questions. The authors also propose a robust architecture that uses a multifaceted cycle collaborative debiasing strategy to overcome biased learning.
strength_weakness: strength_weakness: - First, I have a concern about the quality of the new dataset, it is not verified enough whether the dataset is properly collected. Similar to the case of the VQA-CP dataset, the authors should claim that the proposed dataset is a useful testbed to measure the distribution shift out of distribution robustness by showing the distribution difference between the train and test datasets. However, the authors only show the distribution of the rephrased dataset without showing the distribution difference between the train and test datasets. In short, there is no way to see whether the collected data after paraphrasing and distribution shifting is a valid one or not. In other words, how does the resulting distribution differ from the previous distribution?
The proposed solution also lacks novelty. Extensive work has been done to mitigate modality bias in the VQA-CP dataset, such as [1,2,3]. Therefore, in order to study the modality bias problem in AVQA, the authors should also devise a novel solution specialized for the AVQA problem by comparing it with the existing debiasing methods for VQA. 

[1] Counterfactual Samples Synthesizing for Robust Visual Question Answering. CVPR 2020.

[2] Greedy Gradient Ensemble for Robust Visual Question Answering. ICCV 2021.

[3] Generative Bias for Robust Visual Question Answering. CVPR 2023.

- In addition, the authors should discuss why the proposed method is better than existing solutions theoretically. However, there is no explanation for why simply increasing the logit distance among the modalities in Equation 1 reduces the bias. Is there any reference from which the authors adopted the ideas? What I especially don't understand is whether there is any theoretical evidence as to why it is necessary to add the inverse of distance. How about minus? More theoretical discussions are required.
- In addition, the proposed solution itself also doesn’t make sense. Equation 3, the KL divergence between logits, and equation 2, the Euclidean distance, are the same in that they measure the distance between the logits. In theory, when the temperature term in the softmax function is large enough, the kl divergence loss becomes equivalent to the Euclidean distance. It is unclear what is the meaning of putting these two similar losses in opposite directions? The authors should further explain the theoretical motivation for using those loss functions.
- The performance in Tables 1 and 2 is also questionable. If other existing debiasing methods, such as [1,2,3], were applied to the same baseline, could the proposed method still outperform them? The proposed solution will be meaningful only if the proposed method sufficiently outperforms the existing methods.
I don't understand how a simple method in this paper (ours in Table 1) achieves a favorable performance in MUSIC-AVQA with such a simple baseline architecture. What is the baseline model performance without adding the proposed loss functions? In Table 3, how does the model without any proposed loss function already show a favorable performance that outperforms the state-of-the-art methods?
questions: questions: Please refer to the questions in the weakness part.
limitations: limitations: I can’t find a negative societal impact of this work.
paper_summary: paper_summary: The authors have made a series of contributions to the field of Audio-Visual Question Answering (AVQA). Firstly, the introduction of the innovative dataset MUSIC-AVQA-R, along with its associated evaluation metrics, stands out. This dataset and metrics are crucial for a comprehensive evaluation of AVQA models' reasoning capabilities and their ability to generalize across different scenarios. Secondly, their novel AVQA architecture, which incorporates the MCCD strategy to mitigate training biases, is interesting. This marks the first concerted effort to systematically tackle biases in AVQA tasks, examining both model evaluations and designs. Additionally, the authors have undertaken extensive experiments with both MUSIC-AVQA and MUSIC-AVQA-R, providing robust evidence of the proposed architecture and debiasing strategy's effectiveness and superiority. Finally, their evaluation of 13 multimodal QA methods on the new dataset exposes a significant limitation: these methods struggle to generalize effectively, whether within the distribution or in out-of-distribution contexts.
strength_weakness: strength_weakness: (1) In the dataset developing stage, the authors perform rephrasing for the original question 25 times. For example, given a question in the Appendix B.2 "Which is the musical instrument that sounds at the same time as the pipa?, the authors may obtain 25 new questions in the MUSIC-AVQA-R. I think, in the test stage, the evaluation metrics would be better considering how much a model can make correct predictions for the mentioned group. For example, if a AVQA model can only provide 5 accurate predictions for the 25 questions, you should devise a more fine-grained evaluation metric to evaluate the robustness of AVQA models.
questions: questions: (1) Please see the weaknesses.

(2) I notice that the answer of MUSIC-AVQA is only single word. I mean that the answer of MUSIC-AVQA-R is still a single word. However, I think this may not be in line with the real-world scenario. Because I think the machine may generate a few words rather than a single word.  Which perspectives do you think should be taken to solve it?

(3) There is no more fine-grained split of bias types in the dataset. How do you think we should deal with it?
limitations: limitations: N/A.

Comments: title: Letter of thanks
thank you
Letter of Thanks
Response to the authors
Final rating
Thanks for your recognition.
comment: Dear reviewer,

Thank you for improving the rating.

Best Regards,
My concern is resolved and I just raised the score.
Dear reviewer,

Thanks for your effort in reviewing our manuscript.

Best Regards,
The authors have addressed my concerns. I raised the score accordingly.
Hi Reviewer b8Yd,

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you,
AC
First, we greatly appreciate your suggestions regarding fine-grained evaluation metrics, generative debiasing AVQA models, and more detailed bias collections. These ideas are highly valuable and will undoubtedly contribute to future work.

Second, the inclusion of additional theoretical analyses for Equation (1) and the experimental results without training MCCD will significantly enhance the depth of our method, both theoretically and empirically. We will be incorporating them into the appendix of our manuscript.

Thank you once again for your thorough and thoughtful review of our work.
Thank you for your reply. Your response has effectively addressed my confusion and concerns. I also notice the explanations in the global reply. I believe the theoretical analysis is essential, as it enhances the depth of the manuscript. It would be beneficial to include this analysis in the appendix of the revised version. As a result, I have improved the ratings for this manuscript.