Title: Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering
Venue: NeurIPS 2024
Accepted: yes
Keywords: audio-visual question answering

bias elimination

debiasing

multimodality learning
Abstract: Audio-Visual Question Answering (AVQA) is a complex multi-modal reasoning task, demanding intelligent systems to accurately respond to natural language queries based on audio-video input pairs. Nevertheless, prevalent AVQA approaches are prone to overlearning dataset biases, resulting in poor robustness. Furthermore, current datasets may not provide a precise diagnostic for these methods. To tackle these challenges, firstly, we propose a novel dataset, *MUSIC-AVQA-R*, crafted in two steps: rephrasing questions within the test split of a public dataset (*MUSIC-AVQA*) and subsequently introducing distribution shifts to split questions. The former leads to a large, diverse test space, while the latter results in a comprehensive robustness evaluation on rare, frequent, and overall questions. Secondly, we propose a robust architecture that utilizes a multifaceted cycle collaborative debiasing strategy to overcome bias learning. Experimental results show that this architecture achieves state-of-the-art performance on MUSIC-AVQA-R, notably obtaining a significant improvement of 9.32\%. Extensive ablation experiments are conducted on the two datasets mentioned to analyze the component effectiveness within the debiasing strategy. Additionally, we highlight the limited robustness of existing multi-modal QA methods through the evaluation on our dataset. We also conduct experiments combining various baselines with our proposed strategy on two datasets to verify its plug-and-play capability. Our dataset and code are available at <https://github.com/reml-group/MUSIC-AVQA-R>.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors have made a series of contributions to the field of Audio-Visual Question Answering (AVQA). Firstly, the introduction of the innovative dataset MUSIC-AVQA-R, along with its associated evaluation metrics, stands out. This dataset and metrics are crucial for a comprehensive evaluation of AVQA models' reasoning capabilities and their ability to generalize across different scenarios. Secondly, their novel AVQA architecture, which incorporates the MCCD strategy to mitigate training biases, is interesting. This marks the first concerted effort to systematically tackle biases in AVQA tasks, examining both model evaluations and designs. Additionally, the authors have undertaken extensive experiments with both MUSIC-AVQA and MUSIC-AVQA-R, providing robust evidence of the proposed architecture and debiasing strategy's effectiveness and superiority. Finally, their evaluation of 13 multimodal QA methods on the new dataset exposes a significant limitation: these methods struggle to generalize effectively, whether within the distribution or in out-of-distribution contexts.\n\nSTRENGTHS:\n(1) The robustness of multimodal models is worth exploring in depth. The authors are inspired by robust VQA and propose the first AVQA dataset and corresponding evaluation metrics for robust AVQA. This dataset has a large number of test samples and can provide fine-grained evaluations for both head and tail samples, which is very interesting. The authors also conduct a detailed analysis for this dataset. I think the quality of this dataset would be acknowledged by the community.\n\n(2) The authors suppose that completely avoiding the negative bias in the dataset is challenging. The development of this dataset encourages the community to enhance the model's robustness through the implementation of debiasing strategies. To this end, the authors proposed an AVQA architecture that incorporates the MCCD strategy to overcome the bias. The authors not only explore biases in the AVQA task from the perspective of model evaluations but also from model designs.\n\n(3) The authors carried out extensive experiments on MUSIC-AVQA and MUSIC-AVQA-R to demonstrate the effectiveness and superiority of their proposed architecture and the MCCD strategy. The experimental results in Table 1 and 2 demonstrate the effectiveness of the architecture. In addition, the results also demonstrate the plug-and-play capability of the proposed MCCD strategy. The results in Table 3 demonstrate the effectiveness of the component within the MCCD strategy. I think the experimental results can strongly support their claim for the architecture and MCCD strategy.\n\n(4) The authors evaluate 13 recent multimodal QA methods on the proposed dataset and show their limited ability to generalize not only in-distribution scenarios but also in out-of-distribution situations.  These experiments further strengthen the solidity of this work.\n\n(5) The authors provide detailed descriptions of the dataset and the implementation settings of the architecture. Notably, the authors provide a detailed readme.md file for their uploaded dataset and code.\n\nWEAKNESSES:\n(1) In the dataset developing stage, the authors perform rephrasing for the original question 25 times. For example, given a question in the Appendix B.2 \"Which is the musical instrument that sounds at the same time as the pipa?, the authors may obtain 25 new questions in the MUSIC-AVQA-R. I think, in the test stage, the evaluation metrics would be better considering how much a model can make correct predictions for the mentioned group. For example, if a AVQA model can only provide 5 accurate predictions for the 25 questions, you should devise a more fine-grained evaluation metric to evaluate the robustness of AVQA models.\n\nQUESTIONS:\n(1) Please see the weaknesses.\n\n(2) I notice that the answer of MUSIC-AVQA is only single word. I mean that the answer of MUSIC-AVQA-R is still a single word. However, I think this may not be in line with the real-world scenario. Because I think the machine may generate a few words rather than a single word.  Which perspectives do you think should be taken to solve it?\n\n(3) There is no more fine-grained split of bias types in the dataset. How do you think we should deal with it?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevalent Audio-Visual Question Answering (AVQA) methods often learn biases from the dataset, which can lead to reduced robustness. Additionally, current datasets may not accurately assess these methods. To address these issues, the authors introduce MUSIC-AVQA-R, a dataset created in two steps. First, they rephrase questions in the test split of a public dataset (MUSIC-AVQA) and then introduce distribution shifts to the split questions. Rephrasing the questions allows for a diverse test space, while the distribution shifts enable a comprehensive evaluation of robustness for rare, frequent, and overall questions. The authors also propose a robust architecture that uses a multifaceted cycle collaborative debiasing strategy to overcome biased learning.\n\nSTRENGTHS:\n- The authors raised a new problem, the bias problem in audio-visual question answering.\n- The authors proposed a new dataset.\n\nWEAKNESSES:\n- First, I have a concern about the quality of the new dataset, it is not verified enough whether the dataset is properly collected. Similar to the case of the VQA-CP dataset, the authors should claim that the proposed dataset is a useful testbed to measure the distribution shift out of distribution robustness by showing the distribution difference between the train and test datasets. However, the authors only show the distribution of the rephrased dataset without showing the distribution difference between the train and test datasets. In short, there is no way to see whether the collected data after paraphrasing and distribution shifting is a valid one or not. In other words, how does the resulting distribution differ from the previous distribution?\nThe proposed solution also lacks novelty. Extensive work has been done to mitigate modality bias in the VQA-CP dataset, such as [1,2,3]. Therefore, in order to study the modality bias problem in AVQA, the authors should also devise a novel solution specialized for the AVQA problem by comparing it with the existing debiasing methods for VQA. \n\n[1] Counterfactual Samples Synthesizing for Robust Visual Question Answering. CVPR 2020.\n\n[2] Greedy Gradient Ensemble for Robust Visual Question Answering. ICCV 2021.\n\n[3] Generative Bias for Robust Visual Question Answering. CVPR 2023.\n\n- In addition, the authors should discuss why the proposed method is better than existing solutions theoretically. However, there is no explanation for why simply increasing the logit distance among the modalities in Equation 1 reduces the bias. Is there any reference from which the authors adopted the ideas? What I especially don't understand is whether there is any theoretical evidence as to why it is necessary to add the inverse of distance. How about minus? More theoretical discussions are required.\n- In addition, the proposed solution itself also doesn’t make sense. Equation 3, the KL divergence between logits, and equation 2, the Euclidean distance, are the same in that they measure the distance between the logits. In theory, when the temperature term in the softmax function is large enough, the kl divergence loss becomes equivalent to the Euclidean distance. It is unclear what is the meaning of putting these two similar losses in opposite directions? The authors should further explain the theoretical motivation for using those loss functions.\n- The performance in Tables 1 and 2 is also questionable. If other existing debiasing methods, such as [1,2,3], were applied to the same baseline, could the proposed method still outperform them? The proposed solution will be meaningful only if the proposed method sufficiently outperforms the existing methods.\nI don't understand how a simple method in this paper (ours in Table 1) achieves a favorable performance in MUSIC-AVQA with such a simple baseline architecture. What is the baseline model performance without adding the proposed loss functions? In Table 3, how does the model without any proposed loss function already show a favorable performance that outperforms the state-of-the-art methods?\n\nQUESTIONS:\nPlease refer to the questions in the weakness part.\n\nLIMITATIONS:\nI can’t find a negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first analyzes the bias in the existing benchmark (MUSIC-AVQA) on Audio-Visual Question Answering (AVQA). They find MUSIC-AVQA testing set is created using templates and only has 9k QA pairs with limited vocabulary, leading to the potential correlation between words in questions and answers. Therefore, previous methods optimized on MUSIC-AVQA could \"memorize\" the regularity between questions and answers and may not be proper for AVQA question in real world, which include diverse QA pairs. To evaluate the robustness of the AVQA methods, they use a rephraser to rephrase questions in MUSIC-AVQA testing set, thus create the MUSIC-AVQA-R test set. In addition to the rephrased testing set, they design a model trained with proposed multifaceted cycle collaborative debiasing (MCCD) strategy that decreases the logit between any single modality and multimodality logit. MCCD also includes the loss function to maintain the similarity between bias logits. In experiments, they report results on both MUSIC-AVQA and MUSIC-AVQA-R testing sets. Comparing with previous methods, their proposed approach gets good performances in both MUSIC-AVQA and MUSIC-AVQA-R.\n\nSTRENGTHS:\n(1) The proposed MCCD strategy is model-agnostic. The experiments in Table 1 show the consistent improvements for previous methods when MCCD is applied on MUSIC-AVQA. \n(2) This paper also contribute a new testing set for the research society to evaluate the robustness of AVQA models. \n(3) The proposed model and MCCD achieves good performances in AVQA, obtaining comparable results on MUSIC-AVQA comparing with previous AVQA methods. It also outperforms previous methods in MUSIC-AVQA-R, showing the robustness of the proposed model.\n\nWEAKNESSES:\n(1) In ablation study, have the authors tried to only finetune the linear classifier using multimodal features? Without training with MCCD, I would like to see the performances on MUSIC-AVQA-R and MUSIC-AVQA. \n(2) VisualBERT is originally trained with only image and text, but in the paper it seems VisualBERT is used to encode the audio information as well. Do the authors finetune VisualBERT during training? \n(3) Even though the paper is easy to follow, the presentation can still be improved. In Figure 4, \"Unimodal Bias Learning\" has \"Audio Bias Learner\" but the input is actually video features. Some highlighted regions such as the purple region is not explained well.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-18"}