Combating Visual Question Answering Hallucinations via Robust Multi-Space Co-Debias Learning

Authors: jiawei  zhu
yishu  liu
huanjia  zhu
hui  lin
yuncheng  jiang
zheng  zhang
bingzhi  chen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 09:47:01

Abstract: The challenge of bias in visual question answering (VQA) has gained considerable attention in contemporary research. Various intricate bias dependencies, such as modalities and data imbalances, can cause semantic ambiguities to generate shifts in the feature space of VQA instances. This phenomenon is referred to as VQA Hallucinations. Such distortions can cause hallucination distributions that deviate significantly from the true data, resulting in the model producing factually incorrect predictions. To address this challenge, we propose a robust Multi-Space Co-debias Learning (MSCD) approach for combating VQA hallucinations, which effectively mitigates bias-induced instance and distribution shifts in multi-space under a unified paradigm. Specifically, we design bias-aware and prior-aware debias constraints by utilizing the angle and angle margin of the spherical space to construct bias-prior-instance constraints, thereby refining the manifold representation of instance de-bias and distribution de-dependence. Moreover, we leverage the inherent overfitting characteristics of Euclidean space to introduce bias components from biased examples and modal counterexample injection, further assisting in multi-space robust learning. By integrating homeomorphic instances in different spaces, MSCD could enhance the comprehension of structural relationships between semantics and answer classes, yielding robust representations that are not solely reliant on training priors. In this way, our co-debias paradigm generates more robust representations that effectively mitigate biases to combat hallucinations. Extensive experiments on multiple benchmark datasets consistently demonstrate that the proposed MSCD method outperforms state-of-the-art baselines.

Main paper: 
 introduction: 
 The visual question answering (VQA) task aims to build an agent proficient in collaborative reasoning, leveraging question reasoning and image semantics for making informed predictions. Previous methods have shown significant performances [1, 2, 37] , which typically train VQA models on training data and test on benchmark datasets with similar distributions. Regrettably, contemporary research reveals a significant challenge: many methods experience a notable decrease in model generalization performance when the test data deviates from the training distribution. Indeed, some studies attempt to address biases in modality (language, vision) and data distribution. Through the exploration of these biases [5, 11, [18] [19] [20] [21] 30] , it becomes evident that biases in distribution cause the model to learn idiosyncratic biases closely linked to specific modalities and data labels, rather than focusing on the holistic semantics of the vision-question instance. This phenomenon is termed "VQA hallucination" problem, as shown in Figure 1 How many TVs are on?Hallucination Model: 2Baseline Other(30.96%) 2 (37.38%) 1(31.66%)Correct Answer: 1Train SetFigure 1 : Illustration of the VQA Hallucinations: given a question type ("How many..."), we display the answer distributions for the ground-truth answers from the train and test set. The model suffers from factual hallucinations [17, 27] . An intriguing example of this phenomenon is the emergence of an improbable "2" distribution within VQA-CP v2 dataset [1] .these biases can cause instances and distribution shifts [24, 33, 51] in the feature space to produce hallucination distributions.To further investigate the relationship among biases, instance and distribution shifts, and VQA hallucination, we visualize the feature space divided by question types in Figure 2 . In conventional learning methods, semantic ambiguities in instance and distribution shifts become apparent. For instance, considering the question type "What time. . . ": certain features that should predict answers like "morning" or "night" (which have fewer occurrences in the training) are incorrectly clustered with "afternoon" in the feature space. These shifts negatively affect the out-of-distribution (OOD) robustness performance of the model.In the literature, mainstream VQA methods mainly focus on language priors and can currently be categorized into three types: ensemble model-based, data augmentation-based models, and feature space-based models. These approaches encompass the following components: introducing an ensemble bias model [5, 10, 11, 20, 21] to identify and mitigate biases inherent in each modality or dataset, employing manual annotations or data augmentation techniques [7, 8, 31] to mitigate language priors, and utilizing models that focus on fine-grained feature space learning [4, 18, 22] to address biases based on the frequency or distribution of instances in the feature space. While these methods have shown promise in bolstering robustness, their approach of solely targeting partial biases without considering intrinsic patterns has led to suboptimal performance.This work rethinks the relationship between biases, instance and distribution shifts, and hallucination, and attempts to understand "Why do traditional visual question answering methods produce hallucination problems on OOD data". Given the characteristics of feature space, methods relying on Cross-Entropy (CE) loss face challenges in accurately delineating decision boundaries [4, 13, 18, 36, 52] within the feature space, where the span of features for each class needs to be proportional to the corresponding number of instances. In scenarios where the distributions of train and test sets are similar, the model can effectively discern the separability of each instance. However, training in a biased environment leads the model to prioritize fitting the dominant majority answer class while overlooking the minority answer class outside the distribution. This runs counter to the original intention of visual question answering to understand vision-question clues and reason about answers based on semantics. Consequently, instances from the minority class in the test set may experience a shift, leading to a distribution shift. In such cases, the model fails to attend to the semantics of the instance itself and resorts to making arbitrary predictions. This work explains the hallucination phenomenon from the instance and distribution shift caused by biases.Building upon this conceptual framework, our objective is to mitigate the phenomenon of hallucinations, stemming from modal shortcuts and imbalanced dataset distributions. Leveraging the intrinsic patterns and shared sample structures, we aim to unlock the optimal de-hallucination potential of vision-question instances through a multi-space homeomorphism perspective. Simultaneously, we intend to construct bias examples and modal counterexamples to further refine bias learning and counteract biases more effectively. As illustrated in Figure 3 , our proposed Multi-Space Co-Debias (MSCD) method initiates from a Spherical space comprised of angles, employing bias-aware and prior-aware debias constraints. This approach explicitly calibrates instances to generate discriminative manifold representations, while alleviating prior distribution dependence and improving robustness to combat hallucinations caused by instance and distribution shifts. Furthermore, we harness the fitting characteristics of Euclidean space (utilizing softmax CE loss) to introduce biases components from bias examples and modal-counterexamples, aiding the model in robust learning. The primary contributions of our work can be summarized as follows:• This paper introduces a novel multi-space co-debias learning to address the issue of VQA hallucination stemming from biases. It tackles the problem by Spherical and Euclidean spaces co-debias learning into a unified framework. • Two novel bias-aware and prior-aware debias constraints are designed for spherical debias learning and explicitly construct constraints to calibrate instance shift and distribution shift, thereby alleviating VQA hallucinations. • A well-designed multi-space co-debias paradigm is proposed by deploying a two-stage strategy of Euclidean space, assisting spherical debias learning to expose prior correlations and modality-semantics interplay.• Extensive experiments on two biased benchmark datasets and a balanced dataset demonstrate the effectiveness of the proposed MSCD method to combat VQA hallucination and achieve state-of-the-art performance. 
relatedwork 2.1 robustness methods of vqa tasks: 
 Although performance in the VQA task has approached humanlevel performance, susceptibility to VQA hallucinations persists, resulting in insufficient robustness. The introduction of the new benchmark dataset VQA v2 [16] , complementing VQA v1 [3] , includes question-image pairs with similar semantics and diverse answers. The emergence of bias datasets such as VQA-CP v1, VQA-CP v2 [1] , which use different protocols for VQA datasets, provides benchmarks for debias methods. Notably, the answers to the 𝑡𝑟𝑎𝑖𝑛 Existing research on robustness primarily focuses on various methods to mitigate bias, including addressing modality shortcut bias [19, 34, 38, 45] , dataset bias [7, 28, 31, 50] , and comprehensive bias mitigation strategies [4, 18, 20, 21, 25] . We summarize and discuss these works as follows:Methods that addressing modality interplay: Collaborative reasoning between visual and textual cues is pivotal for fostering robust learning in visual question answering [19, 20, 22, 34, 38, 45] . Recent literature delves into this issue by examining theoretical frameworks such as causal inference [39] and confounding factors [44] . Discuss with the instance, the PW-VQA [45] delves into the confounding effect arising from the interplay of vision and language through a causal lens, shedding light on potential biases in both modalities. Similarly, CVIV [38] employs IVar to amplify visual features, enabling the model to pinpoint essential visual cues. These methods offer valuable insights into the collider bias phenomenon within the realm of vision-language interaction.Methods to balance dataset bias: Existing robust learning frameworks attempt to capture specific dataset biases through annotations. The CSS-based methods, as represented by [7, 28, 31] , generates a plethora of counterfactual samples to rectify data bias. This is achieved by strategically masking key objects in images or words in questions. Additionally, certain studies adopt strategies that don't rely on extensive annotation. The D-VQA [50] employs negative samples and branch detection modules to tackle bias both at the feature and sample levels. These methods explore the effectiveness of changing and adapting when distribution priors change.Methods that comprehensive bias mitigation: Given the elusive and intricate nature of bias, the development of comprehensive debias methods has been paramount. These approaches [10, [20] [21] [22] 25] have yielded promising results, showcasing their potential to mitigate bias effectively. The GGE [20] and GGD [21] framework employ a greedy training strategy on both the biased and base models. This approach systematically infuses bias information to aid the model in bias mitigation. The Ensemble-based models represented by GenB [10] alleviate the bias in the basic model by training a bias model.Different from the above methods, we seek to co-debias constraints on the homeomorphism of multi-spaces to combat hallucinations. Inspired by [4, 18, 22, 43, 47] , we solve the inherent pattern of hallucinations from the root by exploring a new multi-feature space paradigm. That is, this poor out-of-distribution performance is caused by instance shift and distribution shift during training. Considering the geometric characteristics of VQA data, we propose a unified perspective that utilizes Euclidean space and Spherical space co-debias to solve the VQA hallucination problem.By harnessing the capabilities of Spherical space learning to bolster instance discrimination and robustness, many works have adopted the normalization of embeddings to the unit hypersphere [23, 35, 41, 42, 55, 56] . Spherical space learning involves constructing an angular space through a regularized classification function, thereby establishing more rational decision boundaries. Among these notable recent works, A-Softmax [35] was developed to refine discriminative face embeddings, enhancing feature discriminability by directly connecting with hypersphere manifolds. HSME [23] focuses on identifying visible thermal human bodies by leveraging hypersphere manifolds and employing metric learning to attain distinct and robust feature representations. Recently, inspiring research in visual question answering has emerged. AdaVQA [18] pioneered the application of angular space to tackle the language prior problem via feature space learning, effectively mitigating bias. RMLVQA [4] proposed a methodology utilizing adaptive margin loss to improve robustness to bias, taking into account answer difficulty and frequency. Different from these approaches, this work employs instance and distribution shifts to mitigate correlations among deep representations distributed on the hypersphere. This strategy enhances the instance discriminability and unbinds it from the prior distribution, ultimately alleviating the VQA hallucinations caused by instance shift and distribution shift. Pri or: A multi-space co-debias learning is built through the instance fusion module and the angle margin module, and the robustness representation is obtained by co-debias the purified Spherical space and Euclidean space.co s ( ) 
methodology 3.1 problem formulation: 
 Following previous research [1, 4, 18] , the VQA task essentially acts as a multi-label classification problem. Without loss of generality, given a batch of data samples,D = {(V 𝑖 , Q 𝑖 ) , A 𝑖 } 𝑁 𝑖=1, where (V 𝑖 , Q 𝑖 ) is the 𝑖-th image-question pairs of samples with the corresponding ground-truth answer A 𝑖 , and 𝑁 is the number of samples. We need to optimize the mapping function F vqa : V × Q → R to get a joint representation R. The VQA base model embeds the two features for fusion to obtain a joint representation: EQUATIONEQUATIONwhere S (𝑖,𝑘 ) is the score of the 𝑖-th instance corresponding to the 𝑘-th answer of the answer candidates. Through training, our goal is to optimize a network F vqa (𝜃 ) that maximizes OOD performance: maxEQUATIONAmong them, 𝜀 𝑂𝑂𝐷 represents robustness evaluation on OOD data.As mentioned before, the MSCD is combined to calibrate instances and distribution, thereby achieving VQA robustness in distribution imbalance. In the following, different spaces have the same sample structure, and we first introduce the loss function for robust multispace co-debias learning.Euclidean Feature Space: Euclidean space represents the most common zero-curvature manifold [12] . In feature space, the two points can be represented on the plane. The commonly used crossentropy (CE) loss separates features of different classes by maximizing the posterior probability of the ground truth class, but suffers from insufficient class discriminability [18, 35, 41] . The 𝐸 (•) model can be trained by optimizing the CE loss as follows:EQUATIONwhere 𝑃 𝑖 denotes the posterior probability of the 𝑖-th instance representation 𝑅 𝑖 when classified under label 𝑎 𝑖 . The 𝑏 𝑖 and 𝑏 𝑗 refer to the weight biases for answer classes 𝑖 and 𝑗. Here, 𝑁 represents the total number of instances and 𝐶 signifies the number of classes. Spherical Feature Space: To optimize the instance representation space, many studies [35, 52, 57] have been conducted to enhance intra-class compactness and inter-class separation, making the representation more discriminative and the angular representations located on the unit sphere space a more reliable classification metric. Following the previous setting [4, 18] , through instance representation regularization, the answer representations of visual question instances can be converted from Euclidean space to angular space, forming a spherical space. Specifically, we initialize the angle margin by utilizing 𝐿 2 -normalization of the weight vector W 𝑖 and the joint representation R 𝑖 to ensure that the posterior probability is determined by the angle 𝜃 𝑖 . Let 𝜃 𝑖 be the angle between R 𝑖 and W 𝑖 . Therefore, the logits for each instance are transformed as:EQUATIONwhere the instance representation R 𝑖 ∥𝑾 𝑖 ∥ = 1, ∥𝑹 𝑖 ∥ = 1, 𝑏 𝑖,𝑗 = 0 is thus distributed on a hypersphere with a radius 𝑠. This makes the normalized loss as:EQUATION)Although we can learn features with angular boundaries through sphere loss, these representations are still not necessarily discriminative [18, 35, 42] . (B)Under the supervision of Spherical learning, instances have geometrically interpretable representations. Different from previous work, we do not modify the loss function [4, 18, 35, 52] .Angle Margin Module (AMM): In VQA, for the predictions of any image-question instances, the classification result depends on the angle 𝜃 , as shown in Figure 4(a) . Giving a motivating VQA binary-class example as a case, regarding the posterior probability 𝑃 𝑖 (determined by cosine distance) of the predicted feature of the 𝑖-th instance, the final result only depends on the angles 𝜃 1 and 𝜃 2 and the maximum value of cosine distance 𝑃 𝑖 can be obtained as the classification result, and it is extended to VQA multi-class task. Therefore, by mapping these representations onto a hypersphere manifold and distinguishing them by angle. Further, following the previous settings [4, 18] , an adaptive angle margin 𝑚 𝑖 is introduced to effectively change the classification decision through the angle margin, as shown in Figure 4(b) . It is formalized as follows:EQUATIONwhere 𝑚[𝑖] is the adaptive instance angle margin. Adding angle margin (𝜃 𝑖 + 𝑚[𝑖]) to the manifold representations enables the model to distinguish between frequent/rare instances. However, the feature space changes dynamically. The feature space of frequently occurring instances is narrower than the original feature space. Subtle biases will further the instances shift, causing the feature space of the class space of frequent instances to become more crowded, and the model has hidden dangers of invisible hallucinations. To address this limitation, this work further constructs a Spherical debias learning method to eliminate biases. Bias-Aware Debias Constraint (BDC): In the Spherical space, adding the angle margins module can help the model learn instancespecific to improve instance discriminability. In fact, the existence of biased instances still damages the effectiveness of semantic learning of instances. Inspired by [9, 15, 48] , we explicitly construct biasaware debias constraints, which prompt the learned instances to further alleviate the impact of biased instances and focus on the learning of image-question semantics. Intuitively, The M 𝑖 = 𝑠 cos (𝜃 𝑖 ) representation without margins is more likely to contain biased information, as shown in Figure 4(a) . Simultaneously, we introduce prior which R 𝑎 𝑖 = 𝑒 𝑎 (A 𝑖 ) from the corresponding answer prior, which is further transformed M 𝑎 𝑖 into angle space:EQUATIONThis work further optimizes the learned manifold representation M 𝑚 𝑖 . Specifically, the goal of BDC is to bring the manifold representation M 𝑚 𝑖 as close as possible to the prior M 𝑎 𝑖 while pushing away from the biased M 𝑖 .Inspired by [14, 53] , we consider generalizing the maximization formula of state entropy to construct bias-aware debias constraint form as follows:EQUATIONwhere 𝑧 𝑖 represents an representation of an data instance 𝑥 𝑖 , which produces unit-norm, i.e. ∥𝑧 𝑖 ∥ 2 = 1. The ⟨z 𝑖 , z 𝑎 ⟩ represents the positive pair and the negative pair ⟨z 𝑖 , z 𝑏 ⟩ is the same. The method is to instantiate the weighting spherical density function constructed by the von Mises-Fisher distribution [14] with 𝜅 > 0. Inspired by the above, we aim to render the manifold representation located on the hypersphere S 𝑑 . For a given pair of points M 1 and M 2 ∈ S 𝑑 , we employ cosine similarity to compute the angle between manifold representations M 1 andEQUATIONwhere 𝜃 1,2 is the angle between the manifold representation M 1 and M 2 . Hence, the instance decision boundary of M 𝑚 𝑖 for prior 𝑎 and biases 𝑖 is 𝜃 𝑖+𝑚[𝑖 ],𝑎 > 𝜃 𝑖+𝑚[𝑖 ],𝑖 , where 𝑎 and 𝑖 are indexes to positive and negative instances, respectively. This is essentially consistent with the prior that the data is distributed on the manifold: 𝐿 BDC = logIn this loss function, the decision boundary of M 𝑚 𝑖 concerning the biases and priors is defined by 𝜃 𝑖+𝑚[𝑖 ],𝑖 and 𝜃 𝑖+𝑚[𝑖 ],𝑎 . This effectively drives M 𝑚 𝑖 closer to the region corresponding to the correct class 𝜃 𝑖+𝑚[𝑖 ],𝑎 . Therefore, this helps calibrate instance shifts caused by bias. In the ideal case, manifold instances are distributed on the unit sphere following instance semantics.Prior-Aware Debias Constraint (PDC): As mentioned before, BDC effectively calibrates instance shift by leveraging bias-aware constraint in spherical space. We rethink our BDC method, which relies on prior conditions in training, potentially resulting in "Hallucination distribution" problems under distribution shift. In a biased training environment, the predicted answers will be similar to the distribution in the training set [18, 21, 33] . In fact, these manifold representations should be semantic correlations between instances and classes rather than fitting training distributions.Therefore, We focus on mitigating prior dependence and optimizing the manifold representations by perceiving the distribution from the prior within the mini-batch. The relationship between the prior and manifold representation is reflected in the Kullback-Leibler divergence [54] 𝐷 (•∥•) of the probability distribution:EQUATIONwhere 𝑝 𝑎 𝑖 and 𝑝 𝑚 𝑖 are the probability distributions in a mini-batch, respectively. Previous studies reduce distribution dependence by manually adding annotations [7, 28, 31] , which adds more additional costs. To make the predictions generated by the manifold representation statistically close to being independent of the prior, forcing it to focus on the semantics, using probability distribution estimation [26, 29, 54] is more general and more conducive to generalization. Therefore, we propose a prior-aware debias constraint to address this issue. The formulation between instance and prior is as follows in Spherical space:EQUATION)where 𝑁 is the number of prior and manifold representations in a mini-batch. In contrast, for an initialized spherical space, spherical constraints can force the model to learn semantic representations by reducing distribution dependence and bias effects.Considering the synergistic properties of multi-space, for each instance, a homeomorphism can be established between Spherical space and Euclidean space. Since different spaces share joint representations, we can further combat the hallucination problem caused by bias through intrinsic connections. Inspired by [4, 32, 46, 49] , we use overfitting characteristics to perform robust learning.Stage I: Bias-Examples Injection (BEI): Following the previous setting [4] , the bias-examples injection component is a classifier appended to the features R 𝑖 trained using the standard CE loss.EQUATIONwhere Â represents the biased prediction of the joint representation R 𝑖 . The BEI combined with BDC can cluster instances in the feature space based on the source of bias to capture the complete semantic structure information while reducing instance offset.Stage II: Modal-Counterexamples Injection (MCI): However, since bias comes from training data, too much BEI may exacerbate distribution dependence to cause adverse effects. To balance the pros and cons, inspired by [31, 50] , we extend the Euclidean space to generate more modal counterexamples for training. Specifically, we construct vision counterexamples (V - 𝑖 , Q 𝑖 , Â) and questions counterexamples (V 𝑖 , Q - 𝑖 , Â) for each instance by randomly generating and selecting in a mini-batch. Intuitively, in VQA, a question can only be answered if the question and image correspond. When provided with a counterexample instance as input, the VQA model fails to provide the correct answer by minimizing the loss.EQUATIONwhere 𝑓 (•) represents the softmax function and 𝑖 is the index of ground-truth answer A 𝑖 in the answer set |A|. Minimizing 𝐿 MCI will encourage the prediction distribution to be far away from the true answer distribution of the bias due to the lack of supporting visual or question information. The MCI combined with PDC can encourage the model to focus more on the overall semantics of the instance, rather than based on the specific prior shortcut.EQUATIONwhere L EDL combines the CE loss of the training bias and modal counterexample injection components. The W(epoch) sets the period after the epoch from 0 to 1.Finally, we combine these components in a multi-space co-debias framework that can systematically work as a whole and mutually benefit from each other by integrating logits from the spaces to achieve robust representation learning and prior reduction. Therefore, the weighted sum of the four regularization losses and the basic VQA classification loss constitutes the total loss:EQUATIONwhere 𝜆 1 , 𝜆 2 and 𝜆 3 are the hyperparameter settings of different components respectively. 
experiments 4.1 datasets and evaluation metric: 
 The hallucination problem within the robust VQA task is challenging, and models that cleverly exploit biases or shortcuts may generate predictions that influence human decisions. Therefore, we select the VQA-CP v1 and VQA-CP v2 [1] datasets as benchmarks to evaluate the performance under changed prior conditions. All experiments adopt the standard evaluation metric [3] . To verify the efficiency and generalization of our proposed model across two sets of biased VQA datasets, we select the most relevant works (Biases Mitigation Methods) for comparison. More baselines details will be introduced in the supplementary materials. We implemented our MSCD framework in PyTorch with a single RTX 3090 GPU, and used the AdamW optimizer with weight decay 0.001. The learning rate is set to 0.001. The batch size 𝐵 is set to 512. The values of all hyperparameters including 𝜆 1 , 𝜆 2 , 𝜆 3 in MSCD are set to 5.0, 1.0 and 5.0, respectively.As shown in Table 1 , Our method achieves the highest overall accuracy, significantly outperforming all state-of-the-art baselines, across two VQA-CP benchmark datasets. These two datasets are specially designed to assess the ability to address the VQA hallucination problem, and the comparison results demonstrate the effectiveness of our method in effectively tackling these challenges. We observe that MSCD achieves gains of at least 1.88% and 2.25% respectively against the state-of-the-art methods. It is worth noting that the MSCD method achieves state-of-the-art performance on 𝑁𝑢𝑚 and 𝑂𝑡ℎ𝑒𝑟 , which have more demanding inference requirements. On VQA-CP v2, 𝑁𝑢𝑚 is significantly improved by 9.54%, which is exactly what this work wants to see. Due to the presence of hallucinations, simpler type problems such as 𝑌𝑒𝑠/𝑁𝑜 correlations are easily distinguished by bias. By simply and directly reducing bias, the performance of 𝑌𝑒𝑠/𝑁𝑜 may be improved, but it will also weaken the model's reasoning ability to a certain extent.For the multi-space co-debias method, the debias effect and cooperation of each component should be guaranteed in Table 2 . To demonstrate the performance of each component in our MSCD framework, 1) Baseline: This general UpDn architecture is used as a baseline model [2] . 2) Baseline w/ Spherical AMM: We train the frequency-instance margin model to initially construct the sphere space [4] . 3) Baseline w/ Spherical BDC: Based on the model, we built BDC and achieved an Overall-CP performance improvement of 0.53%. It reflects the superiority of BDC in alleviating instance offsets and effectively calibrate the offset instances to improve robustness. 4) Baseline w/ Spherical PDC: Achieved an Overall-CP performance improvement of 0.31%, showing an improvement in robustness against dependence on prior distributions, and the effect of motivating the model to pay attention to semantics is positive. 5) Baseline w/ Sph-debias: Combined with BDC and PDC constraints based on spherical space provides a more positive contribution to reduce instance offset and distribution offset to achieve better performance. 6) Baseline w/ co-debias: Based on the optimal performance of co-debias, we conclude that co-debias in different spaces can further improve the robustness to combat VQA hallucinations. Therefore, the above experimental results primarily prove the superiorities of the multi-space co-debias learning.In order to further verify the superiority of our paradigm, we introduce bias branch [20] and impose BDC and PDC constraints on Euclidean embedding and Euclidean prior embedding. Following the same setting and architecture, the results all have different degrees of decline compared to multi-space co-debias learning. We verify the two most important hyperparameters in the model, The first one is about the 𝜆 2 setting in PDC, which represents the correlation between the learned manifold embedding and the prior distribution. As shown in Table 4 , the unbalanced learning process will bias the learned feature representation towards biased data, and when 𝜆 2 in PDC is a negative number, the model performance will drop sharply. This is mainly because when 𝜆 2 is negative, the distribution represented by the manifold depends more on the prior. When 𝜆 2 is positive, especially when it is 1, the model explicitly learns semantics that focuses on the correlation between instances and classes. The second one is about the setting of 𝐸𝑝𝑜𝑐ℎ in MCI. This parameter controls the timing of modal counterexample injection. As illustrated in Table 3 (B), we quantitatively study the impact of 𝐸𝑝𝑜𝑐ℎ epochs of MCI. This phenomenon can be attributed to the fact that after Spherical debias learning, the high-confidence model can improve the model's sensitivity to semantics by further learning error semantics, thereby reducing its reliance on training priors. When 𝐸𝑝𝑜𝑐ℎ exceeds 8, performance will decrease slightly.A too late priming period will inevitably weaken the positive impact of differential learning and thus produce suboptimal results.Featured Space Visualization: As shown in Figure 5 , we visualize the feature distribution of each instance for two question types [6] . It can be noticed that the instances learned by our model in different question types are more closely related to the answer class to which they belong, making it simpler to distinguish the answer class corresponding to each instance. This reduces instance shift and mitigates VQA hallucinations. In addition, we focus on visualizing the answer distribution of question types in Figure 6 . The training set and test set distributions for these types are noticeably different. Our method effectively mitigates VQA hallucinations, which is the focus of this article. In contrast, our MSCD method achieves satisfactory hallucination mitigation effects through the multi-space co-debiasing learning paradigm. Attention Region Visualization: From the result analysis in Figure 7 , due to the influence of biases, baseline failed to find the target object mentioned in the problem in the image, resulting in erroneous predictions. Taking the "plane" picture as an example, the model will naturally generate an illusive answer to "yes" instead of paying attention to the overall semantics of the question and answer. In contrast, our MSCD demonstrates the ability to mitigate hallucinations and enable the modality to focus on semantics to accurately locate the target object with a high degree of confidence, thereby giving precise answers to the questions posed. 
conclusion: 
 This paper establishes a novel robust visual question answering paradigm (MSCD), which simultaneously utilizes multi-space codebiasing in a unified framework to combat VQA hallucinations. We construct bias-aware and prior-aware constraints in spherical space to improve robustness by alleviating bias effects and distribution dependence, encouraging the model to focus on the semantics of instances. Switching to the space homeomorphism perspective, we employ bias example and modal counterexample learning strategies through the over-fitting characteristics of Euclidean space to further assist robust learning. Experiments conducted on three widely used datasets demonstrate the encouraging performance of our approach compared to existing state-of-the-art methods. 
 
 Appendix:  


Decision: The initial ratings are 1 WR, 1 BR, 1 BA and 1 WA, and they are 3 BA and 1 WA after rebuttal. The reviewers all appreciated excellent writing, clear logic, substantial content, detailed exposition, and are satisfied with the authors' responses. The reviewers all appreciated excellent writing, clear logic, substantial content, detailed exposition, and are satisfied with the authors' responses.

Reviews: review: paper_summary: paper_summary: This paper proposes a robust Multi-Space Co-debias Learning (MSCD) approach for combating VQA hallucinations. This method addresses the problem by integrating spherical and Euclidean spaces into a unified co-debiasing framework. It deploys a two-stage strategy where the Euclidean space assists spherical debiasing to expose prior correlations and modality-semantics interplay. Experiments conducted on two biased benchmark datasets and a balanced dataset demonstrate the effectiveness of the proposed MSCD method in combating VQA hallucinations.
strength_weakness: strength_weakness: 1.	The paper presents a novel multi-space co-debias learning to address the issue of VQA hallucination stemming from biases. 
2.	The experiments are solid. The model shows clear improvements over existing models on the out-of-distribution benchmark datasets.
3.	The writing is easy to follow. The arguments are coherent and logically structured. The figures and tables are well-designed and effectively illustrate the points.
limitations: limitations: 1. As shown in Table 2, the proposed method performs well on the out-of-distribution dataset but is weaker than many existing methods on the standard VQA dataset. Please provide a detailed explanation and analysis.
2. The visualization examples are insufficient. In Figure 7, please provide examples for each type of question, especially for the "how many" questions mentioned in the introduction.
paper_summary: paper_summary: This paper tackles debiasing of VQA models through a multi-space co-debias approach.
It takes the issue of VQA models being biased to in-distribution samples and tries to tackle hallucination through different losses that aid in reducing the biases.
strength_weakness: strength_weakness: The performance of the work in the specified field and given experiments are quite high. 
The idea they present is quite simple yet effective.
limitations: limitations: There are a few concerns to this work. 
First of all, which is the baseline that was used in this paper?
It does not say what the baseline is and it seems that the method that the authors are proposing only seems to work on whatever their baseline is. Although in the Ablation section, they claim to integrate the model [4], the overall CP score does not align with the main table.
Because we do not know what the architecture is based of off, it is hard to say this is a fair comparison.

Also, the method seems to be built off of added loses on other methods, but how does the method work on other architectures or baselines?
This would be very helpful in understanding the contribution of this work.

How about other architectures like the LXMERT baseline?
With UpDn being such an old model now, there should be some comparisons to a Transformer based model.

There also has been a lot of controversy regarding the use of VQA-CP2 as a method of testing biases in VQA models, and this paper
On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law, has advocating the use of other baselines to test true ID-OOD settings. 
Are there any scores available for GQA-OOD, VQA-CE, or such?

Also there are a lot of ?s in the figures.
paper_summary: paper_summary: This paper addresses the challenge of bias in visual question answering (VQA) which can cause semantic ambiguities and generate shifts in the feature space of VQA instances leading to VQA Hallucinations. To combat this issue, a Multi-Space Co-debias Learning (MSCD) approach is proposed that effectively mitigates bias-induced instance and distribution shifts in multi-space under a unified paradigm. The approach uses bias-aware and prior-aware debias constraints utilizing the angle and angle margin of the spherical space to construct bias-prior-instance constraints, further enhancing the robustness of representations. Additionally, inherent overfitting characteristics of Euclidean space are leveraged to introduce bias components from biased examples and modal counterexample injection, further assisting in multi-space robust learning. The proposed method generates more robust representations that effectively mitigate biases to combat hallucinations. Experiments conducted on multiple benchmark datasets demonstrate that the proposed MSCD method outperforms state-of-the-art baselines.
strength_weakness: strength_weakness: 1. The paper is well written. 

2. The idea of multi-space co-debias learning sounds interesting.

3. The experiment results well demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. Figure 1 is not well understood. Does it imply that the model exhibits inconsistent levels of hallucination between the training and test sets?

2. It is recommended that the paper includes a sensitivity analysis for lambda_1 and lambda_3 to better understand their impact on the model's performance.

3. Are methods like VCD, which are designed to address multimodal large models, applicable to this problem? If so, what are the effects or improvements they bring to the issue?
paper_summary: paper_summary: This paper introduces MSCD, a novel robust visual question answering paradigm that combats VQA hallucinations. By utilizing multi-space co-debias and constructing bias-aware and prior-aware constraints in Spherical space, MSCD improves robustness by mitigating bias effects and distribution dependencies. Leveraging Euclidean space's overfitting characteristics, the model employs bias example and modal counterexample learning strategies for further robust learning. Experimental results on three datasets demonstrate MSCD's effectiveness, outperforming existing state-of-the-art methods in VQA.
strength_weakness: strength_weakness: 1.	This paper presents MSCD, a novel approach for combating VQA hallucination by integrating Spherical and Euclidean spaces into a unified framework. The method introduces innovative bias-aware and prior-aware debias constraints, specifically designed for spherical debias learning, to address instance shift and distribution shift, effectively alleviating VQA hallucinations. Furthermore, MSCD's two-stage strategy in Euclidean space enhances spherical debias learning by uncovering prior correlations and modality-semantics interplay. 
2.	Extensive experiments on benchmark datasets demonstrate MSCD's effectiveness, achieving state-of-the-art performance and showcasing its potential in addressing VQA hallucination issues stemming from biases.
limitations: limitations: 1.	The paper is confusing in certain areas of writing. For instance, there is no clear explanation for the letter "B" in Equation 3, or for "b_i" and "b_j" in Equation 5, among others. Additionally, there are several typos, such as "night" being written as "nigh" in line 107. These issues in writing quality somewhat affect the readability and comprehensibility of the paper, especially due to the lack of explanations for specific symbols in the equations.
2.	There seem to be some errors in the experimental results section. For example, in Table 1, the performance of AdaVQA in the Y/N-CP column for VQA-CP v1 (91.17) is better than MSCD (90.49) as presented in the paper, but the results for MSCD are incorrectly bolded. Similarly, in Table 2, the performance of UpDn and LMH in the Others column (55.66 and 54.69) is better than GGD (54.66), yet the results for GGD are bolded. These errors or confusing points detract from the quality of the paper and need to be corrected or further explained.
3.	Table 2 shows that MSCD's performance on the Overall metric in VQA v2 is lower than many baseline methods, although it performs better on VQA-CP v2. Does this indicate that the MSCD method sacrifices performance on standard VQA v2 to improve performance on OOD benchmarks? In other words, is this method a trade-off approach designed to enhance the model's performance on OOD benchmarks?