Debiased Visual Question Answering from Feature and Sample Perspectives

1 Introduction . Vision-and-language tasks [ 5 , 43 , 45 ] require an agent to perform visual-textual reasoning , and play an important role in helping humans or intelligent robots to understand the physical world . As a typical case of the vision-and-language task , visual question answering ( VQA ) [ 8 , 35 , 42 ] aims to answer a textual question based on a given image . In this sense , VQA models should master and apply the strong reasoning ability to address the sophistical questions . However , recent studies [ 3 , 23 , 36 ] demonstrate that many VQA models answer the questions without reasoning . Instead , they excessively rely on superficial correlations ( i.e. , bias ) between the question and answers without considering the image . For example , some VQA models tend to answer “ 2 ” for the questions “ How many . . . ” , since most corresponding answers in the dataset are “ 2 ” . In this case , the VQA models memorise the bias but ignore the image , resulting in a poor generalisation ability . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As is widely known , most machine learning datasets inevitably have biases . Thus , how to train a robust model from the biased dataset remains a major challenge . To alleviate the challenge in VQA , most existing methods mainly focus on weakening the language bias [ 9 , 33 ] . Recently , Niu et al . [ 29 ] applied a cause-effect to alleviate language bias . However , the performance is not sufficiently encouraging , and it introduces additional parameters in the inference phase . Note that the biases also exist in the vision modality . As shown in Figure 1 , many answers to the questions are usually the most salient objects/attributes in the images . In this sense , the VQA models may capture this vision bias to output an answer regarding the salient object in the image , without considering the questions . Thus , alleviating the negative effect of vision bias is also crucial . Recently , a few works [ 10 , 17 ] generated counterfactual training samples by masking or transforming critical objects and words to train the models to alleviate language and vision biases . However , they require expensive annotations . More critically , some “ biases ” captured from the dataset may represent the natural rule in real world , i.e. , commonsense knowledge . They are not harmful to the models . Instead , models benefit from them . For example , “ dog ” is a kind of “ animal ” , and the colour of “ oranges ” is normally “ orange ” . Thus , how to filter and remove the true negative biases in language and vision modalities is challenging . To alleviate the above issues , we propose a method named D-VQA to overcome the negative biases in language and vision modalities from feature and sample perspectives . Specifically , ( 1 ) from the feature perspective , we first build a question-to-answer and a vision-to-answer branch to capture language and vision biases , respectively . Since not all biases are detrimental to the model , two unimodal bias detection modules are devised to filter and remove negative biases only . To maintain the inference efficiency , following existing methods [ 9 , 14 , 33 ] , we do not expect to introduce additional parameters in the inference phase . To realise this expectation , we adopt a contrastive loss to approach the multimodal features to the debiased feature as close as possible . In this way , in the inference phase , we simply remove all of the additional branches , and adopt the multimodal features that have a similar representation as the debiased features to obtain accurate predictions . ( 2 ) From the sample perspective , the VQA model should answer the questions making full use of the information of questions and images , which can be achieved by improving the sensitivity of the model in both the images and questions . To this end , for each sample in the dataset , we construct two types of negative samples to assist the training and improve the sensitivity of the model . Our contributions can be summarised as follows : ( 1 ) In VQA , we alleviate the biases in both language and vision modalities , from feature and sample perspectives . ( 2 ) Since not all biases are harmful , we explicitly devise the bias detection modules to filter and remove the negative biases . ( 3 ) Extensive experiments on the VQA-CP v2 and VQA v2 datasets demonstrate the effectiveness of our D-VQA . 2 Related Work . Biases in Visual Question Answering ( VQA ) . In the vision-and-language field , biases are ubiquitous . For example , in referring expression comprehension ( REF ) task [ 15 , 45 , 48 ] , Cirik et al . [ 13 ] argue that the benchmark datasets such as RefCOCO [ 24 , 28 ] contain biases in which the referent objects are usually the most salient in the image . Thus , the models can capture the biases and obtain high accuracy without the queries . Inevitably , the issues of bias also exist in VQA . As demonstrated by [ 2 , 3 , 21 , 36 ] , existing VQA benchmark datasets such as the VQA v2 [ 18 ] have biases , namely , a shortcut exists between the questions and answers . For example , one simply answering “ tennis ” for the question “ What sports . . . ” will obtain high accuracy , because the model is trained and tested on the COCO image dataset that has many “ tennis ” related images . Moreover , most methods consider language bias only , while ignoring the vision bias . As discussed in [ 29 ] , the questions usually involve the objects seen in the image . In addition , as shown in Figure 1 , the most salient objects/attributes are usually chosen to be the answers . In this sense , the VQA models may also capture this vision bias . Thus , in this paper , we consider alleviating the biases in language and vision modalities . Overcoming biases in VQA . Recently , many methods [ 9 , 10 , 16 , 17 , 22 , 26 , 27 , 29 , 33 , 36 , 39 , 40 , 44 , 49 ] have been proposed to overcome the biases in VQA . Formally , these methods can be categorised into two classes , namely , with and without data augmentations . Specifically , nonaugmentation-based methods [ 9 , 16 , 22 , 26 , 29 , 33 , 36 , 44 ] seek to reduce the language biases explicitly or improve attention on the image , while augmentation-based methods [ 1 , 10 , 17 , 27 , 38 , 40 , 49 ] seek to balance the biased dataset for unbiased training . For the non-augmentation-based methods , Ramakrishnan et al . [ 33 ] adopt adversarial learning between the base VQA model and the question-only model to prevent the VQA model from capturing language bias as much as possible . Inspired by [ 33 ] , Cadene et al . [ 9 ] dynamically adjust the weight of the samples based on how biased the samples are . Moreover , some methods introduce human based visual [ 36 ] and text [ 44 ] explanations to strengthen the visual grounding . However , these methods require human annotations that are hard to obtain . Apart from the above , Niu et al . [ 29 ] introduce the cause-effect to look at language bias and propose a counterfactual inference framework to reduce the biases , which unfortunately introduces additional parameters in inference . For the augmentation-based methods , to promote the VQA model to focus on the critical objects and words , Chen et al . [ 10 ] propose a CSS method to produce massive counterfactual samples by masking the critical objects and words , and assign corresponding ground-truth answers . To make full use of the samples , Liang et al . [ 27 ] model the relationships among original , factual and counterfactual samples to promote learning high-level features . Moreover , the Mutant [ 17 ] generates the samples by semantic transformations of the original images or questions . Without introducing additional annotations , some methods [ 38 , 49 ] build negative samples from the samples available to balance the dataset . In this paper , we overcome the bias from sample and feature perspectives . Specifically , from the sample perspective , inspired by [ 38 , 49 ] , we construct two types of negative samples to assist the training of the model . From the feature perspective , we build question-to-answer and vision-to-answer branches to capture the language and vision biases , respectively . Next , we devise two unimodal bias detection modules to recognise and remove the negative biases . Contrastive Learning [ 41 ] aims to learn the high-level representation by maximising the mutual information between the input samples and positive samples , which has been applied in many fields [ 27 , 32 , 47 ] . Specifically , some contrastive learning methods [ 11 , 19 ] adopt contrastive learning to train the models by decreasing the distance between the feature representations of different augmented views of the same images , while increasing the distance between different images . Liang et al . [ 27 ] adopt contrastive learning to approach the feature representations of the original samples to the factual samples , while keeping away from that between the original and counterfactual samples . In this paper , we adopt contrastive learning to push the multimodal feature to approach the debiased features , and meanwhile to far away from the negative biased features . 3 Proposed Method . As demonstrated by [ 3 , 23 , 36 ] , many VQA models address the questions by exploiting the superficial correlation ( i.e. , bias ) between the questions and answers without considering the images , resulting in poor generalisation ability . ( i.e. , performing worse on an out-of-distribution ( OOD ) dataset , e.g. , VQA-CP [ 3 ] ) . Moreover , bias also exists between the images and answers ( as shown in Figure 1 ) , which is ignored by most previous methods [ 9 , 14 , 33 , 38 , 49 ] . However , some “ biases ” learnt from the dataset indicate the natural rule of the world , such as commonsense knowledge . Thus , how to filter and remove the negative biases in language and vision modalities is challenging . In this paper , we alleviate the negative effect of both language and vision biases from feature and sample perspectives . An overview of our D-VQA method is shown in Figure 2 . Specifically , from the feature perspective ( see Section 3.2 ) , we seek to obtain multimodal features that reduce the negative biases as much as possible . To this end , we construct a question-to-answer and vision-to-answer branch to capture the unimodal biases . Moreover , we devise two unimodal bias detection modules to explicitly recognise and remove the true negative biases . That is , we obtain debiased features by simply removing negative biased features from multimodal features . However , using the debiased feature in inference will introduce additional parameters . To address this issue , inspired by contrastive learning [ 11 , 19 ] , we adopt contrastive loss to approach the multimodal features to the debiased features while keeping away from the negative biased features . In this way , in the inference phase , we discard all of the additional branches and adopt the multimodal features to obtain accurate predictions . From the sample perspective ( see Section 3.3 ) , the VQA models are supposed to simultaneously focus on language and vision modalities when answering the questions . Specifically , given a questionimage pair ( positive sample ) , we construct two types of negative samples by randomly sampling an image or question in mini-batch data . The predicted possibility for the ground-truth answer should be minimised if negative samples are used . In this way , VQA models are able to capture the two modalities information to generate the answer instead of focusing on one modality only . Based on the above discussion , our D-VQA method is able to alleviate the negative effect of the biases in language and vision modalities and improve the model performance on the OOD dataset ( e.g. , VQA-CP v2 [ 3 ] ) .

Summary: This work addresses a known bias in visual question answering tasks, in which networks learn to pick an answer based on a single modality without perceiving the entire input.  The work novelty comes from proposing a set of regularization strategies s.t: (1) reducing "bad" bias that is not a useful commonsense knowledge;  (2) reducing computational inferences through contrastive loss; (3) considering an image-only bias; (4) employ negative sampling to increase attention sensitiveness on both modalities. In both VQA and VQA-CP, the approach shows state-of-the-art performance.