Debiased Visual Question Answering from Feature and Sample Perspectives

1 Introduction . Vision-and-language tasks [ 5 , 43 , 45 ] require an agent to perform visual-textual reasoning , and play an important role in helping humans or intelligent robots to understand the physical world . As a typical case of the vision-and-language task , visual question answering ( VQA ) [ 8 , 35 , 42 ] aims to answer a textual question based on a given image . In this sense , VQA models should master and apply the strong reasoning ability to address the sophistical questions . However , recent studies [ 3 , 23 , 36 ] demonstrate that many VQA models answer the questions without reasoning . Instead , they excessively rely on superficial correlations ( i.e. , bias ) between the question and answers without considering the image . For example , some VQA models tend to answer “ 2 ” for the questions “ How many . . . ” , since most corresponding answers in the dataset are “ 2 ” . In this case , the VQA models memorise the bias but ignore the image , resulting in a poor generalisation ability . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As is widely known , most machine learning datasets inevitably have biases . Thus , how to train a robust model from the biased dataset remains a major challenge . To alleviate the challenge in VQA , most existing methods mainly focus on weakening the language bias [ 9 , 33 ] . Recently , Niu et al . [ 29 ] applied a cause-effect to alleviate language bias . However , the performance is not sufficiently encouraging , and it introduces additional parameters in the inference phase . Note that the biases also exist in the vision modality . As shown in Figure 1 , many answers to the questions are usually the most salient objects/attributes in the images . In this sense , the VQA models may capture this vision bias to output an answer regarding the salient object in the image , without considering the questions . Thus , alleviating the negative effect of vision bias is also crucial . Recently , a few works [ 10 , 17 ] generated counterfactual training samples by masking or transforming critical objects and words to train the models to alleviate language and vision biases . However , they require expensive annotations . More critically , some “ biases ” captured from the dataset may represent the natural rule in real world , i.e. , commonsense knowledge . They are not harmful to the models . Instead , models benefit from them . For example , “ dog ” is a kind of “ animal ” , and the colour of “ oranges ” is normally “ orange ” . Thus , how to filter and remove the true negative biases in language and vision modalities is challenging . To alleviate the above issues , we propose a method named D-VQA to overcome the negative biases in language and vision modalities from feature and sample perspectives . Specifically , ( 1 ) from the feature perspective , we first build a question-to-answer and a vision-to-answer branch to capture language and vision biases , respectively . Since not all biases are detrimental to the model , two unimodal bias detection modules are devised to filter and remove negative biases only . To maintain the inference efficiency , following existing methods [ 9 , 14 , 33 ] , we do not expect to introduce additional parameters in the inference phase . To realise this expectation , we adopt a contrastive loss to approach the multimodal features to the debiased feature as close as possible . In this way , in the inference phase , we simply remove all of the additional branches , and adopt the multimodal features that have a similar representation as the debiased features to obtain accurate predictions . ( 2 ) From the sample perspective , the VQA model should answer the questions making full use of the information of questions and images , which can be achieved by improving the sensitivity of the model in both the images and questions . To this end , for each sample in the dataset , we construct two types of negative samples to assist the training and improve the sensitivity of the model . Our contributions can be summarised as follows : ( 1 ) In VQA , we alleviate the biases in both language and vision modalities , from feature and sample perspectives . ( 2 ) Since not all biases are harmful , we explicitly devise the bias detection modules to filter and remove the negative biases . ( 3 ) Extensive experiments on the VQA-CP v2 and VQA v2 datasets demonstrate the effectiveness of our D-VQA . 2 Related Work . Biases in Visual Question Answering ( VQA ) . In the vision-and-language field , biases are ubiquitous . For example , in referring expression comprehension ( REF ) task [ 15 , 45 , 48 ] , Cirik et al . [ 13 ] argue that the benchmark datasets such as RefCOCO [ 24 , 28 ] contain biases in which the referent objects are usually the most salient in the image . Thus , the models can capture the biases and obtain high accuracy without the queries . Inevitably , the issues of bias also exist in VQA . As demonstrated by [ 2 , 3 , 21 , 36 ] , existing VQA benchmark datasets such as the VQA v2 [ 18 ] have biases , namely , a shortcut exists between the questions and answers . For example , one simply answering “ tennis ” for the question “ What sports . . . ” will obtain high accuracy , because the model is trained and tested on the COCO image dataset that has many “ tennis ” related images . Moreover , most methods consider language bias only , while ignoring the vision bias . As discussed in [ 29 ] , the questions usually involve the objects seen in the image . In addition , as shown in Figure 1 , the most salient objects/attributes are usually chosen to be the answers . In this sense , the VQA models may also capture this vision bias . Thus , in this paper , we consider alleviating the biases in language and vision modalities . Overcoming biases in VQA . Recently , many methods [ 9 , 10 , 16 , 17 , 22 , 26 , 27 , 29 , 33 , 36 , 39 , 40 , 44 , 49 ] have been proposed to overcome the biases in VQA . Formally , these methods can be categorised into two classes , namely , with and without data augmentations . Specifically , nonaugmentation-based methods [ 9 , 16 , 22 , 26 , 29 , 33 , 36 , 44 ] seek to reduce the language biases explicitly or improve attention on the image , while augmentation-based methods [ 1 , 10 , 17 , 27 , 38 , 40 , 49 ] seek to balance the biased dataset for unbiased training . For the non-augmentation-based methods , Ramakrishnan et al . [ 33 ] adopt adversarial learning between the base VQA model and the question-only model to prevent the VQA model from capturing language bias as much as possible . Inspired by [ 33 ] , Cadene et al . [ 9 ] dynamically adjust the weight of the samples based on how biased the samples are . Moreover , some methods introduce human based visual [ 36 ] and text [ 44 ] explanations to strengthen the visual grounding . However , these methods require human annotations that are hard to obtain . Apart from the above , Niu et al . [ 29 ] introduce the cause-effect to look at language bias and propose a counterfactual inference framework to reduce the biases , which unfortunately introduces additional parameters in inference . For the augmentation-based methods , to promote the VQA model to focus on the critical objects and words , Chen et al . [ 10 ] propose a CSS method to produce massive counterfactual samples by masking the critical objects and words , and assign corresponding ground-truth answers . To make full use of the samples , Liang et al . [ 27 ] model the relationships among original , factual and counterfactual samples to promote learning high-level features . Moreover , the Mutant [ 17 ] generates the samples by semantic transformations of the original images or questions . Without introducing additional annotations , some methods [ 38 , 49 ] build negative samples from the samples available to balance the dataset . In this paper , we overcome the bias from sample and feature perspectives . Specifically , from the sample perspective , inspired by [ 38 , 49 ] , we construct two types of negative samples to assist the training of the model . From the feature perspective , we build question-to-answer and vision-to-answer branches to capture the language and vision biases , respectively . Next , we devise two unimodal bias detection modules to recognise and remove the negative biases . Contrastive Learning [ 41 ] aims to learn the high-level representation by maximising the mutual information between the input samples and positive samples , which has been applied in many fields [ 27 , 32 , 47 ] . Specifically , some contrastive learning methods [ 11 , 19 ] adopt contrastive learning to train the models by decreasing the distance between the feature representations of different augmented views of the same images , while increasing the distance between different images . Liang et al . [ 27 ] adopt contrastive learning to approach the feature representations of the original samples to the factual samples , while keeping away from that between the original and counterfactual samples . In this paper , we adopt contrastive learning to push the multimodal feature to approach the debiased features , and meanwhile to far away from the negative biased features . 3 Proposed Method . As demonstrated by [ 3 , 23 , 36 ] , many VQA models address the questions by exploiting the superficial correlation ( i.e. , bias ) between the questions and answers without considering the images , resulting in poor generalisation ability . ( i.e. , performing worse on an out-of-distribution ( OOD ) dataset , e.g. , VQA-CP [ 3 ] ) . Moreover , bias also exists between the images and answers ( as shown in Figure 1 ) , which is ignored by most previous methods [ 9 , 14 , 33 , 38 , 49 ] . However , some “ biases ” learnt from the dataset indicate the natural rule of the world , such as commonsense knowledge . Thus , how to filter and remove the negative biases in language and vision modalities is challenging . In this paper , we alleviate the negative effect of both language and vision biases from feature and sample perspectives . An overview of our D-VQA method is shown in Figure 2 . Specifically , from the feature perspective ( see Section 3.2 ) , we seek to obtain multimodal features that reduce the negative biases as much as possible . To this end , we construct a question-to-answer and vision-to-answer branch to capture the unimodal biases . Moreover , we devise two unimodal bias detection modules to explicitly recognise and remove the true negative biases . That is , we obtain debiased features by simply removing negative biased features from multimodal features . However , using the debiased feature in inference will introduce additional parameters . To address this issue , inspired by contrastive learning [ 11 , 19 ] , we adopt contrastive loss to approach the multimodal features to the debiased features while keeping away from the negative biased features . In this way , in the inference phase , we discard all of the additional branches and adopt the multimodal features to obtain accurate predictions . From the sample perspective ( see Section 3.3 ) , the VQA models are supposed to simultaneously focus on language and vision modalities when answering the questions . Specifically , given a questionimage pair ( positive sample ) , we construct two types of negative samples by randomly sampling an image or question in mini-batch data . The predicted possibility for the ground-truth answer should be minimised if negative samples are used . In this way , VQA models are able to capture the two modalities information to generate the answer instead of focusing on one modality only . Based on the above discussion , our D-VQA method is able to alleviate the negative effect of the biases in language and vision modalities and improve the model performance on the OOD dataset ( e.g. , VQA-CP v2 [ 3 ] ) .

Summary: The paper introduces “D-VQA”, a training approach dedicated to remove harmful biases (both language and vision ones) in VQA models. D-VQA performs in two ways, namely: feature-wise and sampling-wise.  On the features side, during its training, D-VQA attempts to de-bias the features learned by a VQA model by subtracting the (deliberately) biased features learned by unimodal language-only and vision-only models. Moreover, in order to get rid of the unimodal branches during the inference, D-VQA training includes an additional contrastive loss which aims to bring the original VQA features closer to the debiased ones (i.e. the ones obtained after feature subtraction with the unimodal branches).  On the sampling side, D-VQA employs an existing approach proposed in [42] (which is properly mentioned and recognized by the authors) introducing negative sampling triplets for each positive triplet (by randomly changing either an image or a question in a positive triplet).  Combining the 2 aforementioned techniques allows D-VQA to obtain the state-of-the-art results on both VQA-CP and VQAv2 comparing to the existing bias-reduction methods (while using the UpDn model as a backbone). In addition to that, the authors demonstrate that their approach is model-agnostic (for example, it is compatible with a more recent and powerful LXMERT backbone).