Digging out Discrimination Information from Generated Samples for Robust Visual Question Answering
Year: 2023

Abstract:
Visual Question Answering (VQA) aims to answer a textual question based on a given image. Nevertheless, recent studies have shown that VQA models tend to capture the biases to answer the question, instead of using the reasoning ability, resulting in poor generalisation ability. To alleviate the issue, some existing methods consider the natural distribution of the data, and construct samples to balance the dataset, achieving remarkable performance. However, these methods may encounter some limitations: 1) rely on additional annotations, 2) the generated samples may be inaccurate, e.g., assigned wrong answers, and 3) ignore the power of positive samples. In this paper, we propose a method to Dig out Discrimination information from Generated samples (DDG) to address the above limitations. Specifically, we first construct positive and negative samples in vision and language modalities, without using additional annotations. Then, we introduce a knowledge distillation mechanism to promote the learning of the original samples by the positive samples. Moreover, we impel the VQA models to focus on vision and language modalities using the negative samples. Experimental results on the VQA-CP v2 and VQA v2 datasets show the effectiveness of our DDG.

heading: 1 Introduction
text: With the vigorous development of computer vision and natural language processing fields, it has promoted the vision-and-language (Gu et al., 2022; Wen et al., 2023b) field to take a forward step. As a typical task of the vision-and-language field, Visual Question Answering (VQA) (Anderson et al., 2018; Cadène et al., 2019a) requires an agent to fully comprehend the information of the questions and images, and then correctly answer the textual question according to the image. Although recent advances (Cadène et al., 2019a) have achieved impressive performance on the benchmark datasets
∗Corresponding author
(e.g., VQA v2 (Goyal et al., 2017)), numerous studies (Agrawal et al., 2018; Kafle and Kanan, 2017) have shown that some VQA models tend to excessively rely on the superficial correlations (i.e., biases) between the questions and answers, instead of adopting reasoning ability to answer the questions. For example, the VQA models can easily answer “2” and “tennis” for the questions “How many . . . ” and “What sports . . . ”, respectively, would obtain higher accuracy, since the corresponding answers “2” and “tennis” are occupied the most in the dataset. However, memorising the biases to answer the questions would signify flawed reasoning ability, resulting in poor generalisation ability.
To mitigate the bias issues, many methods have been proposed, which can be roughly categorised into three types: 1) enhance visual attention (Selvaraju et al., 2019; Wu and Mooney, 2019), 2) directly weaken the biases (Cadène et al., 2019b; Niu et al., 2021), and 3) balance the dataset (Chen et al., 2020; Zhu et al., 2020). Previous studies have shown the methods that balance the dataset usually outperform other types of methods, since they dig out the natural distribution of the data, and then devise a suitable strategy to overcome the biases. Specifically, CSS (Chen et al., 2020) and Mutant (Gokhale et al., 2020) methods generate counterfactual samples by masking the critical objects or words in the images and questions, respectively. However, these methods require additional annotations that are hard to obtain. To get rid of the dependence on the additional annotations, MMBS (Si et al., 2022) constructs the positive questions by randomly shuffling the question words or removing the words of question types, which destroys the grammar and semantics of the original questions. Moreover, SimpleAug (Kil et al., 2021) and KDDAug (Chen et al., 2022) build the new samples by re-combining the existing questions and images, which may be difficult to assign correct answers for the generated samples. SSL-VQA (Zhu et al.,
6910
2020) and D-VQA (Wen et al., 2021) construct the negative samples by randomly sampling the images or questions in a mini-batch data. Nevertheless, these methods consider the negative samples only, but ignoring the generated positive samples would improve the diversity of the dataset and further promote the robustness of the VQA models.
To overcome the above issues, we propose a method to Dig out Discrimination information from Generated samples (DDG). As pointed out by (Wen et al., 2021), the bias issues exist in both vision and language modalities, we thus construct positive and negative samples in vision and language modalities and devise corresponding training objectives to achieve unbiased learning. Concretely, we feed the samples to the UpDn (Anderson et al., 2018) model pre-trained on the VQA-CP (Agrawal et al., 2018) v2 training set, and select k objects based on the top-k image attention weights of the UpDn as the positive images. The positive questions can be constructed by using the translate-and-back-translate mechanism, e.g., English → French → English. We then combine the positive images and positive questions with original questions and original images, respectively, as positive image and question samples. Based on the positive samples, we adopt a knowledge distillation mechanism (Hinton et al., 2015) to help the learning of the original samples.
Moreover, inspired by (Wen et al., 2021), we construct mismatched image-question pairs as negative samples. Generally speaking, one cannot answer the question correctly given the mismatched image-question pairs, since missing the supporting modality information. To promote the VQA models to focus on the vision and language modalities, we devise a training objective that aims to minimise the likelihood of predicting the ground-truth answers of the original samples when given the corresponding negative samples. Besides, we further introduce the corresponding positive samples to assist the training. Based on the above debiased techniques, our DDG achieves impressive performance on the VQA-CP v2 (Agrawal et al., 2018) and VQA v2 (Goyal et al., 2017) datasets, which demonstrates the effectiveness of our DDG.
Our contributions can be summarised as follows: 1) We devise a novel positive image samples generation strategy that uses the image attention weights of the pre-trained UpDn model to guide the selection of the target objects. 2) We introduce the knowledge distillation mechanism to promote the
learning of the original samples by the positive samples. 3) We adopt the positive and negative samples to impel the VQA models to focus on the vision and language modalities, to mitigate the biases.

heading: 2 Related Work

heading: 2.1 Overcoming biases in VQA
text: Recently, researchers have proposed vast debiased techniques (Selvaraju et al., 2019; Niu et al., 2021; Zhu et al., 2020; Wen et al., 2023a) to alleviate the bias issues in VQA, which can be roughly categorised into three types: 1) enhance the visual attention, 2) directly weaken the biases, 3) balance the dataset. Methods that enhance visual attention. These methods seek to adopt human-annotated information to strengthen the visual attention of the VQA models. Specifically, Selvaraju et al. (Selvaraju et al., 2019) aligned the important image regions identified based on the gradient with the human attention maps to enhance the visual attention in the VQA models. Wu et al. (Wu and Mooney, 2019) introduced a self-critical training objective that matches the ground-truth answer with the most important image region recognised by human explanations. However, these methods require human annotations that are hard to obtain. Methods that weaken the biases. Ramakrishnan et al. (Ramakrishnan et al., 2018) adopted adversarial learning to inhibit the VQA models capture the language biases. Inspired by (Ramakrishnan et al., 2018), Cadene et al. (Cadène et al., 2019b) devised a question-only model to generate weight to re-weight the samples. Moreover, Han et al. (Han et al., 2021) forced the biased models to capture different types of biases, and removed them step by step. Different from the above, Niu et al. (Niu et al., 2021; Niu and Zhang, 2021) introduced the idea of cause-effect to help alleviate the biases. Nevertheless, these methods introduce additional parameters in training or inference phrases. Methods that balance the dataset. CSS (Chen et al., 2020) and Mutant (Gokhale et al., 2020) methods generated massive counterfactual samples by masking the critical objects and words in the images and questions, respectively. However, these methods require additional annotations to assign the answers for the generated samples. To get rid of the dependence on the annotations, KDDAug (Chen et al., 2022) and SimpleAug (Kil et al., 2021) constructed the samples by re-composing the
existing questions and images, which however, is hard to assign the correct answers for the generated samples. SSL-VQA (Zhu et al., 2020) and D-VQA (Wen et al., 2021) constructed the negative samples by randomly sampling the images or questions in a mini-batch data. Nevertheless, these methods ignored the positive samples could improve the diversity of the dataset, which was helpful for improving the robustness of the VQA models. Moreover, Si et al. (Si et al., 2022) constructed the positive question samples by randomly shuffling the words or removing the words of question types, which destroys the semantics of the original questions.
Different from the above methods, we seek to construct positive samples and negative samples in both vision and language modalities, and devise corresponding debiased strategies to achieve unbiased learning.

heading: 2.2 Knowledge Distillation
text: Knowledge Distillation (KD) (Hinton et al., 2015) is a universal model compression method that seeks to train a small student model guided by a large teacher model. Due to the effectiveness of KD, the idea has been applied to other tasks, e.g., longtail classification (He et al., 2021; Xiang et al., 2020), object detection (Chen et al., 2017; Wang et al., 2019), and video captioning (Pan et al., 2020; Zhang et al., 2020). Recently, some debiased VQA methods (Niu and Zhang, 2021; Chen et al., 2022) introduced KD to alleviate the bias issues. Specifically, Niu et al. (Niu and Zhang, 2021) devised two teachers (i.e., ID-teacher and OOD-teacher) to generate “soft” labels to guide the training of the student model (i.e., the baseline model) with the KD mechanism. Inspired by IntroD, Chen et al. (Chen et al., 2022) adopted a multi-teacher KD mechanism to help generate robust pseudo labels for all newly composed image-question pairs. In our DDG, we seek to improve the reasoning ability of the VQA models with the help of the generated positive samples, via the KD mechanism.

heading: 3 Digging out Discrimination Information from Generated Samples
text: As shown by (Agrawal et al., 2018; Kafle and Kanan, 2017), VQA models tend to capture the biases in a dataset to answer questions, instead of adopting the reasoning ability, resulting in poor generalisation ability. Moreover, bias issues exist
in both vision and language modalities (Wen et al., 2021). To address the above issues, we seek to construct both positive and negative samples in vision and language modalities, and devise corresponding debiased strategies to achieve unbiased learning. The overall framework is shown in Figure. 1.

heading: 3.1 Preliminary
text: Visual question answering (VQA) requires an agent to answer a textual question given a corresponding image. Traditional VQA methods (Anderson et al., 2018; Kim et al., 2018; Ben-younes et al., 2019; Cadène et al., 2019a) regard the VQA task as a multi-class classification problem, where each class corresponds to a unique answer. To be specific, given a VQA dataset D = {(vi, qi, ai)}Ni=1 with N samples, where vi ∈ V (image set), qi ∈ Q (question set) are the i-th sample in D, and ai ∈ A (answer set) is a corresponding ground-truth answer, VQA methods seek to learn a multimodal mapping: V ×Q → [0, 1]|A| to generate an answer distribution over the answer set A. Generally speaking, most VQA models usually contain four parts, namely, vision feature encoder ev(·), language feature encoder eq(·), multimodal feature fusion module f(·, ·), and classifier c(·). These modules can be formed as a traditional VQA model:
P (A|vi, qi) = c(f(ev(vi), eq(qi))). (1)
Formally, since regarding the VQA task as a multiclass classification problem, the VQA models can be optimised by a binary cross-entropy loss Lvqa, which can be formulated as:
Lvqa = − 1
N
N∑
i=1
ailog(σ(P (A|vi, qi)))+
(1− ai)log(1− σ(P (A|vi, qi))), (2)
where σ denotes the sigmoid activation function, and ai is the target score obtained based on the answer ai that humans annotated for (vi, qi).

heading: 3.2 Sample Generation
text: Our method aims to adopt the generated samples to achieve unbiased learning. Hence we present how to generate the positive and negative samples at first. As pointed out by (Wen et al., 2021), biases exist in both language and vision modalities. To overcome the bias issue, we seek to generate positive and negative samples regarding the vision and language modalities for each original sample, to assist the training process.
Positive samples generation. To mitigate the bias issue over vision and language modalities in VQA, we build two types of positive samples, i.e., a positive question sample, and a positive image sample.
Specifically, to generate the positive image samples, we seek to draw support from the image attention weights in a pre-trained baseline VQA model. We have empirically found that although the baseline models (e.g., UpDn (Anderson et al., 2018)) achieve unsatisfactory performance in the out-of-distributions (OOD) test set (e.g., VQA-CP v2 (Agrawal et al., 2018) dataset), they still obtain promising performance in the independent and identically distributed (IID) dataset (e.g., VQA v2 dataset (Goyal et al., 2017)). In other words, the baseline models can identify the target objects in the images referred to in the questions to accomplish answering during the training process, regardless of whether capturing the biases. Hence, the image attention weights of the pre-trained UpDn (Anderson et al., 2018) model can help find target objects as positive image samples, which can exclude the background information of the images.
Given the sample (vi, qi) from the VQA- CP v2 training set, we first feed it to the UpDn model pretrained on the VQA-CP v2 training set and would obtain the image attention weights of the UpDn
model regarding the objects in image vi. Note that we select k objects based on the top-k image attention weights as the positive image samples (v+i , qi), where k is a hyper-parameters.
To generate the positive question samples, previous methods (Si et al., 2022) seek to adopt some data augmentation methods to expand the data, e.g., randomly shuffle the question words or remove question category words. However, these methods would severely destroy the grammar and semantics of the original question, resulting in changing the semantic information of the questions. To mitigate this issue, inspired by (Tang et al., 2020), we adopt the translate-and-back-translate mechanism to generate the positive question samples. Specifically, we first use pre-trained English-toFrench and English-to-German translation models to translate the original question to French and German, respectively. Then we use corresponding pretrained back-translation models to translate them back into English. 1 Moreover, we further adopt a pre-trained sentence similarity model to choose a back-translated question sample that has the highest similarity score with the original question as the positive question sample. Note that for some sim-
1All pre-trained translation models are obtained from the Hugging Face repository.
ple questions, they would still keep the same even feeding them to the translate-and-back-translate process. To generate positive question samples for these questions, we substitute the words in the question with synonyms based on the pre-trained synonym word substitution model. 2 In this way, we obtain the positive question samples (vi, q+i ).
Based on the above, we would obtain two types of positive samples (i.e., (v+i , qi) and (vi, q + i )) for each sample (vi, qi), in which the positive image samples have foreground information in the image and the positive question samples are semantic equivalent to the original question. Negative samples generation. Inspired by (Wen et al., 2021), we construct the negative samples over language and vision modalities by randomly sampling one question and one image in a minibatch data for each sample. Specifically, given a mini-batch data {(vb, qb)}Bb=1, for each sample (vi, qi), we randomly sample one image v−i and one question q−i from {(vb, qb)}Bb=1 to form the negative samples, namely, negative question sample (vi, q−i ) and negative image sample (v − i , qi).

heading: 3.3 Generated Samples Driven Robust VQA
text: Positive samples driven robust VQA. We attempt to achieve robust VQA with the help of the generated positive samples. Specifically, given an original sample (vi, qi) and its counterpart positive samples (v+i , qi) and (vi, q + i ), we first feed them into the VQA models to obtain the predictions P (A|vi, qi), P (A|v+i , qi), and P (A|vi, q+i ). Generally speaking, the ensemble predictions usually perform better than the predictions before the ensemble. We thus adopt a simple ensemble strategy (i.e., averaging these predictions) to obtain ensemble predictions Pens = (P (A|vi, qi) + P (A|v+i , qi) +P (A|vi, q+i ))/3. One intuitive way to make the VQA models achieve better performance with the help of the positive samples is to adopt a knowledge distillation mechanism (Hinton et al., 2015). Concretely, we regard the ensemble prediction Pens and the original prediction P (A|vi, qi) as a teacher and a student, respectively, and then introduce a Kullback-Leibler (KL) Divergence Ldis as the objective to optimise the VQA models, which can be formulated as:
Ldis = N∑
i=1
Pens log Pens
P (A|vi, qi) . (3)
2The pre-trained model is from the GitHub repository.
By minimising the KL divergence, the VQA models can extract discrimination information from the positive samples to help better answer the original questions qi correctly based on the images vi.
To guarantee the teacher (i.e., the ensemble prediction Pens) performs better than the student (i.e., the original prediction P (A|vi, qi)), we still use the binary cross-entropy loss Lens on Pens to further optimise the VQA models. Negative samples driven robust VQA. Besides adopting the positive samples to assist the training process, we also introduce the debiased strategy on negative samples to alleviate the bias issues. As shown by (Agrawal et al., 2018), the bias issue usually denotes the VQA models tend to capture the superficial correlations between one modality and the answers to make a prediction on the questions. To mitigate this issue, one direct solution is to improve the attention on both language and vision modalities information when the VQA models answer the questions. We thus consider adopting the negative samples to achieve this aim.
Intuitively, given a mismatched image-question pair, the VQA models even the human being cannot make a correct prediction. Drawing from this insight, when given original samples and the counterpart negative samples, we can alleviate the biases by giving contrary training objectives to the negative samples. This encourages the VQA models to answer the questions by paying more attention to the information of each modality. Concretely, inspired by (Wen et al., 2021), given an original sample (vi, qi, ai) and its counterpart negative samples (v−i , qi) and (vi, q − i ), the VQA models cannot answer correctly when feeding the negative samples, which can be achieved by minimising the possibility of predicting the ground-truth answer:
Lneg = δ(P (A|v−i , qi))[x] + δ(P (A|vi, q−i ))[x], (4)
where x is the index of ground-truth answer ai in the answer set A, and δ is the softmax activation function. Minimising the training objective Lneg encourages the VQA models not to give the ground-truth answer when feeding the mismatched image-question pairs. Thus, the VQA models are able to consider both image and question information before making a prediction, which implicitly alleviates the bias issue.
Moreover, to further enhance the attention of VQA models towards both vision and language modalities, we introduce positive samples into the
training process. Specifically, given positive image samples (v+i , qi, ai) and negative image samples (v−i , qi), when feeding them to the VQA models, one hopes the VQA models can answer correctly with high confidence on the positive samples, while having low prediction confidence to the groundtruth answer with the negative samples. This can be formulated as:
max δ(P (A|v+i , qi))[x]− δ(P (A|v−i , qi))[x].
Maximising the objective encourages the VQA models to make accurate predictions for the matched image-question pairs, while discouraging the models from generating the ground-truth answer when provided with negative image samples. This impels the VQA models to allocate more attention to the vision modality.
By leveraging the monotonicity property of the logarithmic function log, we convert the maximisation problem into an equivalent minimisation problem. This transformation can be mathematically formulated as follows:
Limg = − log(σ(δ(P (A|v+i , qi))[x]− δ(P (A|v−i , qi))[x])) (5)
Moreover, with regard to the negative samples, the prediction scores associated with the groundtruth answer of the corresponding positive samples can serve as an indicator of the extent to which VQA models capture biases. The higher the prediction score δ(P (A|v−i , qi))[x]), the greater the degree of bias it represents. Therefore, it should be subject to a higher penalty in the loss Limg. Inspired by the focal loss (Lin et al., 2017), we consider the prediction score δ(P (A|v−i , qi))[x]) as a measure of the degree of bias, and thus the loss Lweightimg can be reformulated as:
Lweightimg = −δ(P (A|v−i , qi))[x] ∗ Limg (6)
We would obtain Lweightque in the same way. Thus the weighted loss is formulated as follows:
Lweight = Lweightimg + Lweightque . (7)

heading: 3.4 Overall Training Objective
text: In total, our overall training objective can be formulated as:
L = Lvqa+Lens+Ldis+Lneg+λ∗Lweight, (8)
where λ is a hyper-parameter.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our DDG on the OOD dataset VQACP v2 (Agrawal et al., 2018) and IID dataset VQA v2 (Goyal et al., 2017) validation set based on the standard evaluation metric (Antol et al., 2015). Due to the page limitation, we put the implementation details and compared methods into the Appendix.

heading: 4.2 Quantitative results
text: We report the experimental results on the VQA-CP v2 and VQA v2 datasets in Table 1. From these results, we have the following observations: 1) On the whole, the methods that balance the datasets outperform the other two types of methods i.e., enhance visual attention and directly weaken the biases. This demonstrates that alleviating the biases by paying more attention to the natural distribution of the data would obtain higher performance. 2) Our DDG outperforms most compared methods. Specifically, our DDG surpasses SCR (Wu and Mooney, 2019), GGE-DQ (Han et al., 2021), SSL-VQA (Zhu et al., 2020), and KDDAug (Chen et al., 2022) by approximately 12%, 3%, 3%, and 1%, respectively. These results demonstrate the effectiveness of our DDG. 3) Although our method performs slightly worse than the Mutant (Gokhale et al., 2020) and D-VQA (Wen et al., 2021), our DDG achieves higher performance on the VQA v2 dataset. Moreover, Mutant constructed the counterfactual samples highly relying on the additional annotations, while our method build the samples without introducing additional annotations. Meanwhile, compared to the D-VQA method, our method performs better when the data is limited, which can be shown in Table 2. These results further demonstrate the effectiveness of our DDG.
Benefiting from the training process based on the positive samples, our DDG performs better than all the compared methods on the VQA v2 dataset. Specifically, our DDG outperforms SSLVQA (Zhu et al., 2020) and D-VQA (Wen et al., 2021) by around 1.8% and 0.6%, respectively, which demonstrates our DDG is able to improve the model performance on both IID (i.e., VQA v2 dataset) and OOD (i.e., VQA-CP v2) datasets, further implying the superiority of our DDG.

heading: 4.3 Qualitative results.
text: To further demonstrate the effectiveness of our DDG on alleviating the biases, we provide the
qualitative results on the VQA-CP v2 dataset in Figures. 2 and 3. From the results in Figure 2, UpDn (Anderson et al., 2018) and SSL-VQA (Zhu et al., 2020) fail to find the target objects mentioned in the question within the image, leading to erroneous predictions. In contrast, our DDG demonstrates a remarkable ability to accurately localize the target objects with a high degree of confidence, resulting in precise answers to the posed questions. These visualisation results demonstrate the effectiveness of our DDG. Moreover, in Figure. 3, we provide visualisations of the answer distributions obtained by various approaches for different question types, namely "How many . . . ", "Is this . . . ", and "How many people are in . . . ". From the results, we have the following observations: 1) the training answer distribution is different from that in the test set, which is very challenging. 2) The
UpDn model excessively fits the biases in the training set, and thus outputs a similar answer distribution with the training set given the test set, resulting in poor performance. 3) SSL-VQA seeks to alleviate the bias issue, which however is limited. Our DDG is able to alleviate the biases effectively, and thus achieves similar answer distributions with the test set, embodying the better generalisation ability.

heading: 4.4 Ablation studies
text: Effect of the scale of the training set. To demonstrate the effectiveness of our method in the datalimited scenario, we conduct experiments on different scales of the training data. Specifically, on the VQA-CP v2 dataset, we manually split the training set into different proportions (i.e., from 20% to 80% of the original training data), while the test set is unchanged. From the experimental results in Table 2,
we find that our method performs better when the training data is limited, which is more practical and suitable for the real world. Specifically, our method performs better than SSL-VQA (Zhu et al., 2020) and D-VQA (Wen et al., 2021) on any proportions of the training data, especially when only remains 20% of the training data, our DDG outperforms SSL-VQA and D-VQA by around 3%. These results demonstrate the superiority of our DDG in the data-limited scenarios.
Effect of each component of our DDG. We conduct ablation studies on the VQA-CP v2 dataset to evaluate each component in our DDG, and show the experimental results in Table 3. From these results, we have the following observations: 1) when introducing the ensemble binary cross-entropy loss Lens with the positive samples, the model performance improves by around 5% compared with the UpDn (Anderson et al., 2018) model (i.e., 41.53% vs. 46.63%), which demonstrates the positive samples are able to assist the training process to alleviate the bias issue. 2) By incorporating the KL loss Ldis, the performance would be further improved (i.e., 46.63% vs. 47.77%), which highlights the ensemble prediction is able to guide the training of the original prediction. 3) Upon introducing Lneg and Lweight, which leverage negative samples, the performance would improve substantially (i.e., 47.77% vs. 61.14%). This significant enhancement underscores the significance of promoting the attention of VQA models towards both vision and language modalities when answering the questions.
These results further demonstrate the effectiveness of each component in our DDG. Effect of k. k denotes the number of target objects that are selected as the positive samples, which can be referred to in Section 3.2. To demonstrate the effect of the k on the model performance, we conduct experiments on the VQA-CP v2 dataset regarding different k. From the results in Table 4, we have the following observations: 1) with the increase of k (e.g., from 3 to 10), the model performance exhibits a gradual improvement.The results indicate that a higher value of k increases the likelihood that the positive image samples indeed encompass the target objects mentioned in the corresponding questions. 2) Once k exceeds 10, the model performance starts to drop, thereby illustrating that an excessively high value of k introduces extraneous background information that adversely affects the model’s performance. These results demonstrate that an appropriate k helps to obtain the best performance on our DDG. Evaluation of λ. λ is the weight of the loss Lweight. We conduct ablation studies about different λ on the model performance, and show the experimental results in Table 5. From the results, the best performance is obtained in our DDG when λ = 0.05, and the performance is drop whenever λ is higher or lower than 0.05. These results demonstrate that a suitable weight of loss Lweight helps to obtain better performance.
Evaluation of different backbones. Our DDG is model-agnostic. To demonstrate the effectiveness of our DDG on different backbones (i.e., SAN (Yang et al., 2016) and UpDn (Anderson et al., 2018)), we conduct experiments on the VQA-CP v2 dataset, and show the results in Table 6. From the results, our DDG consistently achieves a substantial improvement in model performance, regardless of which backbone it is. These results further embody the superiority of our DDG.

heading: 5 Conclusion
text: In this paper, we have proposed a novel method named DDG to alleviate the bias issues in VQA from vision and language modalities. Specifically, we construct both positive and negative samples in vision and language modalities without using additional annotations, in which the positive questions have similar semantics to the original questions, while the positive images contain foreground information. Based on the positive samples, we heuristically introduce the knowledge distillation mechanism to facilitate the training of the original samples through guidance from positive samples. Moreover, we put forth a strategy that encourages VQA models to focus more on the vision and language modalities when answering the questions, aided by the negative samples. Extensive experiments on the VQA-CP v2 and VQA v2 datasets show the effectiveness of our DDG.

heading: Acknowledgements
text: We would like to thank all the anonymous reviewers for their constructive comments and suggestions. This work was partially supported by STI 2030—Major Projects (2022ZD0208900), Key Realm R&D Program of Guangzhou 202007030007, Program for Guangdong Introducing Innovative and Enterpreneurial Teams 2017ZT07X183, National Natural Science Foundation of China (NSFC) 62072190.
Limitations
The paper focuses on the VQA task only, we will extend our method to other multimodal tasks in future works, e.g., referring expression comprehension (REC). Moreover, although our DDG outperforms most of the state-of-the-art methods, the performance is still a long way from humans.
Ethics Statement
The authors declare that they have no conflict of interest. This paper introduces a novel method named DDG to overcome the bias issue in Visual Question Answering (VQA). Mitigating the biases can impel the VQA models to adopt real reasoning ability to answer the questions, instead of using the captured biases. Hence, this research can promote the development of the AI robot, e.g., dialogue robots, and facilitate people’s daily lives. The failure of the debiased technique may result in the collapse of the VQA system in environments that have seen less or even never seen. Moreover, we evaluate our DDG on the benchmark out-of-distribution (OOD) dataset and demonstrate the remarkable debiased ability.

heading: 6 Appendix

heading: 6.1 Datasets
text: We conduct experiments on the VQA-CP (Agrawal et al., 2018) v2 and VQA (Goyal et al., 2017) v2 datasets. Specifically, the training set of VQA-CP v2 contains approximately 121k images and 483k questions, while the test set contains around 98k images and 220k questions.

heading: 6.2 Compared methods
text: We compare our DDG with existing state-of-theart methods, including 1) methods that enhance visual attention: HINT (Selvaraju et al., 2019) and SCR (Wu and Mooney, 2019). 2) Methods that weaken the biases: AdvReg (Ramakrishnan et al., 2018), RUBI (Cadène et al., 2019b), Re-Scaling (Guo et al., 2022), DLR (Jing et al., 2020), VGQE (KV and Mittal, 2020), LMH (Clark et al., 2019), IntroD (Niu and Zhang, 2021), CFVQA (Niu et al., 2021), RMFE (Gat et al., 2020), CKCL (Pan et al., 2022), LPF (Liang et al., 2021), GGE-DQ (Han et al., 2021), and D-VQA (Wen et al., 2021). 3) Methods that balance the dataset using additional annotations: CSS (Chen et al., 2020), CSS+CL (Liang et al., 2020), CSS+ (Chen et al., 2021), ECD (Kolling et al., 2022), and Mutant (Gokhale et al., 2020). (4) Methods that balance the dataset without introducing additional annotations: CVL (Abbasnejad et al., 2020), Unshuffling (Teney et al., 2021), MMBS (Si et al., 2022), SimpleAug (Kil et al., 2021), RandImg (Teney et al., 2020), SSL-VQA (Zhu et al., 2020), and KDDAug (Chen et al., 2022). Our DDG generates positive and negative samples without introducing additional annotations to help alleviate the biases, which belongs to the methods in the fourth part.

heading: 6.3 Implementation Details
text: Following existing VQA methods (Anderson et al., 2018; Cadène et al., 2019b; Zhu et al., 2020), we extract the top-36 object features with a dimension of 2048 in each image by the Faster-RCNN (Ren et al., 2015) model that is pre-trained by (Anderson et al., 2018). Moreover, each question is first truncated or padded into the same length (i.e., 14), and then encoded by the Glove (Pennington et al., 2014) embedding with a dimension of 300. The dimension of the question encoder (i.e., single layer GRU (Cho et al., 2014)) is 1280.
Inspired by SSL-VQA (Zhu et al., 2020), we introduce one Batch Normalisation (Ioffe and
Szegedy, 2015) layer before the classifier of UpDn (Anderson et al., 2018). We train our method for 30 epochs with the Adam (Kingma and Ba, 2015) optimiser. Specifically, we adopt Lens and Ldis to train the baseline model for 12 epochs, and introduce Lneg and Lweight at the 13-th epoch. The learning rate is set to 1e-3, and decreases by half every 5 epochs after 10 epochs. The batch size is set to 256. We set k and λ to 10 and 0.05, respectively. We implement our method based on PyTorch (Paszke et al., 2019), and the model is trained with one Titan Xp GPU. Moreover, our method does not introduce additional parameters except the backbone model.
Note that our method is model-agnostic and can be applied to different backbones of VQA models. To better demonstrate the effectiveness of our DDG, we conduct experiments based on different backbones, including UpDn (Anderson et al., 2018), and SAN (Yang et al., 2016) in the same settings. Moreover, we perform experiments over three rounds using varying seeds, and present the results in terms of mean values. The source code and the pre-trained models are available at DDG.

heading: 6.4 Training Method
text: We provide the training method of our DDG in Algorithm 1. Specifically, when the training epoch is lower than the threshold τ , we forward the base model Mb with the original and positive samples to calculate the knowledge distillation loss Ldis and binary cross-entropy loss Lens. Moreover, when the training epoch is higher than threshold τ , we construct the negative image and question samples without introducing additional annotations, and then forward Mb with the negative samples and obtain the loss Lneg and Lweight. Finally, we update Mb based on the overall loss L.

heading: 6.5 More Ablation Studies
text: Effect of the training strategy. As shown in Algorithm 1, we adopt knowledge distillation loss Ldis and ensemble binary cross-entropy loss Lens to train the base model for 12 epochs. To demonstrate the effectiveness of the training strategy, we conduct experiments about the training strategies on the VQA-CP v2 dataset, and the results are shown in Table 7. From the results, the strategy that trains the model with the KL loss in the whole training process performs worse than that training for 12 epochs. We infer that the KL loss may accelerate the fitting of the training dataset, which hinders the
Algorithm 1 Training method of our DDG. Require: Training data {(vi, qi, ai)}Ni=1, generated positive image samples { (v+i , qi, ai) }N i=1 , generated
positive question samples { (vi, q + i , ai) }N i=1
a base model Mb, batch size b, threshold τ . 1: Randomly initialise the parameters of Mb. 2: while not converge do 3: Randomly sample a mini-batch data {(vi, qi, ai)}bi=1 from the training data, and obtain the corresponding positive samples {(v+i , qi, ai)}bi=1, and {(vi, q+i , ai)}bi=1. 4: Forward Mb with the training data and the positive samples, and then obtain the predictions P (A|vi, qi), P (A|v+i , qi), P (A|vi, q+i ), and the ensemble prediction Pens. 5: Calculate the binary cross-entropy loss Lvqa for P (A|vi, qi) by Eq.(2). 6: if the training epoch lower than τ then 7: // Introduce Knowledge Distillation Mechanism 8: Calculate the Knowledge Distillation Loss Ldis based on P (A|vi, qi) and Pens via Eq. (3). 9: Calculate the binary cross-entropy loss Lens for Pens by Eq. (2). 10: else 11: // Introduce Negative sample loss 12: Randomly sample images and questions from the mini-batch data {(vi, qi, ai)}bi=1 to form the negative samples as {(v̄i, qi, ai)}bi=1 and {(vi, q̄i, ai)}bi=1. 13: Forward Mb with negative samples, and obtain the predictions P (A|v−i , qi) and P (A|vi, q−i ). 14: Calculate the loss Lneg based on the predictions of the negative samples via Eq. (4). 15: Calculate the loss Lweight based on the predictions of both positive and negative samples by Eqs. (6) and (7). 16: end if 17: Update Mb by minimising the overall loss L (obtained via Eq. (8)). 18: end while
training process of the negative sample losses Lneg and Lweight.
The objective of the KL loss is to make the two distributions close. In our method, we seek to make the predictions of the original samples approach to the ensemble predictions. Thanks to the generated high quality positive samples, the KL loss can improve the robustness of the VQA models (Refer to in Line 1-2 of Table 7), to some extent. However, if we adopt the KL loss in the whole training process, the VQA models will fit the data distributions excessively, and thus may hinder the training process of the negative sample losses. The experimental results in Table 7 also confirm it. For example, our DDG with the training strategy that introduces KL loss in the overall training process still performs better than that training using only (Ldis and Lens), but performs worse than that training the model using KL loss for 12 epochs.
Comparison with the state-of-the-art methods regarding of the number of training samples. As shown in Table 8, the VQA-CP v2 dataset comprises 438k training samples, while KDDAug, SimpleAug, and our DDG generate an additional
4088k, 3081k, and 1752k augmented training samples, respectively. Despite using fewer augmented samples than KDDAug and SimpleAug, our DDG outperforms these methods by around 8% and 1%, respectively, which demonstrates the effectiveness of our DDG.

heading: 6.6 More Visualisation Results
text: Qualitative results. We provide more visualisation results in Figure. 4 to present the effectiveness of our DDG. From the results, our DDG localise the target objects more accurately than the UpDn (Anderson et al., 2018) model and SSL-VQA method, and thus makes a more correct prediction than the compared methods. These visualisation results demonstrate the effectiveness of our DDG. Visualisation of the generated samples. As shown in Section 3.2, we have generated both positive image and question samples. To evaluate the generated methods, we provide some visualisation results about the augmented questions and selected target objects in Table 9 and Figure. 5, respectively. From the results in Table 9, our augment questions have similar semantics to the original questions, which demonstrates our generated questions are reasonable as the positive samples. Moreover, although the baseline model UpDn (Anderson et al., 2018) trained on the VQA-CP (Agrawal et al., 2018) v2 dataset achieves poor performance on the test set, the UpDn model still can obtain good performance on the training set. Thus, we adopt the image attention weights of the pre-trained UpDn model to help find the objects that are relevant to the questions. As shown in Figure. 5, we show the objects with the top-3 attention weights of the pre-trained UpDn model in the images. From the results, the pre-trained UpDn model can localise the target objects referred to in the questions, which demonstrates that selecting the objects with top-k image attention weights of the pre-trained UpDn model is reasonable, and can exclude background information.
Q: What is the green stuff? Q: What does the man in the blue shirt have in his hand? Q: What is the man holding in his right hand?
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discuss the limitations of our work in Section Limitations
3 A2. Did you discuss any potential risks of your work? We discuss the potential risks of our work in Section Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? In Section Abstract and Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We conduct experiments on the VQA v2 and VQA-CP v2 datasets. In overall Sections
3 B1. Did you cite the creators of artifacts you used? In overall sections
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In Section Appendix
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Section Appendix
C 3 Did you run computational experiments? In Section Experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In Section Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? In Section Experiments and Section Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Section Experiments and Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Section Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Limitations
The paper focuses on the VQA task only, we will extend our method to other multimodal tasks in future works, e. g. , referring expression comprehension (REC). Moreover, although our DDG outperforms most of the state-of-the-art methods, the performance is still a long way from humans