Venue: NAACL 2021

# SOT-ing VQA Models : Contrastive Gradient Learning

for Improved Consistency

Sameer Dharur\({}^{1}\)   Purva Tendulkar\({}^{1\to 3}\)

Dhruv Batra\({}^{1,2}\)   Devi Parikh\({}^{1,2}\)    Ramprasaath R. Selvaraju\({}^{1\to 4}\)

###### Abstract

Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world - they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the reasoning question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub-questions higher than irrelevant questions for an <image, reasoning-question> pair. We show that SOrT improves model consistency by up to 6.5% points over existing approaches, while also improving visual grounding and robustness to rephrasings of questions.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3103–3111 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Current visual question answering (VQA) models struggle with consistency. They often correctly answer complex reasoning questions, i.e, those requiring common sense knowledge and logic on top of perceptual capabilities, but fail on associated low-level perception questions, i.e., those directly related to the visual content in the image. For e.g., in Fig 1, models answer the reasoning question "Was this taken in the daytime?" correctly, but fail on the associated perception question "Is the sky bright?" indicating that the models likely answered the reasoning question correctly for the wrong reason(s). In this work, we explore the usefulness of leveraging information about _sub-questions_, i.e., low-level perception questions relevant to a reasoning question, and _irrelevant questions_, i.e., any other questions about the image unrelated to the reasoning question, to improve consistency in VQA.

Selvaraju et al. (2020) have studied this problem and introduced the VQA-Introspect dataset that draws a distinction between higher-level reasoning questions and lower-level perception sub-questions. We augment this dataset with additional perception questions from the VQAv2 dataset such that each <image, reasoning question> pair contains a set of relevant perception questions, which we refer to as _sub-questions_ (e.g.,"Is the sky bright?" in Fig 1) and irrelevant perception questions, which we refer to as _irrelevant questions_ (e.g., "Is the train moving?" in Fig 1) throughout this paper.

We use Gradient-based Class Activation Mapping (Grad-CAM) vectors [8] - a faithful function of the model's parameters, question, answer and image - to propose an interpretability technique that determines the questions most strongly correlated with a reasoning question for a model. This is measured by ranking questions based on the cosine similarity of their Grad-CAM vectors with that of the reasoning question. We find that top-performing VQA models often rank _irrelevant questions_ higher than relevant questions.

Motivated by this, we introduce a new approach based on contrastive gradient learning to fine-tune a VQA model by enforcing relevant _sub-questions_ to be ranked higher than _irrelevant questions_ while answering a reasoning question.

This is achieved by forcing the cosine similarity of the reasoning question's Grad-CAM vector with that of a _sub-question_ to be higher than with that of an _irrelevant question_. We find that our approach improves the model's consistency, defined as the frequency with which the model correctly answers a _sub-question_ given that it correctly answers the reasoning question.

Additionally, we assess the effects of our ap

proach on visual grounding by comparing Grad-CAM heatmaps with human attention maps collected in the VQA-HAT dataset Das et al. (2016). We find that our approach of enforcing this language-based alignment through better ranking of _sub-questions_ also improves visual grounding. We also demonstrate that training VQA models by aligning Grad-CAM vectors helps in improving robustness to rephrasings of questions, as evaluated on the VQA-Rephrasings dataset Shah et al. (2019).

## 2 Related Work

**Visual Question Answering**. The VQA task Agrawal et al. (2015) requires answering a free-form natural language question about visual content in an image. Previous work has shown that models often do well on the task by exploiting language and dataset biases Agrawal et al. (2017); Zhang et al. (2015); Ramakrishnan et al. (2018); Guo et al. (2019); Manjunatha et al. (2018). In order to evaluate the consistency of models, Selvaraju et al. (2020) collected a new dataset, VQA-Introspect, with human explanations via _sub-questions_ and answers for reasoning questions in the VQA dataset.

**Model Interpretability**. While prior work has attempted to explain VQA decisions in the visual modality Selvaraju et al. (2019, 2019); Qiao et al. (2017); Liang et al. (2019), the multi-modal task of VQA has a language component which cannot always be explained visually, i.e., visual regions can be insufficient to express underlying concepts Goyal et al. (2016); Hu et al. (2017). Park et al. (2018) and Wu and Mooney (2019) generate textual justifications through datasets curated with human explanations. Our approach differs by using Grad-CAM vectors which are fully self-contained and faithful to the model, requiring no additional parameters or datasets to interpret its decisions. In recent work on Human-AI collaboration Bansal et al. (2019, 2021), a key finding is that optimizing solely for model accuracy does not always lead to better overall utility in real-world, high-stakes datasets where AI systems advise humans on making decisions. Instead, improvements on yardsticks related to the trustworthiness of predictions are important steps towards successfully deploying these algorithms. We believe that consistency, the core focus of our work, is an intrinsically important post-hoc explanatory metric and a proxy for common-sense reasoning which could lead to stronger collective performance in such collaborative settings.

**Aligning network importances**. Ross et al. (2017) introduced an approach to train models with input-gradient penalties that led to the generation of faithful explanations and improved generalizability on image classifiers. Selvaraju et al. (2019) introduced an approach to align visual explanations with regions deemed important by humans, thereby improving visual grounding in VQA models. In followup work, Selvaraju et al. (2020) introduced an approach to align attention maps for the reasoning question and associated perception _sub-questions_ from VQA-Introspect to improve language based grounding. In contrast to attention maps, our work encourages Grad-CAM vectors of a reasoning question to be closer to those of _sub-questions_ and farther away from those of _irrelevant questions_. Intuitively, this means that we are making the neurons used while answering a reasoning question to be similar to those used while answering a _sub-question_ and dissimilar to those used while answering an _irrelevant question_. Our experiments show that this alignment improves the model's consistency and visual grounding.

Figure 1: The approach for SOrT. The reasoning question _Was this taken in the daytime?_ has the sub-question _Is the sky bright?_ and an irrelevant question _Is the train moving?_ We tune the model using cross entropy losses and a contrastive gradient loss to align the reasoning question’s Grad-CAM vector with that of its sub-question, while also distancing it from that of an irrelevant question.

 

Approach

### Preliminaries

**Grad-CAM.** Grad-CAM, introduced by Selvaraju et al. (2019), is a technique to obtain visual explanations from any CNN-based deep neural network. In this work, we adopt Grad-CAM to compute the contribution of a neuron at the layer in a VQA model where the vision and language modalities are combined. This is computed by first taking the gradient of the predicted output class score with respect to the neuron activations in the layer. We then point-wise multiply this with the corresponding activations to obtain our Grad-CAM vector. Specifically, if \(y^{c}\) denotes the score of the ground-truth output class and \(A_{k}\) the activations of layer \(k\) of the model, the Grad-CAM vector \(G_{k}^{c}\) is computed as follows,

\[G_{k}^{c}=\frac{\partial y^{c}}{\partial A_{k}}*A_{k} \tag{1}\]

Unlike Grad-CAM visualizations, these vectors are not visually interpretable as they are not computed on the final convolutional layer of the CNN.

**Consistency in VQA models**. As defined in Selvaraju et al. (2020), the consistency of a VQA model refers to the proportion of _sub-questions_ answered correctly, given that their corresponding reasoning questions were answered correctly. If a model is inconsistent, it is likely relying on incorrect perceptual signals or biases in the dataset to answer questions. Models that are consistent and based on appropriate perceptual signals are more likely to be reliable, interpretable and trustworthy.

### Sub-question Oriented Tuning

The key idea behind Sub-question Oriented Tuning (SOrT) is to encourage the neurons most strongly relied on (as assessed by Grad-CAM vectors) while answering a reasoning question ("Was this taken in the daytime?" in Fig 1) to be similar to those used while answering relevant _sub-questions_ ("Is the sky bright?") and dissimilar to those used while answering _irrelevant questions_ ("Is the train moving?"). This enforces the model to use the same visual and linguistic concepts while making predictions on the reasoning question and the _sub-questions_. Our loss has the following two components.

**Contrastive Gradient Loss**. With the Grad-CAM vectors of the reasoning question \((G_{R})\), _sub-question_\((G_{S})\) and _irrelevant question_\((G_{I})\), we formalize our contrastive gradient loss \(\mathcal{L}_{\text{CG}}\) as,

\[\mathcal{L}_{\text{CG}}=\text{max}\left(0,\quad\underbrace{\overbrace{G_{R} \cdot G_{I}}^{\text{cosine-sim}(G_{R},~{}G_{I})}}_{\text{cosine-sim}(G_{R},~{}G _{S})}\right) \tag{2}\]

**Binary Cross Entropy Loss**. To retain performance of the model on the base task of answering questions correctly, we add a Binary Cross Entropy Loss term (\(\mathcal{L}_{\text{BCE}}\)) that penalizes incorrect answers.

**Total Loss**. Let \(\mathrm{o_{R}}\), \(\mathrm{gt_{R}}\), \(\mathrm{o_{S}}\), \(\mathrm{gt_{s}}\), \(\mathrm{ot_{l}}\) and \(\mathrm{gt_{l}}\) represent the predicted and ground-truth answers for the reasoning, _sub-questions_ and _irrelevant questions_ respectively, and \(\lambda_{1}\), \(\lambda_{2}\), \(\lambda_{3}\) be tunable hyperparameters. Our total loss \(\mathcal{L}_{\text{SOT}}\) is,

\[\begin{split}\mathcal{L}_{\text{SOT}}&=\mathcal{ L}_{\text{CG}}+\lambda_{1}\mathcal{L}_{\text{BCE}}(\mathrm{o_{R}},\mathrm{ gt_{R}})\\ &+\lambda_{2}\mathcal{L}_{\text{BCE}}(\mathrm{o_{S}},\mathrm{ gt_{S}})+\lambda_{3}\mathcal{L}_{\text{BCE}}(\mathrm{o_{I}},\mathrm{gt_{l}})\end{split} \tag{3}\]

## 4 Experiments

**Dataset.** Our dataset pools VQA-Introspect and VQAv2 such that for every reasoning question in VQA-Introspect, we have a set of <_sub-question_, _answer_> pairs and a set of <_irrelevant question_, _answer_> pairs. The training/val splits contain 54,345/20,256 <image, reasoning question> pairs with an average of 2.58/2.81 _sub-questions_ and 7.63/5.80 _irrelevant questions_ for each pair.

**Baselines.** We compare SOrT against the following baselines: **1) Pythia**Jiang et al. (2018), and **2) SQuINT** in which Selvaraju et al. (2020) fine-tuned Pythia with an attention alignment loss to ensure that the model looks at the same regions when answering the reasoning and _sub-questions_.

### Metrics

**Ranking.**

**1) Mean Precision@1 (MP@1)**. Proportion of <image, reasoning question> pairs for which the highest ranked question is a _sub-question_. **2) Ranking Accuracy**. Proportion of <image, reasoning question> pairs whose _sub-questions_ are all ranked above their _irrelevant questions_. **3) Mean Reciprocal Rank (MRR)**. Average value of the highest reciprocal rank of a _sub-question_ among all <image, reasoning question> pairs. Higher is better. **4) Weighted Pairwise Rank (WPR) Loss**. For pairs of incorrectly ranked <sub, irrelevant> questions, this computes the differences of their 

similarity scores with the reasoning question. Averaged across all pairs, this computes the _extent_ by which rankings are incorrect. Lower is better.

**Model Performance.****1) Quadrant Analysis.****a**: \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf V}\) The pairs where reasoning and _sub-questions_ are both correctly answered. **b**: \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf X}\) The pairs where the reasoning question is correctly answered, while the _sub-question_ is incorrectly answered. **c**: \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf C}\) The pairs where the reasoning question is incorrectly answered, while the _sub-question_ is correctly answered. **d**: \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf X}\) The pairs where reasoning and _sub-questions_ are both incorrectly answered.

**2) Consistency.** The frequency with which a model correctly answers a _sub-question_ given that it correctly answers the reasoning question.

\[\text{Consistency}=\frac{|\hskip-5.690551pt\begin{array}{ccc}{\mathbb{R}} ^{\prime}&{\mathbb{S}}^{\prime}\\ {\mathbb{R}}^{\prime}&{\mathbb{S}}^{\prime}+{\mathbb{R}}^{\prime}&{\mathbb{S} }^{\prime}\end{array}$} \tag{4}\]

**3) Reasoning Accuracy**. The accuracy on the reasoning split of VQAv2 dataset, and

**4) Overall Accuracy**. Accuracy on the VQAv2 validation set.

### Results

We attempt to answer the following questions:

* the soft metric that computes the extent by which rankings are incorrect
- is a substantial 12% points over Pythia. This confirms that our approach helps better distinguish between the relevant and irrelevant perceptual concepts needed for answering a reasoning question.

**Does recognizing relevant sub-questions make models more consistent?** We find that the improved ranking of _sub-questions_ through SOrT improves consistency by 6.5% points over Pythia, 1.47% points over SQuINT and 0.4% points over an approach that just uses _sub-questions_ while discarding _irrelevant questions_1. As seen in Table 1, the consistency gains are due to significant improvements in the \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf V}\) and \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf X}\) quadrants. As seen in Table 1, the consistency gains are due to significant improvements in the \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf V}\) and \({}^{\mbox{\bf R}}\!\times\!\mbox{\bf S}\!\times\!\mbox{\bf X}\) quadrants. This comes at the expense of a drop in overall accuracy and reasoning accuracy by \(\sim\)1% point, likely due to the active disincentization of memorizing language priors and dataset biases through our contrastive gradient learning approach.

Footnote 1: These numbers are averaged values from 10-fold cross validation runs on the val split. The std dev values observed were 0.3 for Pythia and 0.41 for SQuINT and SOrT.

Gradient-based explanations have been shown to be more faithful to model decisions compared to attention maps Selvaraju et al. (2019). Our results confirm this by showing that aligning Grad-CAM vectors for reasoning and _sub-questions_ makes models more consistent compared to SQuINT, which aligns their attention maps. Fig 2 shows an example of improved consistency using SOrT. The Pythia model answers its _sub-question_ incorrectly. Our approach ranks the relevant _sub-question_ higher than the _irrelevant_ ones and answers it correctly - thus improving consistency.

Figure 2: An example of improvement in consistency between Pythia (top) and SOrT (below) brought about by better sub-question ranking.

 

- 80.73 (SOT) v/s 79.98 (SQuINT) v/s 79.51 (Pythia). Interestingly, we observe a minor improvement in accuracy as well
- 66.52 (SOT) v/s 65.45 (SQuINT) v/s 66.38 (Pythia). This confirms the effectiveness of our approach for both semantic and syntactic consistency.

**Does enforcing language-based alignment lead to better visual grounding?** To evaluate this, we compute visual grounding through Grad-CAM applied on the final convolutional layer. We then compute the correlation of Grad-CAM heatmaps with the validation split of the VQA-Human ATtenion (VQA-HAT) dataset Das et al. (2016), comprising 4,122 attention maps. This dataset contains human-annotated 'ground truth' attention maps which indicate the regions humans chose to look at while answering questions about images in the VQAv1 dataset. The proposed method to compare human and model-based attention maps in this work was to rank their pixels according to their spatial attention, and then compute the correlation between these two ranked lists.

We find that our approach gets a Spearman rank correlation of \(0.103\pm 0.008\), versus \(0.080\pm 0.009\) for Pythia and \(0.060\pm 0.008\) for SQuINT. These statistically significant improvements indicate that enforcing language-based alignment during training improves visual grounding on an unseen dataset. A qualitative example that demonstrates the superior visual grounding of SOiT compared to its baselines is shown in Fig 3. For the question _Is the baby using the computer?_ and its corresponding answer _Yes_, we see that the Grad-CAM heatmap generated by SOiT is closest to that of the human attention map. It is also the only heatmap in this example that actually points to the fingers of the child, which is the essential visual component for answering the question.

## 5 Discussion

In this work, we seek to improve consistency in VQA. We first develop language-based interpretability metrics to measure the relevance of a lower-level perception question while answering a higher-level reasoning question. Evaluating state-of-the-art VQA models on these metrics reveals that models often rank irrelevant questions higher than relevant ones. We present SOiT (Sub-question Oriented Tuning), a contrastive gradient learning based approach for teaching VQA models to distinguish between relevant and irrelevant perceptual concepts while answering a reasoning question. SOiT aligns Grad-CAM vectors of reasoning questions with those of _sub-questions_, while distancing them from those of _irrelevant questions_. We demonstrate SOiT's effectiveness on datasets that test for semantic as well as syntactic consistency without major changes to accuracy, while also improving visual grounding.

Figure 3: A qualitative example of the improvement in visual grounding by SOiT. For the <question, answer> pair of _<Is the baby using the computer?, Yes_>, we see the comparison of the Grad-CAM heatmaps generated by the 3 models and the human attention map. SOiT’s heatmap is most closely aligned with that of the human attention map.

 

Acknowledgements

The Georgia Tech effort was supported in part by NSF, AFRL, DARPA, ONR YIPs, ARO PECASE, Amazon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.

## 7 Ethical Considerations

The key ethical considerations for this work relate to fairness. Although not ubiquitous in application today, the progress of research in VQA necessitates work in the direction of transparency so as to build trust among users before these systems are widely deployed in the real world.

Prior work in this domain has revealed VQA models to exploit visual and language based priors in the datasets they are trained on Das et al. (2016); Agrawal et al. (2017); Zhang et al. (2015); Ramakrishnan et al. (2018); Guo et al. (2019); Manjunatha et al. (2018). Such models tend to compound the biases prevalent in these datasets, and could have detrimental effects on fairness. Our work could better explain these biases by identifying the most relevant perceptual concepts used by the model while answering reasoning questions. In addition, by improving consistency and visual grounding in VQA systems, our work contributes to mitigating some of these biases.

## References

* Agrawal et al. (2017) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Anirudhha Kembhavi. 2017. Don't just assume; look and answer: Overcoming priors for visual question answering.
* Agrawal et al. (2015) Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. 2015. Vqa: Visual question answering.
* Bansal et al. (2021) Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S. Weld. 2021. Is the most accurate ai the best teammate? optimizing ai for teamwork.
* Bansal et al. (2019) Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-ai team performance. _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_, 7(1):2-11.
* Das et al. (2016) Abihshek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2016. Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
* Goyal et al. (2016) Yash Goyal, Akrit Mohapatra, Devi Parikh, and Dhruv Batra. 2016. Towards transparent ai systems: Interpreting visual question answering models.
* Guo et al. (2019) Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yibing Liu, Yinglong Wang, and Mohan Kankanhalili. 2019. Quantifying and alleviating the language prior problem in visual question answering.
* Hu et al. (2017) Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In _The IEEE International Conference on Computer Vision (ICCV)_.
* Jiang et al. (2018) Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018. Pythia v0.1: the winning entry to the vqa challenge 2018.
* Liang et al. (2019) Junwei Liang, Lu Jiang, Liangliang Cao, Yannis Kalantidis, Li-Jia Li, and Alexander G. Hauptmann. 2019. Focal visual-text attention for memex question answering. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 41(8):1893-1908.
* Manjunatha et al. (2018) Varun Manjunatha, Nirat Saini, and Larry S. Davis. 2018. Explicit bias discovery in visual question answering models.
* Park et al. (2018) Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence.
* Qiao et al. (2017) Tingting Qiao, Jianfeng Dong, and Duanqing Xu. 2017. Exploring human-like attention supervision in visual question answering.
* Ramakrishnan et al. (2018) Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization.
* Ross et al. (2017) Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations.
* Selvaraju et al. (2019a) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2019a. Grad-cam: Visual explanations from deep networks via gradient-based localization. _International Journal of Computer Vision_, 128(2):336-359.
* Selvaraju et al. (2019b) Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalin Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. 2019b. Taking a hint: Leveraging explanations to make vision and language models more grounded 

Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Ribeiro, Besmira Nushi, and Ece Kamar. 2020. Squinting at vqa models: Incorporating vqa models with sub-questions.
* Shah et al. (2019) Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering.
* Wu and Mooney (2019)Jialin Wu and Raymond J. Mooney. 2019. Faithful multimodal explanation for visual question answering. In _Proceedings of the Second BlackboxNLP Workshop at ACL_, pages 103-112, Florence, Italy.
* Zhang et al. (2015) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2015. Yin and yang: Balancing and answering binary visual questions.

 

Appendix

### Experimental Details

#### a.1.1 Algorithms

We use the Pythia model for our experiments. Specifically, for our SOrT approach, we compute Grad-CAM vectors for the reasoning question, _subquestions_ and _irrelevant questions_ on each image at the layer where the vision and language modalities are combined. We then use customized losses described in Section 3.2 of the paper. The mathematical computation of consistency is described in Section 4.1, while the ranking metrics are described below.

* "Among all the related questions for a pair, was atleast the highest ranked related question a relevant perception sub-question?"

This is illustrated in an example below across two sets.

Example Query 1 : "What is the capital of the USA?"

Predicted Ranking 1 : ["New York", "Washington DC", "San Francisco"]

Ground Truth Answers 1 : [0, 1, 0]

Example Query 2 : "Where is the Golden Gate Bridge located?"

Predicted Ranking 2 : ["San Francisco", "Atlanta", "Los Angeles"]

Ground Truth Answers 2 : [1, 0, 0]

Across these two examples, the Mean Precision@1 value would be \(\frac{1}{2}\) since only one of them has its highest ranked item as a correct answer.

**Ranking Accuracy**. This computes the proportion of pairs in which all the relevant perception sub-questions are ranked higher than the irrelevant questions. This would represent a perfect ranking capability of the model.

Example Query 1 : "Cities in Asia."

Predicted Ranking 1 : ["Stockholm", "Beijing", "New Delhi"]

Ground Truth Answers 1 : [0, 1, 1]

Example Query 2 : "Planets in the solar system."

Predicted Ranking 2 : ["Neptune", "Jupiter", "Phobos"]

Ground Truth Answers 2 : [1, 1, 0]

The combined Ranking Accuracy across these two examples would be \(\frac{1}{2}\) since all the correct answers are ranked higher than the incorrect ones only in the second set.

**Mean Reciprocal Rate (MRR)**. This is a variation of MP@1 which captures the highest rank of a relevant item in a list. In our case, the reciprocal rank is concerned with the highest rank of a relevant perception sub-question among all the ranked related questions for a pair. The reciprocal of this highest relevant rank is averaged across the entire dataset. This is represented in the example below.

Example Query 1 : "What is the capital of the USA?"

Predicted Ranking 1 : ["New York", "Washington DC", "San Francisco"]

Ground Truth Answers 1 : [0, 1, 0]

Example Query 2 : "Where is the Golden Gate Bridge located?"

Predicted Ranking 2 : ["San Francisco", "Atlanta", "Los Angeles"]

Ground Truth Answers 2 : [1, 0, 0]

Across these two examples, the MRR could be calculated as follows :

\[RR_{1}=\frac{1}{2} \tag{5}\] \[RR_{2}=\frac{1}{1}=1.\] \[MRR=\frac{1}{2}*(RR_{1}+RR_{2})=\frac{1}{2}*\frac{3}{2}=\frac{3}{4}\]

**Weight Pairwise Rank (WPR) Loss**. All the above metrics only account for the ranking of the candidate questions for a given pair, but do not consider the _extent_ by which these questions differ in their rankings. Concretely, to have a comprehensive understanding of the relevance of each question, we need to account for the magnitude of their similarity scores with the reasoning question in our overall metric.

For a pair, we create a parallel list of ranked questions in which all the relevant perception sub-questions are higher than the other questions, while retaining the same similarity scores as computed for the originally ranked list. We then compare these two lists pair-wise, i.e, in each index, and sum up the differences of the similarity scores if the rankings are different between the two lists.

 

This provides us a way to measure not just the deviation from the desired order of rankings but also the magnitude of the differences in similarity scores which are responsible for the erroneous rankings. If \(S\) could be represented as the set of size \(n\) containing all such incorrectly ranked pairs \((r.r^{\prime})\) with scores (\(\alpha\), \(\alpha^{\prime}\)), we could compute the WPR loss for each set as the sum of the absolute values of the differences between each \(\alpha\) and \(\alpha^{\prime}\).

\[\text{WPR}=\frac{\sum_{(r,r^{\prime})\in S}|\alpha-\alpha^{\prime}|}{n} \tag{6}\]

This is then averaged across the entire dataset. We illustrate an example for a single set. Query : "Which of these is a national capital?" Predicted Ranking With Scores : [("Mexico City", 0.9), ("Miami", 0.8), ("Copenhagen", 0.7)] Ground Truth Answers : [1, 0, 1] Parallel List w.r.t Ground Truth Answers : [("Mexico City", 0.9), ("Copenhagen", 0.7), ("Miami", 0.8)]

\[\text{WPR}=\frac{1}{2}*(0.1+0.1)=0.1 \tag{7}\]

#### a.1.2 Source Code

Our source code is accessible here : [https://github.com/sameerdharur/sorting-vqa](https://github.com/sameerdharur/sorting-vqa).

#### a.1.3 Computing Infrastructure

The computing infrastructure used for training and running each model described in the paper was 1 NVIDIA TITAN Xp GPU.

#### a.1.4 Runtime

The average training time for the model on each combination of hyperparameters was roughly 12 hours.

#### a.1.5 Parameters

The details on the parameters of the model can be found in the Pythia paper referenced in the main section.

#### a.1.6 Validation Performance

The results of the validation performance on each of the different metrics have been included in Section 4.2 of the main section. The metrics have been explained above, with the source code linked above.

#### a.1.7 Hyperparameter Search

For the best performing models, the values of \(\lambda\) described in the losses of Section 3.2 are \(\lambda_{1}=\lambda_{2}=2.27,\lambda_{3}=0.0003\). These values were selected based on the differing scales of the loss components and chosen from running hyperparameter sweeps. The rest of the hyperparameters were unchanged from those reported for the best performing Pythia model.

A total of \(294\) hyperparameter trial runs were conducted with \(\lambda_{1}\) and \(\lambda_{2}\) ranging from 0.025 to 25, and \(\lambda_{3}\) ranging from \(1e\)-\(5\) to \(100\).

These values were picked by a combination of uniform sampling and random tuning, and were optimized on a combination of consistency and accuracy. As mentioned in Section 4.2, the expected validation results fall within the statistical range of the results defined by a standard deviation of 0.3 and 0.41 for Pythia and SQuINT/SOrT.

#### a.1.8 Datasets

As detailed in Section 4, our dataset is a combination of the VQA-Introspect and VQAv2 datasets. In total, our train/val splits contain 54,345/20,256 <image, reasoning question> pairs with an average of 2.58/2.81 sub-questions and 7.63/5.80 irrelevant questions for each pair respectively. Subsets of this data have been attached in a zip file with this submission to serve as representative examples. The details on the VQA-Introspect and VQAv2 datasets, which are publicly available, can be found in the corresponding papers cited in the main section. The VQA-HAT dataset used in the visual grounding analysis and the VQA-Rephrasings dataset used for evaluating syntactic consistency are also publicly accessible