SlowGuess/NAACL_2021_OCR huggingface dataset:
Abhishek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2016. Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yash Goyal, Akrit Mohapatra, Devi Parikh, and Dhruv Batra. 2016. Towards transparent ai systems: Interpreting visual question answering models.
Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yibing Liu, Yinglong Wang, and Mohan Kankanhalli. 2019. Quantifying and alleviating the language prior problem in visual question answering.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In *The IEEE International Conference on Computer Vision (ICCV)*.
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018. Pythia v0.1: the winning entry to the vqa challenge 2018.
Junwei Liang, Lu Jiang, Liangliang Cao, Yannis Kalantidis, Li-Jia Li, and Alexander G. Hauptmann. 2019. Focal visual-text attention for memex question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1893-1908.
Varun Manjunatha, Nirat Saini, and Larry S. Davis. 2018. Explicit bias discovery in visual question answering models.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence.
Tingting Qiao, Jianfeng Dong, and Duanqing Xu. 2017. Exploring human-like attention supervision in visual question answering.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2019a. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128(2):336-359.
Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. 2019b. Taking a hint: Leveraging explanations to make vision and language models more grounded.
Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Ribeiro, Besmira Nushi, and Ece Kamar. 2020. Squinting at vqa models: Interrogating vqa models with sub-questions.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. 2019. Cycle-consistency for robust visual question answering.
Jialin Wu and Raymond J. Mooney. 2019. Faithful multimodal explanation for visual question answering. In Proceedings of the Second BlackboxNLP Workshop at ACL, pages 103-112, Florence, Italy.
Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2015. Yin and yang: Balancing and answering binary visual questions.
A.1.1 Algorithms
We use the Pythia model for our experiments. Specifically, for our SOrT approach, we compute Grad-CAM vectors for the reasoning question, subquestions and irrelevant questions on each image at the layer where the vision and language modalities are combined. We then use customized losses described in Section 3.2 of the paper. The mathematical computation of consistency is described in Section 4.1, while the ranking metrics are described below.
Mean Precision@1 (MP@1). For a given ordering of related questions (based on 1 of the 3 similarity scores), we compute the fraction of pairs in which a relevant perception sub-question was ranked the highest, i.e., had the highest similarity score with that of the reasoning question. This is equivalent to setting a bare-bones expectation of reasoning ability for the model - "Among all the related questions for a pair, was at least the highest ranked related question a relevant perception sub-question?"
This is illustrated in an example below across two sets.
Example Query 1: "What is the capital of the USA?"
Predicted Ranking 1: ["New York", "Washington DC", "San Francisco"]
Ground Truth Answers 1: [0, 1, 0]
Example Query 2: "Where is the Golden Gate Bridge located?"
Predicted Ranking 2: ["San Francisco", "Atlanta", "Los Angeles"]
Ground Truth Answers 2: [1, 0, 0]
Across these two examples, the Mean Precision@1 value would be  $\frac{1}{2}$  since only one of them has its highest ranked item as a correct answer.
Ranking Accuracy. This computes the proportion of pairs in which all the relevant perception sub-questions are ranked higher than the irrelevant questions. This would represent a perfect ranking capability of the model.
Example Query 1 : "Cities in Asia."
Predicted Ranking 1 : ["Stockholm", "Beijing", "New Delhi"]
Ground Truth Answers 1: [0, 1, 1]
Example Query 2: "Planets in the solar system."
Predicted Ranking 2 : ["Neptune", "Jupiter", "Phobos"]
Ground Truth Answers 2:[1,1,0]
The combined Ranking Accuracy across these two examples would be  $\frac{1}{2}$  since all the correct answers are ranked higher than the incorrect ones only in the second set.
Mean Reciprocal Rate (MRR). This is a variation of MP@1 which captures the highest rank of a relevant item in a list. In our case, the reciprocal rank is concerned with the highest rank of a relevant perception sub-question among all the ranked related questions for a pair. The reciprocal of this highest relevant rank is averaged across the entire dataset. This is represented in the example below.
Predicted Ranking 2 : ["San Francisco", "Atlanta", "Los Angeles"]
Across these two examples, the MRR could be calculated as follows:
$$
R R _ {1} = \frac {1}{2}
$$
$$
R R _ {2} = \frac {1}{1} = 1. \tag {5}
$$
$$
M R R = \frac {1}{2} * \left(R R _ {1} + R R _ {2}\right) = \frac {1}{2} * \frac {3}{2} = \frac {3}{4}
$$
Weight Pairwise Rank (WPR) Loss. All the above metrics only account for the ranking of the candidate questions for a given pair, but do not consider the extent by which these questions differ in their rankings. Concretely, to have a comprehensive understanding of the relevance of each question, we need to account for the magnitude of their similarity scores with the reasoning question in our overall metric.
For a pair, we create a parallel list of ranked questions in which all the relevant perception subquestions are higher than the other questions, while retaining the same similarity scores as computed for the originally ranked list. We then compare these two lists pair-wise, i.e., in each index, and sum up the differences of the similarity scores if the rankings are different between the two lists.
This provides us a way to measure not just the deviation from the desired order of rankings but also the magnitude of the differences in similarity scores which are responsible for the erroneous rankings. If  $S$  could be represented as the set of size  $n$  containing all such incorrectly ranked pairs  $(r,r')$  with scores  $(\alpha, \alpha')$ , we could compute the WPR loss for each set as the sum of the absolute values of the differences between each  $\alpha$  and  $\alpha'$ .
$$
\mathrm {W P R} = \frac {\sum_ {(r , r ^ {\prime}) \in S} | \alpha - \alpha^ {\prime} |}{n} \tag {6}
$$
This is then averaged across the entire dataset.
We illustrate an example for a single set.
Query : "Which of these is a national capital?"
Predicted Ranking With Scores : [(“Mexico City”, 0.9), (“Miami”, 0.8), (“Copenhagen”, 0.7)]
Ground Truth Answers : [1, 0, 1]
Parallel List w.r.t Ground Truth Answers : [(“Mexico City”, 0.9), (“Copenhagen”, 0.7), (“Miami”, 0.8)]
$$
\mathrm {W P R} = \frac {1}{2} * (0. 1 + 0. 1) = 0. 1 \tag {7}
$$
A.1.2 Source Code
Our source code is accessible here : https://github.com/sameerdharur-sorting-vqa.
A.1.3 Computing Infrastructure
The computing infrastructure used for training and running each model described in the paper was 1 NVIDIA Titan Xp GPU.
A.1.4 Runtime
The average training time for the model on each combination of hyperparameters was roughly 12 hours.
A.1.5 Parameters
The details on the parameters of the model can be found in the Pythia paper referenced in the main section.
A.1.6 Validation Performance
The results of the validation performance on each of the different metrics have been included in Section 4.2 of the main section. The metrics have been explained above, with the source code linked above.
A.1.7 Hyperparameter Search
For the best performing models, the values of  $\lambda$  described in the losses of Section 3.2 are  $\lambda_{1} = \lambda_{2} = 2.27, \lambda_{3} = 0.0003$ . These values were selected based on the differing scales of the loss components and chosen from running hyperparameter sweeps. The rest of the hyperparameters were unchanged from those reported for the best performing Pythia model.
A total of 294 hyperparameter trial runs were conducted with  $\lambda_{1}$  and  $\lambda_{2}$  ranging from 0.025 to 25, and  $\lambda_{3}$  ranging from  $1e - 5$  to 100.
These values were picked by a combination of uniform sampling and random tuning, and were optimized on a combination of consistency and accuracy. As mentioned in Section 4.2, the expected validation results fall within the statistical range of the results defined by a standard deviation of 0.3 and 0.41 for Pythia and SQuINT/SOrT.
A.1.8 Datasets
As detailed in Section 4, our dataset is a combination of the VQA-Introspect and VQAv2 datasets. In total, our train/val splits contain 54,345/20,256 <image, reasoning question> pairs with an average of 2.58/2.81 sub-questions and 7.63/5.80 irrelevant questions for each pair respectively. Subsets of this data have been attached in a zip file with this submission to serve as representative examples. The details on the VQA-Introspect and VQAv2 datasets, which are publicly available, can be found in the corresponding papers cited in the main section. The VQA-HAT dataset used in the visual grounding analysis and the VQA-Rephrasings dataset used for evaluating syntactic consistency are also publicly accessible.