SlowGuess/ACL_2020_OCR huggingface dataset:
V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, page 3.
Gabriel Grand and Yonatan Belinkov. 2019. Adversarial regularization for visual question answering: Strengths, shortcomings, and side effects. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 1-13, Minneapolis, Minnesota. Association for Computational Linguistics (ACL).
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700-6709.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8779-8788.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1988-1997. IEEE.
Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1983-1991. IEEE.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: Understanding data visualizations via question answering. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5648-5656.
Kushal Kafle, Robik Shrestha, and Christopher Kanan. 2019. Challenges and prospects in vision and language research. Frontiers in Artificial Intelligence.
Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017. Data augmentation for visual question answering. In Proceedings of the 10th International Conference on Natural Language Generation (INLG), pages 198-202.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems (NeurIPS), pages 1541-1551.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS).
8177
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. 2019. Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2591-2600.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. 2019. Answer them all! toward universal visual question answering models. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Bernard L Welch. 1938. The significance of the difference between two means when the population variances are unequal. Biometrika, 29(3/4):350-362.
Jialin Wu and Raymond Mooney. 2019. Self-critical reasoning for robust visual question answering. In Advances in Neural Information Processing Systems (NeurIPS), pages 8601-8611.
We compare four different variants of HINT and SCR to study the causes behind the improvements including the models that are fine-tuned on: 1) relevant regions (state-of-the-art methods) 2) irrelevant regions 3) fixed random regions and 4) variable random regions. For all variants, we fine-tune a pretrained UpDn, which was trained on either VQACpv2 or VQAv2 for 40 epochs with a learning rate of  $10^{-3}$ . When fine-tuning with HINT, SCR or our method, we also use the main binary cross entropy VQA loss, whose weight is set to 1. The batch size is set to 384 for all of the experiments.
HINT
Following (Selvaraju et al., 2019), we train HINT on the subset with human-based attention maps (Das et al., 2017), which are available for  $9\%$  of the VQA-CPv2 train and test sets. The same subset is used for VQAv2 too. The learning rate is set to  $2 \times 10^{-5}$  and the weight for the HINT loss is set to 2.
Since (Wu and Mooney, 2019) reported that human-based textual explanations (Huk Park et al., 2018) gave better results than human-based attention maps for SCR, we train all of the SCR variants on the subset containing textual explanation-based cues. SCR is trained in two phases. For the first phase, which strengthens the influential objects, we use a learning rate of  $5 \times 10^{-5}$ , loss weight of 3
Table A3: Results on VQA-CPv2 and VQAv2 datasets for the baseline UpDn, visual sensitivity enhancement methods (HINT and SCR) and our own regularization method, including the published (pub.) numbers.
VQA-CPv2	VQAv2
Baseline - Without visual grounding
UpDn	0.0110	0.0155
Grounding using human-based cues
HINT	0.1020	0.1350
SCR	0.0340	-0.0670
Grounding using irrelevant cues
HINT	-0.0048	-0.0200
SCR	0.0580	-0.0100
Grounding using fixed random cues
HINT	0.0510	0.0620
SCR	-0.0250	-0.0350
Grounding using variable random cues
HINT	0.0570	0.0623
SCR	-0.0380	0.0246
Regularization by zeroing out answers
Ours1% fixed	-0.1050	-0.1200
Ours100%	-0.0750	-0.0100
and train the model to a maximum of 12 epochs. Then, following (Wu and Mooney, 2019), for the second phase, we use the best performing model from the first phase to train the second phase, which criticizes incorrect dominant answers. For the second phase, we use a learning rate of  $10^{-4}$  and weight of 1000, which is applied alongside the loss term used in the first phase. The specified hyperparameters worked better for us than the values provided in the original paper.
Our Zero-Out Regularizer
Our regularization method, which is a binary cross entropy loss between the model predictions and a zero vector, does not use additional cues or sensitivities and yet achieves near state-of-the-art performance on VQA-CPv2. We set the learning rate to:  $\frac{2 \times 10^{-6}}{r}$ , where  $r$  is the ratio of the training instances used for fine-tuning. The weight for the loss is set to 2. We report the performance obtained at the  $8^{th}$  epoch.
A.2 Results
Correlation with Ground Truth Visual Cues
Following (Selvaraju et al., 2019), we report Spearman's rank correlation between network's sensitivity scores and human-based scores in Table A3. For HINT and our zero-out regularizer, we use human-based attention maps. For SCR, we use textual explanation-based scores. We find that HINT
8178
trained on human attention maps has the highest correlation coefficients for both datasets. However, compared to baseline, HINT variants trained on random visual cues also show improved correlations. For SCR, we obtain surprising results, with the model trained on irrelevant cues obtaining higher correlation than that trained on relevant visual cues. As expected, applying our regularizer does not improve rank correlation. Since HINT trained on relevant cues obtains the highest correlation values, it does indicate improvement in visual grounding. However, as we have seen, the improvements in performance cannot necessarily be attributed to better overlap with ground truth localizations.
A Note on Qualitative Examples
Presentation of qualitative examples in visual grounding models for VQA suffers from confirmation bias i.e., while it is possible to find qualitative samples that look at relevant regions to answer questions properly, it is also possible to find samples that produce correct answers without looking at relevant regions. We present examples for such cases in Fig. A3. We next present a quantitative assessment of visual grounding, which does not suffer from the confirmation bias.
Quantitative Assessment of Grounding
In order to truly assess if existing methods are using relevant regions to produce correct answers, we use our proposed metric: Correctly Predicted but Improperly Grounded (CPIG). If the CPIG values are large, then it implies that large portion of correctly predicted samples were not properly grounded. Fig. A4 shows  $\%$  CPIG for different variants of HINT trained on human attention-based cues, whereas Fig. A5 shows the metric for different variants of SCR trained on textual explanation-based cues. We observe that HINT and SCR trained on relevant regions have the lowest  $\%$  CPIG values (70.24% and 80.22% respectively), indicating that they are better than other variants in finding relevant regions. However, only a small percentage of correctly predicted samples were properly grounded (29.76% and 19.78% for HINT and SCR respectively), even when trained on relevant cues.
Breakdown by Answer Types
Table A4 shows VQA accuracy for each answer type on VQACPv2's test set. HINT/SCR and our regularizer show large gains in 'Yes/No' questions.
Table A4: VQA accuracy per answer-type on VQACPv2 test set.
Over-all	Yes/No	Num	Other
Baseline - Without visual grounding
UpDn	40.1	41.1	12.0	47.2
Grounding using human-based cues
HINT	48.2	65.2	13.8	47.5
SCR	49.1	70.3	11.5	48.0
Grounding using irrelevant cues
HINT	48.0	67.2	13.5	47.1
SCR	49.2	73.4	11.5	46.4
Grounding using fixed random cues
HINT	48.1	66.9	13.8	46.9
SCR	49.1	74.7	12.2	45.1
Grounding using variable random cues
HINT	48.1	67.1	13.9	46.9
SCR	49.2	74.7	12.2	45.1
Regularization by zeroing out answers
Ours1% fixed	48.9	69.8	11.3	47.8
Ours100%	48.2	66.7	11.7	47.9
We hypothesize that the methods help forget linguistic priors, which improves test accuracy of such questions. In the train set of VQACPv2, the answer 'no' is more frequent than the answer 'yes', tempting the baseline model to answer 'yes/no' questions with 'no'. However, in the test set, answer 'yes' is more frequent. Regularization effects caused by HINT/SCR and our method cause the models to weaken this prior i.e., reduce the tendency to just predict 'no', which would increase accuracy at test because 'yes' is more frequent in the test set. Next, all of the methods perform poorly on 'Number (Num)' answer type, showing that methods find it difficult to answer questions that are most reliant on correct visual grounding such as: localizing and counting objects. Finally, we do not observe large improvements in 'Other' question type, most likely due to the large number of answers present under this answer type.
Accuracy versus Size of Train Set
We test our regularization method on random subsets of varying sizes. Fig. A6 shows the results when we apply our loss to  $1 - 100\%$  of the training instances. Clearly, the ability to regularize the model does not vary much with respect to the size of the train subset, with the best performance occurring when our loss is applied to  $1\%$  of the training instances. These results support our claims that it is possible to improve performance without actually performing visual grounding.
8179
Ground Truth Localization
HINT trained on relevant cues
HINT trained on irrelevant cues
HINT trained on random cues
Q: Is this food sweet? A: yes
Remarks: The most sensitive regions for irrelevant/random variants do not contain food, yet their answers are correct.
Q: What is the swimmer doing? A: surfing
Remarks: Models trained on irrelevant/random cues do not look at the swimmer at all, yet produce correct answer.
Q: Is the sport being played tennis or volleyball? A: tennis
Remarks: None of the variants look at relevant regions, and yet produce correct answer.
Figure A3: Visualizations of most sensitive visual regions used by different variants of HINT to make predictions. We pick samples where all variants produce correct response to the question. The first column shows ground truth regions and columns 2-4 show visualizations from HINT trained on relevant, irrelevant and fixed random regions respectively.
Q: Has the boy worn out his jeans? A: yes
Remarks: All of the variants look at both relevant and irrelevant regions to produce correct answer.
Figure A4:  $\%$  CPIG for baseline and different variants of HINT and our method, computed using ground truth relevant regions taken from human attention maps (lower is better).
Figure A5:  $\%$  CPIG for baseline and different variants of SCR and our method, computed using ground truth relevant regions taken from textual explanations (txt).
Figure A6: The regularization effect of our loss is invariant with respect to the dataset size.