VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives
Authors: Zhuofan Ying
Peter Hase
Mohit Bansal
Venue: NeurIPS 2022 Accept (2022)
Keywords: Visual question answering
feature importance supervision
right-for-the-right reason metrics
explanation faithfulness

Abstract: Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions. However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Question Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI. In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility). Our best performing method, Visual Feature Importance Supervision (VISFIS), outperforms strong baselines on benchmark VQA datasets in terms of both in-distribution and out-of-distribution accuracy. While past work suggests that the mechanism for improved accuracy is through improved explanation plausibility, we show that this relationship depends crucially on explanation faithfulness (whether explanations truly represent the model’s internal reasoning). Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful. Lastly, we show that, surprisingly, RRR metrics are not predictive of out-of-distribution model accuracy when controlling for a model’s in-distribution accuracy, which calls into question the value of these metrics for evaluating model reasoning.

VISFIS: Visual Feature Importance Supervision with Right-fortthe-Rght-RRaa Objectives
Zhuofan Ying,' * Peter Hase," and Mohit Bansal Department of Computer Science University of North Carolina Chapel Hill {zfying, peter, mbansal) unc edu
Abstract
Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions. However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Ques- tion Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI. In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well performance on several Right- figtt-for-the--rr Right-Rease (RRR) metrics by optimizing for four key model objectives: (I) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance predictions to changes in unimportant features (Invariance); and (4) alignment between model FI expla- nations and human explanations (Plausibility). Our best performing method, Visual Feature Importance Supervision (VIsFIS), outperforms strong baselines benchmark VQA datasets terms of both in-distribution and out-of-distribution accuracy. While past work suggests that the mechanism for improved accuracy is through improved explanation plausibility, we show that this relationship depends crucially on explanation faithfulness (whether explanations truly represent the model's internal reasoning). Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but faithful. Lastly, we show that, surprisingly. RRR metrics are not predictive of out-of distribution model accuracy when controlling for a model's 's in-distribution accuracy, which calls into question the value of these metrics for evaluating model reasoning."
Introduction
Many past works aim to teach models to ignore spurious features by making use of additional information about which features in input important [31, 46, 61]. For example, individual words can be annotated as (un)important in NLP tasks [46, 66]. or regions of image pixels can be highlighted by humans as extra supervision for vision tasks [11, 51]. In this broad class of feature importance (FI) supervision methods, human annotations of important features are typically provided for individual datapoints, and methods often use data augmentation or gradient supervision to encourage models to rely only on important features when making predictions. Such approaches have seen performance improvements in image classification [50, 7], text classification [35, 66, 58], and multimodal visual question answering (VQA) tasks [47, 36).
Equal contribution.
2 All supporting code for experiments in this paper available at http:///ghuubb.bommrri/n//ieeees
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Visual Feature Importance Supervision
VISFIS -Cask CSaft Luse Liar.II CAlign
All Features
Emportant Features
Unimporiant Features
Important-Random
Model
Model
Differendablc Model Dutput
Input
Desired Output
Accurate
Accurate Output
Uncertain
Same Output Important Festures
Desired Output
Objertive
Task
Cauft
Lune
Objoctive
Aligm
Metric
Test BCC
RRR-Suff
BRR-Une
RRR-inv
Metric
Plausbilit
Right for the Right Reasons
Explanation Metric
We depict five core desiderata for VQA models with associated metries and objectives. We seek models that (1) are accurate given the full visual input, (2) are accurate given only important features, (3) are appropriately uncertain given only unimportant features, (4) are invariant to unimpor- tant features, and (5) yield FI estimates (explanations) that align with human FI. VISFIS combines the five objectives. Note objectives #2-#5 require additional Fl annotations. Darkened image regions correspond to bounding box representations that have been Replaced (see Sec. 3).
Figure I:
One of the primary motivations behind these approaches is improve task accuracy by making models "Right for the Right Reasons" [46]. While existing Fl supervision methods for VQA models can improve accuracy [47, 64), recent work has cast doubt on whether human supervision is the source of these improvements. Specifically, Shrestha et al. [48] show that VQA improvements persist even when random image F/ annotations are used supervision, suggesting that existing approaches may not extract meaningful signal from the human annotations. Motivated by this shortcoming, we explore several aspects of the supervision problem in the context of VQA tasks:
Improving VQA accuracy with FI supervision via four key model objectives (Sec. 7.1). Past VQA methods focus on data augmentation techniques [36] or ways to directly supervise model feature importance [47, 64]. We make use of four key objectives (represented in Fig. 1): (1) a Sufficiency objective encouraging the model to predict the correct label given only important input features [7]; (2)
an Uncertainty objective encouraging max-entropy outputs when given only unimportant features; (3) an Invariance objective encouraging model outputs to be invariant to changes in unimportant features [46]; and (4) an Align objective that penalizes the model when its FI estimates differ from human Fl annotations [47]. Our best performing method, termed Visual Feature Importance Supervision (VIsFIS), combines these strategies strategies in a novel manner, improving both in-distribution and dittr--distribution accuracy. Following guidelines from Shrestha al. [48], show that this improvement does not occur with random supervision, meaning VISFIS learns from human supervision itself. Lastly, after analyzing how explanation plausibility and faithfulness [23] relate to accuracy at the datapoint level, we suggest that Fl supervision improves prediction accuracy by improving the plausibility of faithful Fl explanations, rather than plausibility alone past work has suggested [47, 7].
Evaluating models on new Right for the Right Reason (RRR) metrics (Sec. 7.4). Beyond measuring model accuracy, past works evaluate models few Right for the Right Reason metrics in order to understand whether model reasoning is correct [47, 48, 65, 7, 19, 16]. Correct reasoning is valuable because suggests that models will generalize test data that might not be able to verify their performance on, which can occur e.g. when there are exponentially many cases we wish to test or when such data is prohibitively expensive to collect. We propose broad set of RRR metries for model evaluation, with similar motivation to our key model objectives above (see Fig. 1). Specifically, in addition to measuring existing metrics for (I) in-distribution accuracy, (2) out-of-distribution accuracy [1, 10, 6, 8, 7], and (3) model accuracy on sufficient feature subsets (RRR-SIA (uff) [7], we also evaluate (4) model uncertainty given uninformative inputs (RRR-Une Unc), and (5) model invariance to changes in unimportant features InvR-Inv). These metrics help verify that models can: arrive at correct answers relying only on features that are actually important (metric #3), are invariant to the addition or removal of unimportant features that should not affect the label (metric #4), and are appropriately uncertain about the model class when the input contains no meaningful evidence for any class (metric #5).

---

Predicting model generalization to OOD data with RRR metrics (Sec. 7.5). The practical value of the above RRR metries can be considered in terms of their ability inform us about model performance on data that we are not able to test on. We simulate this situation by evaluating models on in-distribution (ID) data and predicting whether the models will generalize to OOD data based on their accuracy and RRR metrics for ID data. Surprisingly, we find that both existing RRR metrics and our new ones do not better predict OOD accuracy than ID accuracy does on its own. This finding suggests that these metries may not be good evaluation of model reasoning, and that there is no good replacement yet for evaluating model accuracy on OOD data in addition to ID data.
In summary of our contributions, we show that:
1. Fl supervision can improve both ID and OOD model accuracy on several benchmark VQA datasets. In particular, VISFIS improves over unsupervised baselines and the previous state-of-th ofttto-f--he-rr arrt on CLEVR-XAI by up to 4.7 points on OOD data.
2. Explanation plausibility correlates with model accuracy only when explanations are also faithful, which sheds light on the mechanism by which FI supervision improves model accuracy.
3. FI supervision improves model performance on several RRR metrics, including new invariance and uncertainty metrics.
4. RRR metrics do not correlate better with OOD accuracy than ID model accuracy does on its own. Consequently, RRR metrics may not be as valuable as previously thought.
2 Related Work
Supervising FI explanations. Past works primarily supervise gradient-basee [46] or attention-bassd model explanations [66, 52, 17, 91. For example, Ross et al. [46] enforce an &g norm on the gradient of the loss w.r.t. the model input for features marked as unimportant by human Fl explanation. This method appears in several later works [50, 18, 51J. In a VQA setting, Selvaraju et al. [47] and Wu and Mooney [64] align the entire input gradient with human FI. In addition using input gradients (termed Vanilla Gradient), we consider the Expected Gradients method [15], a computationally efficient implementation of Integrated Gradients [54], Omission or perturbation- -based approaches have seen more limited use. Kennedy et al. [28] regularize omission-based FI toward 0 for group identifiers in hate speech detection. In addition to simple leave-one-out [33] and keep-one-in methods, we propose a differentiable version of the popular linear method, SHAP [37]. For survey of methods we refer readers to Friedrich al. [16]. Following the analysis of Shrestha et al. [48], we use a random supervision baseline to show that VISFIS succeeds by virtue of additional supervision and not simply via model regularization.
Supervised data augmentation. This line of work uses human explanations guide data augmenta- tion, sometimes in a human-in in loop manner. For instance, Teney et al. [58] present LIME explana- tions to people and solicit feedback that is converted into counterexamples for model training. Liang et al. [34] use expert natural language counterfactual explanations to manufacture new labeled inputs.
We build on prior work for our data augmentation objectives. Similar to Chang et al. [7] and concurrent work from Singla et al. [51], our Sufficiency objective encourages models to be accurate given inputs with sufficient features selected according to human FI. Our Uncertainty objective reduces model confidence when no important features are provided, while the most related objectives from Chang al. [7] and Liu et al. [36] encourage different answer rather than an uncertain output. We are not aware of objectives encouraging invariance to changes in unimportant features as our invariance objective does. Other concurrent work encourages models to always predict the true label even when unimportant features are swapped with other unimportant features from the data distribution [19].
Right for the Right Reason metrics. Only a few past works explicitly evaluate RRR metrics in addition to test set accuracy. Our metrics include the existing RRR-Suff metric [7] and our RRR-Inv and RRR- Une metrics. While explanation plausibility is regularly proposed as an RRR metric [47, 48, 65, 43, 16], we show that the relationship between accuracy and plausibility is controlled by explanation faithfulness, meaning that plausibility on its own should not be an RRR metric. Recently, Joshi et al. [27] propose number of distinct distribution shifts in text classification for evaluating FI supervision techniques according to model OOD accuracy. We use a "changing prior" distribution shift standardly used in past work for VQA [11, 59], and moreover, the focus of our work on novel RRR objectives, metrics, and analysis of how supervision improves models.
3

---

New machine learning metrics are often justified from first principles or on the basis of strong correlation with some other good metric, like a human rating [41]. Here, we assess RRR metrics on the basis of their correlation with OOD accuracy, while controlling for model ID accuracy. This means that we measure the correlation between ID and OOD accuracy like in past studies [57, 39, 38, 60), but we also consider RRR metrics additional explanatory variables for OOD generalization.
3 Terminology and Notation
FI Explanations. We distinguish between a human explanation and a model explanation é that obtained algorithmically explain how model arrived at its prediction for some datapoint. In this paper, human explanations are real-valued annotations for input features (which are bounding box representations for each of our datasets). The score for each bounding box an indication of its importance to determining the datapoint label, which could roughly be thought of as an answer to the question, "why did data point receive label y" [40]. For several objectives and metrics, we binarize the explanations, selecting a threshold based on the data distribution (see Appendix E).
Replace Functions. Both generating and evaluating model explanations often require "hiding" input features from a model. In practice, we must replace features with some baseline value [53]. One simple and common way replace features is use an all-zeros feature [32, 54, 4]. We compare among several Replace functions to find the best function for learning from FI supervision (see Appendix B). We ultimately select the All-Negative-One function, which replaces a bounding box feature vector with the all negative ones vector, {-1} We use to denote version of the input where features where is are replaced via our Replace function.
Model Notation, We parametrize a distribution p(ul [x) = fa( for classification purposes. Here, y arg max,, Jo (x), is the model prediction, fo(z) y is the predicted probability, and y is the space of eligible answers, which is a large set that shared across all questions in our VQA tasks.
4 Methods for Learning from Human Feature Importance Supervision
We now describe how to optimize for several key model desiderata using human FI supervision (repre- sented in Fig. 1). In Sec. 4.5, we give the overall objective for Visual Feature Importance Supervision (VISFIS), which combines the objective terms below in order to improve model generalization.
4.1 Accuracy Given Sufficient Information
Goal. Like Chang al. [7], we hope for image processing models to make accurate predictions given subsets of image features that are sufficient for arriving at the correct label, since this suggests that a model recognizes that the important features in fact important.
Method. Access to human explanations should enable us automatically construct sufficient inputs with some amount of unimportant information removed [7]. In particular, for every input we can Replace unimportant features, while
create another datapoint by using the human explanation keeping the same label. The corresponding objective is given as:
Lsur (0, y, e) = CrossEnt(fozz.)
(1)
This objective differs from previous instantiations [7] by virtue of the Replace function used (see Sec. 4.5). We compare Lau against an unsupervised baseline using random feature subsets. That is, random distribution D. specifies how likely it is that we Replace a feature:
Random 0, D.) E.,D, (fossEnt((fo((....))) v)
which, when training via SGD, is estimated using sample per datapoint per batch.
4.2 Uncertainty Given Only Unimportant Information
Goal. We would prefer for model to give uncertain outputs for inputs with no important features, meaning the model should give near-uniform distribution over classes. Since there is no evidence for any given class, the model should not be confident the input belongs in particular class.

---

Method. With this goal in mind, Chang et al. [7] train models to give less confident outputs for images with important foreground features removed. More specifically, they encourage the model to predict any class except the image's s true class. In contrast, we penalize KI. divergence between the model output distribution and a uniform distribution,
Luee (0, æ, e) = (Unnf(()")))
(3)
where (1)) is the uniform distribution and u = 1 - indicates unimportant features.
4.3 Invariance to Unimportant Information
Goal. We would like models to be invariant to changes an image's unimportant features. This prop- erty is desirable because means that model correctly treats unimportant features as unimportant. Method. We first describe a simple data augmentation approach, then describe an FI supervision approach similar to past work [50, 7].
In data augmentation approach, train model to produce the same outputs for two inputs that share the same important information while differing in the unimportant information they contain. Specifically, we
obtain an input with both important and unimportant features, denoted by Teuu = Replace(a, U U.) Then, we penalize the KL divergence between the output distributions on the inputs T. and Fecu- The resulting objective is then:
Liw (0, (0,2, 2. e, D.)) = EEWDD (L(So(Fe)) fo
where D, is distribution over binary vectors. LImv-DA is estimated with one sample like LSuff-
In an Fl supervision approach, we first obtain model explanations at the datapoint level as Explain fo, x. 9). where Explain is differentiable explanation method (possible methods described below). Then we seek to directly penalize models for treating unimportant features as important. To do so, we encourage FI scores for unimportant features to be 0:
LI (ê..) = ||
(5)
where e. is the subset of the explanation over features marked unimportant by e. Past work uses an le distance for this objective [50, 7]. while we use an l, penalty after normalizing explanations to unit length, since explanations from different FI methods have different scales.
We consider few options for differentiable explanation methods. Past work has primarily used gradient-based [46, 47, 50, 18, 64, 7] and attention-basel explanations [66, 52, 17, 9J. We adopt existing gradient/attention methods and provide new differentiable perturbation-base methods.
1. Gradient- baaded explanations. One can optimize objectives involving gradient-based explanations w.r.t. by computing second derivatives like Vo Vzfo(()) in framework like PyTorch [42]. We use a simple Vanilla Gradient method and the Expected Gradients method (see Appendix A).
Attention-baser explanations. We supervise bounding box attention weights in the UpDn model [2], but early experiments suggest this is not an effective method and we do not explore it further.
3. Perturbation-bee based explanations. Perturbation-base methods like SHAP [37] are very popular ex- planation methods, but have seen only limited use for FI supervision [28]. We consider leave-one- out method (LOO), a keep-one-in method (KOI), Average Effect, and SHAP (see Appendix A).
In Appendix D, we discuss limits on the compute budget used for each method during model training.
4.4 Aligned Model and Human Feature Importance
Goal. Alignment between human and model explanations has frequently been proposed as a goal for models [47, 48, 65, 16]. In general, past works assume that model explanations are faithful, meaning they accurately communicate a model's 's internal reasoning [23]. This assumption is necessary for the alignment between model and human explanations, termed plausibility by Jacovi and Goldberg [23], to be evidence that model reasoning is similar to human reasoning. Of course, model explanations are not guaranteed be faithful. To the extent that they are faithful, however, encouraging explanation plausibility during training may help align model reasoning with human reasoning.
5

---

Method. We first obtain model explanations at the datapoint level as é Explain(fe, 2. () (see Sec. 4.3 above). Then, we can measure the difference between and the human explanation e using an 1p distance, cosine similarity, or a differentiable ranking function [47, 64]. We use cosine similarity since model explanations and human explanations do not share the same scale. Our objective thus:
Lalign (0, F. e, = cos-sim
66)
4.5 Overall Objective for VISFIS: Visual Feature Importance Supervision
We combine the supervised objective terms from above achieve the corresponding model desiderata simultaneously. Following objective tuning experiments showing that Inv-FI outperforms Inv-DA (see Appendix Table 11), we use Inv-FI rather than Inv-DA, and therefore our final VISFIS objective is:
Crsk + XzLsuut + Yu + X4 CAlign As CIm-FF
where Trusk is a standard supervised cross-entropy loss. Besides tuning the values of X one at a time, we also tune the Replace function and Fl method used in this objective, making sure to use comparable compute budgets across FI methods. Replace functions we consider are listed in Appendix B (results in Table 6), and FI methods in Appendix A (results in Tables 9 and 10). Following tuning, we find that it is preferable negative ones vector, [-1)d, and surprisingly, we find that Vanilla Gradient is the best performing Fl method, surpassing all perturbation-basee methods as well the Expected Gradients method.
Replace bounding box representations with the
Metrics
Next, we describe the RRR and explanation metrics for each of our model desiderata outlined above. We also measure model ID and OOD accuracy (distribution shifts described Sec. 6). As with the model objectives, we the All-Negative-Oner Replace function as needed.
RRR-Suffiency. We measure model accuracy inputs containing only features selected as important by their respective human explanation (similar to [7]). The remaining features are Replaced.
CER-Uncertainty. We propose to measure how uncertain the model prediction is given only unim- portant information. Specifically, we report the average model predicted probability when provide only unimportant features to the model (according to the human explanation), so lower is better.
Invarinarie We propose to calculate the agreement between model predictions with the input Ie and three Eeuu that each include random number of unimportant features. The final metric is averaged over three random u for each test point, then over all test points.
Explanation Plausibility. Our explanation plausibility metric is the Spearman's 's rank correlation between the human and model feature importance vectors, similar to past work [11, 47J. We use continuous FI estimates in order to calculate the rank correlation. rank correlation is preferrable here because human and model FI explanations do not lie in the same space.
Explanation Faithfulness. We use two standard faithfulness metrics [13]. Sufficiency measures whether keeping important features (according to model explanation é) leads the model to retain its confidence in its original prediction: Suff fo, x,
fx() fol fz())) Comprehensiveness measures whether removing important features from an input leads to a decline in model confidence, Comp( fo, x, é) (=)) fo(x)) g where 2 Replace (ar, 1 é) is the complement of features in zz- We average these score over several sparsity levels of è keeping or removing the top 10%, 25%, 50% of features [13]. Note compute these metrics using the best available explanation method per dataset, as measured by explanation faithfulness (comparison in Appendix G).
6 Experiment Setup
Datasets. We perform experiments on three benchmark datasets: CLEVR-XAI [5], GQA [21], and VQA-HAT [1I]. CLEVR-XAI is an algorithmically generated dataset based on CLEVR [26] and provides ground truth visual segmentation masks for each question. CLEVR-XAI is limited in visual varieties and vocabularies, but it offers Fl supervision in a controlled, low-noise setting. GQA

---

contains compositional reasoning questions over naturalistic images. GQA also includes the program for generating the questions and the ground- -ruth scene graph from the Visual Genome dataset [30]. This allows us to obtain bounding boxes of relevant objects identified through the question program, which we use as Fl supervision. VQA-HAT is based on VQAv1 [3], including naturalistic images and questions with mouse tracking de-blurring used to collect image Fl annotations from humans. For VQA, we report model performance on the more challenging other type questions as recommended by Teney et al. [59].
Distribution Shifts. We create both ID and OOD test sets for each dataset, so we always have four data splits: Train, Dev, Test-ID, and Test-OOD (split sizes shown in Table 1). To obtain OOD data, we use distribution shifts sim- ilar to those in VQA-CP, which are intended to vary the linguistic bias between ID and OOD splits [1]. We apply the same procedure for distribution shift on all three datasets for comparability.
Table 1: Dataset split sizes.
Dataset
Train
Dev
Test-ID ID
Test-OOD
CLEVR-XAI
83k
14k
21k
22k
GQA-101k
101k
20k
20k
20k
VQA-HAT
36k
6k
9k
9k
In detail, create groups of questions according to the first few words in each question (indicating the type of question), and allocate groups unevenly into ID and OOD sets, randomly assigning 80% of each group one and 20% to the other. The ID set is split into Train, Dev, and Test-ID. Model se- lection is done according to Dev set performance. We further downsample the very large GQA dataset from about 100k for training and 20k for other splits. See Appendix Fig. 5 for training size ablation analysis. We note that we avoid several pitfalls in evaluating VQA models against distribution shifts, as outlined by Teney et al. [59]. See Appendix Table for sensitivity analysis with randomly resplit data.
Human Feature Importance. For all datasets, we obtain human Fl scores at the bounding box (BB) level for detected BBs from the Faster-RCNN detector [45]. Following Selvaraju et al. [47], for both VQA-HAT and CLEVR-XAI we obtain importance scores from pixel-level annotations as st === - /LE + )²), where st is the score for the hthh detected BB and E and Eo are the average pixel-Ievel importance score inside and outside the BB, respectively. VQA-HAT has real-valued pixel-Ievel scores, while for CLEVR-XAL, we set the pixel-level score to for pixels within the segmentation mask and 0 elsewhere. For GQA, since we have BB level annotations, we calculate the importance score based on the intersection over union (IoU) between ground-truth important BBs and detected BBs: 2² = maxteg loU B3 BBx)) where B² is the BB of the 1tth detected object and
By is the BB of the 1-th ground-truth object. With importance scores for each BB, we manually set a threshold for determining important and unimportant objects (0.85, 0.55, and 0.3 for CLEVR-XAI, VQA-HAT, and GQA respectively). See Appendix E for sensitivity analysis for this threshold.
Models. We run experiments with UpDn [2] and LXMERT [56]. Both models rely on bounding box representations generated by a pretrained Faster R-CNN model [45] (further details in Appendix F).
Hypothesis Testing. We conduct hypothesis tests via a bootstrap resampling model seeds and datapoints 10k times [14]. We obtain 95% confidence intervals in the same way.
7 Experiment Results
7.1 Can FI Supervision Improve Model Accuracy for VQA?
Design. Using UpDn on our three datasets, we compare VISFIS with previous state-of-the-art Fl supervision methods for VQA tasks [47, 64] as well as for image classification [50, 7]. We give results for LXMERT only on CLEVR-XAI, since GQA and VQA are a part of the pretraining data for LXMERT [56]. Note we test on the more challenging other type questions only for VQA, following Teney et al. [59]. Selvaraju et al. [47] use Lalign with a ranking loss to align Vanilla Gradient explanations and human FI supervision. Wu and Mooney [64] propose a relaxed version of the ranking loss that binarizes important and unimportant features according to human Fl supervision and encourages higher model Fl for important objects than unimportant ones. The other methods we consider all use an LFI-Mv objective with an l2 penalty on Vanilla Gradient explanations. On top of this, Chang et al. [7] add an Lsaf objective with a Shuffle Replace function that randomly permutes features rather than replacing them, to preserve the marginal data distribution, and Singla et al. [51] add an Lsuft objective with a Gaussian noise Replace function. Our unsupervised baselines are models trained with only label supervision using Randd-annoo-
7

---

Table 2: Test accuracy across FI supervision methods and datasets with an UpDn model. We bold/underline numbers higher than the best unsupervised baseline a significance threshold of p < .55 (and bold is better than underline at < .05).
CLEVR-XAI
GQA-101k
VQA-HAT
Method
ID
OOD
ID
OOD
OOD
Baseline
71.37
36.80+1.00
51.8240.22
31.8040.64
37.5
28.7641.10
Suff-Random
71.72+0.57
39.0840.80
51.5940.65
31.65=0.82
37.99=1.35
19.34
Selvaraju et al. [47]
71.32+0.58
37.96+1.00
51.3840.62
31.99=0.77
36.93+1.37
27.3
Wu and Mooney [64]
71.48+0.64
37.31+0.86
51.54+0.67
31.61
37.24=1.32
28.26+1.15
Simpson al. [50]
71.22+0.60
37.54+0.71
52.10=0.68
31.99=0.77
37.66*1.30
28.73+1.44
Chang et al. [7]
70.77=0.56
35.38+0.92
50.29:0.65
30.40=0.86
32.55=1.41
17.9
Singla et al. [51]
71.5420.58
38.25+1.39
52.42+0.66
32.58=0.59
38.28=1.37
29.25+2.12 12
VISFIS
72.82=0.56
43.78+1.11
54.81+0.61
34.88=0.80
38.75+1.35
31.21+1.28
w/ Rand. Supervis.
69.70+0.67
33.28
49.82+0.62
29.93=0.89
37.16+1.30
27.51
Results. We show results for UpDn in Table 2 and for LXMERT in Table 3. First, we find that Fl supervision can meaningfully improve model accuracy. With UpDn on CLEVR-XAL. VISFIS improves ID accuracy by 1.1 points (#0.5; p=le-4) and OOD accuracy by 4.7 points (+1.4; p<le-4) over the strongest bascline without supervision, Suff-Random (see Appendix Fig. 9 for breakdown in improvements by CLEVR question type). Trends are similarly positive on the other datasets and with LXMERT, where VISFIS outperforms the baseline by 0.48 points (+0.35; p<.01) on ID data and 1.07 points
Table 3: LXMERT + CLEVR-XAL results.
Method
ID Acc
OOD Acc
Baseline
86.91+0.43
73.76+0.72
Suff-Random
86,53=0.47
73.52+1.07
Selvaraju al. [47]
87.03=0.43
74.56+0.58
Wu and Mooney [64]
86.73+0.46
73.9740.75
Simpson et al. [50]
86.22+0.57
73.1241.28
Chang et al. [7]
85.05
67.2742.27
Singla et al. [51] VISFIS
87.08=0.46
74.10+0.75
87.39+0.45
74.83+0.70
w/ Rand. Supervis.
85.84+0.83
71.11+1.4
(p<le-4) on OOD data (for results on all VQA question types, see Appendix Table 15). These improvements do not persist when using random explanations (last row), meaning they are caused by the human supervision. Finally, we observe that VISFIS is the best overall method across datasets and architectures, as other methods typically do not improve accuracy over an unsupervised baseline. The next best method is that of Singla et al. [51], which improves over an unsupervised baseline only for the GQA dataset with UpDn, but VISFIS still outperforms Singla et al. [51] there by 2.39 (+0.55; p<le-4) points on ID data and 2.31 points (10.66; p<le-4) on OOD data.
7.2 How Does FI Supervision Improve Accuracy?
Design. Past work hypothesizes that FI supervi- sion improves accuracy by aligning model and human FI [47, 7]. Surprisingly, we find that the relationship between model test accuracy and av- erage explanation plausibility fairly weak (lin- ear correlation on UpDn+CLEVR XAI models p=0.1 14). Here, we argue that plausible explanations alone are not evidence of correct model reasoning, but plausible and faithful ex- planations are. Using 4 million ID/OOD test predictions from UpDn+CLEVR-XA. models, visualize trendlines from logistic regressions predicting model accuracy based on plausibility and faithfulness at the datapoint level, grouped into Worst, Middle, and Best faithfulness cate- gories based metrics (see Appendix E).
Dutapoint-level Accuracy vs. Plausibility
1.00
0.75
Accuracy
0.50
Explanation
Fantrfulness
0.25
Best
Middlle
Worst
0.00
Explanation Pluusibility
Figure 2: Datapoint-level accuracy by explanation plausibility, averaged across CLEVR-XAI models. Trendlines are logistic regressions. When explana- tions are more faithful, their alignment with human explanations better correlates with model accuracy.
Results. Fig. 2 shows that an explanation for datapoint becomes more plausible, the model more likely to correctly predict that point's label, but only when the explanation also faithful. Indeed, maximally plausible and faithful explanation has about 90% chance of being correct,
8

---

Vow Image
Humann Explanatio
Model Explar udion
Ippui mmage
Human Explanation
Model Exp dtion
Faithfulness
Queedoon
Presddir:
day gight
billy low Fathruieene
Fow many zubras see?
Pauntillyy sgh
Question:
Predicion: gellow Bil
Question
Prodicion: hokids dog X Flasttilly Nigh
What color are Iis cat' ayea?
iow
What type thi??
Fallkiined::
Plausibility
Figure 3: Qualitative visualization of the relationship between accuracy, plausibility, and faithfulness represented in Fig. 2. In a low faithfulness setting (in terms of explanation sufficiency), a data point with an implausible explanation can still have correct prediction (bottom left), while a data point with highly plausible explanation can have incorrect prediction (bottom right). Among higher faithfulness points (top row), data with more plausible explanations tend to be correctly predicted.
while a minimally plausible but highly faithful explanation has closer to 12.5% of being correct. For unfaithful explanations, plausibility has essentially no relationship with accuracy. Though these trends are not necessarily causal, they are consistent with the view that when model predictions are correct, it is because their true reasoning (as revealed by faithful explanations) aligns with human reasoning. Fig. 3 qualitatively illustrates this relationship among faithfulness, plausibility, and accuracy with example data points and model predictions. We emphasize that while past work has treated plausibility as an RRR metric [47, 48, 65, 43, 16), the results here demonstrate that plausibility alone cannot be measure of model correctness.
7.3 Which FI Supervision Objectives Improve Accuracy?
Design. We ablate across objective terms from Sec. 4 for UpDn on CLEVR-XAL XAIL The weight for each objective term is tuned while using only that objective, then kept fixed when objectives are combined (further details in Appendix F). We consider another kind of ablation experiment where use random supervision for one objective at a time VISFIS, with results in Appendix Table 12.
Results, In Table 4, we find that each individual objective is valuable on its own, and they do well when combined. Relative the Baseline OOD accuracy, Huff-Human adds 4.1 points, Une adds 1.54 points, Inv-FI adds 2.08 points, and Align adds 4.81 points. When the four objectives are combined in VISFIS, the improvement rises to 6.98 points.
7.4 Can FI Supervision Make Models Right for the Right Reasons?
Design. We report RRR metrics as well as explanation plausibility the UpDn+CLEVR-XC models from objective ablation above.
Table 4: Objective ablation for UpDn CLEVR-XAI
Acc
RRR Metries
Expl.
Objective
ID
OOD
Suff'
Inv
Une
Plaus.
Results. In Table 4, we find that FI supervision generally improves RRR metric scores. Compared to the Baseline, VISFIS achieves 27.8 points better Sufficiency. 13.8 points better Invariance, and 11.5 points better Uncertainty. The best unsupervised method closes the gap slightly on RRR-Suff and RRR-Inv. Specifically, Suff-Randon is only 3.97 points worse than Suff-Human on RRR-Suff, and only 0.53 points worse than Inv-DA on RRR-Inv. It suggests that FI supervision noticeably improves RRR metries, meaning model behavior better fulfills the theoretical desiderata from Sec.
Baseline
71.37
36.80
48.82
77.89
55.17
28.82
Inv-DA
71.17
35.91
72.53
93.12
76.29
14.33
Inv-FI
71.41
38.88
45.31
76.34
71.41
28.60
Unc
71.30
38.34
10.75
86.58
4.16
8.56
Align
72.04
41.61
61.19
79.51
64.22
37.20
Suff- Random
71.73
39.08
73.59
92.59
60.93
17.32
Suff- Human
71.87
40.91
76.94
90.82
81.42
16.27
+ Align
72.42
41.63
78.55
89.69
80.02
35.73
Une
72.33
41.54
77.83
89.70
41.68
23.41
VIsFIS
72.82
43.78
76.65
91.72
43.64
22.67

---

7.5 Do RRR Metrics Predict OOD Generalization?
Design. We measure the correlation between RRR metries (caleulated with ID data) and OOD accuracy across a large set of models. We report results here for all UpDn models on CLEVR-XAI, with similar results for GQA/VQA and LXMERT given in Appendix Table 16. We consider few possible model metrics, including several composite metrics that combine model-level metrics. To optimally weight the individual metries, fit statistical models to the data that predict OOD accuracy given the available metrics. Since this risks overfitting the composite metrics to the data we have, we perform metrics as training data and 10 for testing each time. The final metrics we consider are: (1) ID accuracy on its own as a baseline, (2) RRR metries on their own, (3) ID accuracy plus average model confidence, (4) ID accuracy plus explanation metrics (for plausibility and faithfulness), (5) ID accuracy plus RRR metries, and (6) All Metrics, which uses all available metrics.
cross-validation resampling model-level statistics 10k times, using 90 models'
Results. In Table 5, we show the average correla- tions between each metric and model OOD accuracy achieved in our cross-validation. Interestingly, we find that RRR metrics do not achieve a better cor- relation with OOD accuracy than ID accuracy does on its own. ID accuracy alone has a correlation of 0.863 with OOD accuracy, while using ID Acc + RRR metrics achieves a correlation of 0.852. In fact, the only additional metric that improves one's ' abil- ity to predict OOD accuracy is the average model confidence on ID data (more confident models have slightly better OOD accuracy), though this does not hold for LXMERT models (see Appendix Table 16). All Metrics
Table 5:
Correlations between metrics and OOD accuracy, with 95% confidence intervals.
(Merric, OOD Acc)
Metric
Train
Test
RRR-Suff
0.278
0.333 #.0021
RRR-Inv
0.149
0.157 =.058
RRR-Unc
0.029
0.021 #.003
ID Acc
0,870
0.863 =.0018
Model Conf.
0.909
0.907 +.0010
Expl. Metries
0.875
0.861 #.0066
RRR-all
0.874
0.852 +.033
0.925
0.891 #.0014
These results cast doubt on the value of RRR met-
rics. If ID accuracy on its own is better predictor of OOD accuracy than RRR metrics, then RRR metrics may not be a better measure of the quality of model reasoning than ID accuracy is. We believe the RRR metrics considered in this paper are still theoretically justified as model desiderata, but we cannot recommend them as measures of model generalization to OOD data.
8 Discussion & Conclusion
Limitations. Though we evaluate with two standard model architectures and three datasets, our conclusions may be limited to settings using Faster-RCNN [45] bounding box representations the feature space rather than pixel space. Additionally, though we follow existing guidelines with our distribution shifts [11, 59], we do not measure model generalization across all typical kinds of shifts [44]. Lastly, we note that FI supervision methods are limited by the need for additional annotations.
Ethics. We hope that our findings regarding model accuracy and explanation plausibility/faaituulees will help dispel the notion that models reason like humans (or are more grounded) simply because model explanations look similar to human explanations, which can cause unwarranted trust in ML models [24]. We do not foresee specific ethical risks arising from this work that do not already apply to the general use of machine learning for visual question answering tasks, such the potential deployment of ML models in settings where they may harm people [62, 55].
Conclusions. In this paper, we show that (1) FI supervision can improve VQA model accuracy via our VISFIS method, (2) accuracy improvements appear to stem from improving explanation plausibility specifically for faithfully explained data, (3) Fl supervision can improve RRR metric performance, and (4) RRR metrics not actually correlate well with OOD accuracy.
Acknowledgements
We thank Jaemin Cho for helpful discussion of this work, as well as Derek Tam, Xiang Zhou, and Archiki Prasad for useful feedback. This work was supported by ARO Award W91 INF21 10220, DARPA
(MCS) Grant N660001-19--40331 NSF-AI Engage Institute DRL- 21 2213, and a Google PhD Fellowship. The views contained this article are those of the authors and not of the funding agency.
10

---

References
[1] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer. Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971 1971-4980, 2018. 2,7
[2] Peter Anderson, Xiaodong Hc, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018. URL https: /ppp/accecccsssseeeee thecvf contenntttt....mmm _ttp___vp____0_01 Up_ Top- Down_cVPPp @oom__cvR_2018_pappee pdf. 5, 7,
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
[4] Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojeiech Samek. Evaluating recur- rent neural network explanations. In Proceedings of the 2019 ACL Workshop BlackboxNLPP Analyzing and Interpreting Neural Networks for NLP,, pages 113-126, Florence, Italy, Au- gust 2019, Association for Computational Linguistics, doi: 10. 10.18653/1/119-4813.. URL //ttp::/www.... aclweb. org/ anthology/WW9949330
https::
[5] Leila Arras, Ahmed Osman, and Wojciech Samek. Clevr-xai: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion, 81:14-40, 2022. ISSN 1566-2535. doi: https://doi.org/100 (0f...nfffu..221..111
URL https:
//github. howtthub.ommaaaeeeeee http:///eeddaddddeaaaneeeeeeeeee xai. 6, 19
[6] Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019. 2
[7] Chun-Hao Chang, George Alexandru Adam, and Anna Goldenberg. Towards robust classifica- tion model by counterfactual and invariant data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15212-15221, 2021. URL https://arxiv.oom org/ ppfppff216666 /216/2206..0 01127. pdf. 1, 3, 4, 5, 6, 7, 8, 26
[8] Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10800- thecvf com/ content cowteet_cvpR___00000 wwt_cc/R_2220peeeeeeeeeeee 10809, 2020. URL Sttr_aaaleessSSttteiiiiieeee Roo_uuuttttttoooooottttoooeeeee Visual _isual_uuesiiainnn..... Chen Chen_Counterfactual Saapalls__aala_aaeaeel Answering_CVPR__OOO pwppr.ppppppp... pdf. 2
[9] George Chrysostomou and Nikolaos Aletras. Enjoy the salience: Towards better transformer- based faithful explanations with word salience, arXiv preprint arXiv:2108.13759 2021. URL
attps://arrte org/ pdf /2108. 13759. 13759.pdf..... 5.5
[10] Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:/909.0.3.. 2019.
[11] Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, 2017. URL https: / http////oppp/nn////ee ece. edu/" http://aahhhddz//eee
1, 3, 6, 10, 19
[12] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019. 21
[13] Jay Yo Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. Eraser: benchmark evaluate rationalized nlp models. ACI 2020, volume abs/1911 1.03429, 2020. URL https: //
pdf. :/ //axxvv.. fexxiv.org// org/pdf / /1111 1009911038000 03429, .df. 6
11

---

[14] Bradley Efron and Robert J Tibshirani. An Introduction to the Bootstrap. CRC press, 1994. 7
[15] Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 3(7):620-633 2021. URL petps://arxxor wtg/pdf/1900
10670. 10f...pdf... 3, 17
[16] Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kensting. A typology to explore and guide explanatory interactive machine learning. arXiv preprint arXiv:2203.03663 2022. URL https: /wwxiv..iivvvv/////// org/ http/fff22200 03668. pdf. 2,
[17] Yuyang Gao, Tong Sun, Liang Zhao, and Sungsoo Hong. Aligning eyes between humans and deep neural network through interactive attention alignment. arXiv preprint arXiv:2202.02838, 2022. URL https: /axvv...vvvvv org/pot /2202.222222222 222.2222222....... pdf. 3,5
[18] Reza Ghaeini, Xiaoli Z. Fern, Hamed Shahbazi, and Prasad Tadepalli. Saliency learning: Teaching the model where to pay attention. In NAACL, 2019. URL https:// arxiv. org/ pdf/ /102..92..00 08649.6648888 pdf. 3,5
[19] Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille. Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5078-5088, June 2022. 2,
[20] Peter Hase, Harry Xie, and Mohit Bansal. The and search methods for feature importance explanations. NeurIPS, 34, 2021. URL https: // //arxiv.. prrg//ff//// /2166. 00786 pdf. 23
problem in explainability
[21] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019. URL https: //cs. stanford. eew.anforddddd//.aoeoeeooee edu/ add/peoplee /doraradd////a//e gqa/ index. html. 19, 20
[22] Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning interpretability by saliency guided training. NeurlPS, 34, 2021. URL petttps://arxtte org/ pdf /2111 14338. pdf. 25
[23] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? In ACL 2020, 2020. URL https : wwwww/wwwwwwww.../////w//o// aclweb. org/
anthology/ /aalooyyy22200 2020. acl main. www.mmm.....mmmm..mmmmm..m...... pdf. 2,5
[24] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence:
Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 635, 2021. URL org/ httg//d/////// htfp//f//000/ 20t.//211.07877. .ddf. 10
https://arti wwtp/i/v..g///oooooooeooe
(25] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3543-3556, 2019. 17
[26] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017. 6
[27] Brihi Joshi, Aaron Chan, Ziyi Liu, Shaoliang Nie, Maziar Sanjabi, Hamed Firooz, and Xiang Ren. Er-test: Evaluating explanation regularization methods for nlp models. arXiv preprint arX1v:2205.12542, 2022. 3
[28] Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. Contextualizing hate speech classifiers with post-hoe explanation. ACI,, 2020. URL. attps://aclamttholggs 22g/ acl- 483, .df. 3,
12

---

[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 21
[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017. URL https:/ poapps///axiiv.a/ arg//bs//a 1602. 07332. 7
[31] Todd Kulesza, Margaret Burnett, Weng- Keen Wong, and Simone Stumpf. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings the 20th in- ternational conference on intelligent user interfaces, pages 126-137, 2015. URL https://dd //dl. acm org/ /tt////pfffft/ 1111:////0111111117//0825533 http:///w..............aooooooo.
wcC.://c..H.......aarrroooaooooa NyyPfapp7746225viVycc..
[32] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprini arXiv:1612.08220,, 2016. URL https://arxxv.orrg//fff /1611 08220. 18.. 4, 17
[33] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding Neural Networks through Represen- tation Frasure. arXiv: [cs], 2016. URL http:///axxvvv.///aaae/ org/abs/ aaaaaba///// /a1a//11.... 1112108220........ arXiv: 1612.08220.
[34] Weixin Liang, James Zou, and Zhou Yu. ALICE: active learning with contrastive natural language explanations. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, EMNLP, pages -390- Association for Computational Linguistics, 2020. URL https: //www. org/ anthology/2029 emnlp-maine 355/.
[35] Frederick Liu and Besim Avci. Incorporating priors with feature attribution on text classification. org/p19- 1ff...df..
In ACL, 2019. URL
[36] Yibing Liu, Yangyang Guo, Jianhua Yin, Xuemeng Song, Weifeng Liu, Liqiang Nie, and Min Zhang. Answer questions with right image regions: A visual attention regularization approach. ACM Transactions Multimedia Computing, Communications, and Applications (TOMM), 18 (4):1-18, 2022. URL attps://arxxs
org/pdf/ /2102. 01916. pdf. 1,
[37] Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process- ing Systems 30, pages 4765-4774. 2017. URL http: /ttp:///pperre..e nips cc/paper/ 7062-a a- wit---anniieee--aa aaproocchpppacccc--e to- seweerppreinnmmmdell.
in wwdem- www.diptiiiiin.e.i pdf. 5, 17, 18
(38) Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint arXiv:2012, 15483, 2020. URL https://arxiii .rr.iiiorrrr/ooo wwt.odt////oooooooooooooooo... /20p//ff22222222 15483. pdf.
[39] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribu- tion shift on question answering models. In International Conference on Machine Learning. pages 6905-66 6916. PMLR, 2020. URL btteps://arxivy org/pdf /2004. 14444. .df.. 4
[40] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell., 267:1 38, 2019. doi: 10. 10.1016/j.aa a016/.. 2018.07.007. URL attpps://oot httpg//..1116///// artint. 2018. 07.007...
[41] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL 2002, 2002. URL https: wwwwb..wwwwwwb/bbb////.........b org/ anthology/PO2- 1040. .df..
aclweb.
[42] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NewrIPS, 2017. URL https::// htttt::///ppeerviee net/ppdfidd=BJJnmmeeee 5, 21
13

---

[43] Grusha Prasad, Yixin Nie, Mohit Bansal, Robin Jia, Douwe Kiela, and Adina Williams, To what extent do human explanations of model behavior align with actual model behavior? In Proceedings of the Fourth BlackboxNI.P Workshop Analyzing and Interpreting Neural Networks for NLP, 2021. URL pdf. 3,9
org/2021. blackboxnlp-1..1.
[44] Joaquin Quinonero-Candll,, Masashi Sugiyama, Anton Schwaighofer, and Neil Lawrence. Dataset shift in machine learning. Mit Press, 2008. URL http:://tt //www acad. bg/ ebook/nl The MIT Press. Dataset. Shift. in.iiii.aaaiiii Machine Learning. Feb. 2009. eoo..eBooooooeee pdf. 10
[45] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015. 7, 10
[46] Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi Velez. Right for the right reasons: Training differentiable models by constraining their explanations. In ISCAI, pages 2662-2670, 2017. doi: 10. 00.2496//jccai.201777// 0371 pdf. 1, 2, 5
URL hww.hhttp////eet //wp::///wwwwww/ jecai. org/ prggppooceedings/21111
[47] Ramprasaath Ramasamy Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry P. Heck, Dhruv Batra, and Devi Parikh. Taking HINT: leveraging explanations make vision and language models more grounded. In ICCV, pages 2591-2600. IEEF, 2019. doi: 10.1 10.1109//CC/22019.00 http:/pppps//lllaeeeeee Takiaaaaaaa___ainng.a.a a HNNT_Leveraggin Explanations aanaiions.... tt_ _Make_ Vision_ and Laad__Lnnguge__ee_
URL https://openaccess thecvf. com http//ooteet__teeeeeeeeeee ICCV_
ICCV ICCV 2019 2t11_00a1re_____________11 ppp.p..
2, 3, 5, 6, 8, 9, 21, 26
[48] Robik Shrestha, Kushal Kafle, and Christopher Kanan. A negative case analysis of visual grounding methods for VQA. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 8172-811 8181, Online, July 2020. Association for Computational Linguistics. doi: 10. 18553v112000.0a00 main.727 URL https::/aclanthholo.0 org/2020) acl-main. 727. 2, 5,
[49] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Workshop at International Confer- ence on Learning Representations, 2013. URL https: htopps://aaae /t////axxt //taaaavvvggg///eee// www.oogg/////...oo httgg/pdf/3112/ 6034. pdf.
17
[50] Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. Gradmask: Reduce overfitting by regularizing saliency. In MIDI 2019 Extended Abstracts, 2019. URL https: / //arxiv. httg/pdf/1// 1904. 07478. pdf. 1, 7, 8,2, 26
[51] Sahil Singla, Mazda Moayeri, and Soheil Feizi. Core risk minimization using salient imagenet. arXiv preprint arXiv:2203. 15566, 2022. URL https://arxii: org/pdff 3, 7, 8, 26
podf/2200 2200.. 15566. pdf.
[52] Joe Stacey, Yonatan Belinkov, and Marek Rei. Supervising model attention with human explanations for robust natural language inference. In AAAI, 2022. URL aattps://arxit org/pdf /2104.100000 08142. pdf. 5, 17
[53] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020. URL https://distill. http:/il...pp///0e// pub/2020/ attribution- ettriibutioon-baeellnee///
[54] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks.
In International Conference on Machine Learning, pages 3319-3328, 2017. URL https:
/ //arxiv. www.oogggg..ooe ppf///dff//// 1703. 01365. pdf. 3, 4, 17
[55] Harini Suresh and John Guttag. A framework for understanding sources of harm throughout the machine learning life cycle. Equity and Access Algorithmss Mechanisms, and Optimization, pages 1-9, 2021. URL https: //ddl acm, ht///oi///dd http///ff/11..0. 11.....113444111111 3483305. 10
14

---

[56] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers, In EMNLP, 2019.
[57] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. image classification. Advances in Neural Information Processing Systems, 33: 18583-18599, 2020. URL htttps:/proceeteee wwwrrns.uupp..... www.iips.ccpppprr////ee.ee cc/ppaper/020//tlle. pdf. 4
Measuring robustness to natural distribution shifts in
[58] Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. Learning what makes
difference from counterfactual examples and gradient supervision. European Conference on Computer Vision, pages 580-599. Springer, 2020. URL ptttps://arxivy weexiv..rgg/eee org/pdf 1tppf/200 /2044. 09034. pdf. 3,3
[59] Damien Teney. Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton Van Den Hengel. On the value of out-of-distribution testing: example of goodhart's law. Advances in Neural Information Processing Systems, 33:407-417, 2020. URL
pawrrcceeinggsnnuur.p...... pttprr/p...cpppppe/////////////e pdf. 3,7, 10
0B511700000112222556666990000
[60] Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance sometimes inversely correlated on real-world datasets, arXiv preprint arXiv:2209.006111 2022.
URL htttt::/t/aaaaii org/paf 2209 00w000061..001 .df...........
[61] Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pages 239-245, 2019. URL https://nnn research..... github. io/
XIML .df.
[62] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, al. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 2021. URL https://arx: 04359, pdf. 10
org/pdf/ /2112/2/2112.. 2212204459990
[63] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, 2019, URL attps://arze org/ ppf///ff//// 1908. 04626. pdf. 17
[64] Jialin Wu and Raymond Mooney. Self-critical reasoning for robust visual question answering. Advances in Neural Information Processing Systems. 32, 2019. URL heurips cc
pdf. 1,
2, 5, 6, 7, 8, 21, 26
[65] Jianwei Yang, Jiayuan Mao, Jiajun Wu, Devi Parikh, David D Cox, Joshua Tenenbaum, and Chuang Gan. Object-centric diagnosis of visual reasoning. arXiv preprint 2rXiv:2012.11587. 2020. URL atttps://rrze www.ii....rrgoeee org/pdf/2012 11587.pd 2, 3, 5,
[66] Ruiqi Zhong, Steven Shao, and Kathleen McKeown. Fine-grained sentiment analysis with faithful attention. arXiv preprint arXiv: 10Xiv:/908.06977 2019. URL https: //trx//axxvvvv//////// org/pdf/ 1908. 1908.06870. pdf. 3,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] All claims made in the abstract and introduction are properly supported by our paper's results and contributions.
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Also see Section 8.
15

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code and instructions for reproducing the main results are in supplement. The datasets are freely available online (urls Appendix E).
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6 and Appendix E and F.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] All main experiments are conducted with random seeds. Hypothesis testing are conducted with 10k bootstrapping. See Section 7.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix F.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? existing assets properly cited.
[Yes] All creators of
(b) Did you mention the license of the assets? [Yes] All assets are used in accordance with their licenses (listed in Appendix E).
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [Yes] See Section 8.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "qhnTIF5tq-e", "reviewer": "Reviewer_jZyx", "summary": "The present an approach to utilize human explanations to train the VQA models to be right for the right reason. In particular, the paper proposed five desiderates to help the VQA model by ensuring (1) important objects sufficiently contribute to the right answer; (2) irrelevant objects produce uniform answer scores; (3) model explanations are similar to human explanations, and (4) model explanations for unimportant objects are close to zero. The paper verifies the contribution of these four objectives and demonstrates a clear advantage over the strong random shuffle baseline. ", "questions": "See the weakness part.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper presents the four desiderates for training the VQA model to be right for the right reasons and proposes the four additional objectives to instantiate each desideratum.  Strong experiment designs verify the value of each objective. The comparison with the baseline model (random shuffle ) and previous work is convincing. \n\nWeakness:\nNot much to say. A minor point is that in section 7.2, the authors try to prove the faithful plausible explanation helps improve accuracy by showing that the accuracy positively correlated to the plausibility. As the goal is to show the improvement, it could be better to compare with a random shuffle baseline by showing the accuracy-plausibility curve for the three categories of faithfulness.\n\nThe paper is original, of good quality, and well-written.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "blIIvEPqlu", "reviewer": "Reviewer_DBWx", "summary": "The paper proposes VISFIS, which aims to improve the performance with the annotated important feature. The model is trained with multiple losses to meet the different restrictions, like with/without important feature and penalization of unimportant features. The paper also proposed an RRR metrics considering RRR-Suffiency, RRR-Uncertainty, RRR-Invariance, Explanation Plausibility, and Explanation Faithfulness. The method is evaluated on CLEVR-XAI, GQA-100k, and VQA-HAT.", "questions": "Line 186: Please cite the paper claimed that experiment result of inefficiency of the attention-based method.\n\nIn the Table. 3, compare with Selvaraju et al, the improvement of proposed method is limited. Also, in Table. 2, the method fails in VQA-HAT and it is not explained well. It is not clear what is the meaning of \"this seems to be an artifact of the VQA dataset\". And if there are \"statistical biases\" in the VQA, we can expect to see a larger gap between OOD ACCs than ID ACCs since the OOD ACCs may have different \"statistical biases\". However, I cannot observe such phenomenon in Table. 2. Can you explain it more detailly?\n\nBold the Suff-Random OOD result on VQA-HAT in Table. 2.\n\nPlease briefly introduce the Suff-human. It is not mentioned until Sec 7.3.\n\nIn Table. 4, it is interesting that the entire model cannot beat the ablated ones in the proposed RRR metrics, but no explanation is given.", "limitations": "Yes. The author discussed about Ethics and Limitations in the paper. I agree that te proposed method does not raise the negative societal impact. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe idea is straight forward, and the approach is clear and easy to follow. \n\nample experiments to show the capability of the proposed methods.\n\nWeakness:\n\nSome descriptions in this paper are a little confusing. \n\nMissing some necessary citation, like the claim of \"early experiments suggest ...\". \n\nThe improvement of the method is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SC95Vi-SOS", "reviewer": "Reviewer_wap4", "summary": "The paper proposes a new multi-faceted training objective that tries to enforce models to make predictions driven by known \"right image regions\" while being invariant to features outside of them (The paper names this FI explanation). The first is achieved via enforcing sufficiency (model should make a prediction given limited but sufficient input), and plausibility (produced importance map should align with known importance map); The latter is achieved via uncertainty (enforcing a max-entropy prediction when predictions are made based on \"not right\" regions), and invariance (when inputs change in the \"not right\" regions, the predictions should remain the same). The paper also proposes new metrics to measure whether a model is \"right for the right reasons (RRR)\". The experiments show that the proposed whole pipeline is indeed more successful than a randomly supervised counterpart. However, the authors find that simply improving RRR metrics does not translate to better OOD performance. ", "questions": "Please take a look at the weaknesses above. Specifically, I would like answers to these questions\n\n- Q1: Why does Suf-Random work so well? \n\n- Q2: Why is the fact that the proposed objectives improve RRR important if 1) they are identical to what was optimized during training and 2) the RRR metrics proposed do not really correlate with OOD performance. \n\n- Q3: Please (re)highlight what new and significant conclusions the readers can take from this work that is not found (via a combination) of prior work.\n\n", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- **[S1]:  The proposed objectives are well-defined, sensible, and implemented satisfactorily:** All four objectives used to train the model (+1 for measuring faithfulness) do truly reflect the desiderata from a model that satisfy the RRR property. While portions of this are found in the literature, this work uses a conjunction of all of them and also does a comprehensive study of which of them are more successful, and which translate to better accuracy or a more faithful model. \n\n- **[S2]: The experiments are thorough and well-presented:** The paper (+ appendix) shows a thorough and well-selected range of experiments that are presented clearly. Especially, the ablations with regard to each of the components in the loss function is well-appreciated by this reviewer. \n\n\n# Weaknesses\n\n- **[W1]: Originality and Significance**: My major concern with this submission is the originality and significance of the conclusions reached by the paper. While the paper does a thorough and sensible job of presenting various aspects of enforcing correct grounding to improve OOD performance, the major parts of it are already studied by prior work (This is accurately and adequately reflected in the related section of the submission). The conclusions and results show only a moderate amounts of improvment over existing work and furthermore offer no explanation about why the \"random\" baselines are still so successful. \nFinally, the conclusions about lack of correlation of RRR and OOD results are not as significant as they might appear at a first glance and they do not add significantly to information already known previously (W3)\n\n- **[W2]: RRR metrics that very closely track actual objectives are not super useful:** The paper reports a greatly improved RRR metrics by use of the objectives described. However, the training objectives are identical to what the RRR measures, which makes it unsurprising that they would do better than the baselines. While this reviewer agrees that the RRR metrics are sensible, there is a good chance that the metrics are not being measured correctly due to noise in the data in case of GQA and VQA-HAT; In VQA-HAT, the \"scratch-off\" style data collection still allows information from the image to be seen at a lower resolution; Furthermore, humans can make assumptions and the \"scrached-off\" section might not be a suitable source to determine the sufficiency (the scratched off regions might not be the only region that is enough to answer) or plausibility. Similarly for GQA, the paper uses\n\n>... obtain bounding boxes of relevant objects identified through the question program, which we use as FI supervision.\n\nwhich can also be noisy in cases of relationships with other objects (e.g., person throwing frisbee). In this case, the same types of correlations present in the \"FI\" supervision during training, will also exist during measurement. \n\nIn short, RRR metrics themselves are not immune from the types of issues present in data that plague \"traditional\" in-domain evaluation. A stronger impact could be reached if the paper discussed how the proposed metrics worked in an absence of a strong and clean supervision about and whether the objectives can learn effectively in presence of noise/sparse training signal. This is not adequately addressed. \n\n- **[W3]: Conclusions reached by the paper are stronger than the degree to which they are actually supported by the paper:** Here are the conclusions copied directly from the paper. \n\n> In this paper, we show that (1) FI supervision can improve VQA model accuracy via our VISFIS method, (2) accuracy improvements appear to stem from improving explanation plausibility specifically for faithfully explained data, and (3) FI supervision can improve RRR metric performance, and (4) RRR metrics do not actually correlate well with OOD accuracy.\n\nFor 1, and 2; The paper correctly acknowledges that 1 alone is not a strong indicator of success as shown by prior work (e.g., [41], [51], since a random baseline can also do almost as well); If #2, was more thoroughly explored, this conclusion would be better supported. However, the paper only presents that the proposed method is *slightly* better than the random baseline, which is still a **very strong** contender (E.g.,2nd best for CLEVR and VQA-HAT among 7 tested). The paper claims that the VQA-HAT is an outlier in some regards, but does not fully explore or offer grounded explanations about why this might be the case.\n\nFor 3, It is unsurprising that this is the case (see W2)\n\nFor 4,  While the full range of RRR metrics as proposed by this work is not studied by prior work; a combination of prior work already establishes (at least a part of) this conclusion. E.g., [40] shows both improved alignment and improved OOD accuracy; [41] shows that it is possible to have one without having the other. While a thorough approach presented here is welcome, it is not a highly novel or significant observation (See W1 for more information). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept