VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives

Abstract: Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions. However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Question Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI. In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility). Our best performing method, Visual Feature Importance Supervision (VISFIS), outperforms strong baselines on benchmark VQA datasets in terms of both in-distribution and out-of-distribution accuracy. While past work suggests that the mechanism for improved accuracy is through improved explanation plausibility, we show that this relationship depends crucially on explanation faithfulness (whether explanations truly represent the model’s internal reasoning). Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful. Lastly, we show that, surprisingly, RRR metrics are not predictive of out-of-distribution model accuracy when controlling for a model’s in-distribution accuracy, which calls into question the value of these metrics for evaluating model reasoning.

Reviews and author responses:

 > What I mean is that say you have a VisFIS model with all your objectives, the explanation are classified into 3 categories. You then train a random shuffle model and collect the results corresponding to the question id in the 3 categories according to the VisFIS model and draw the graph. 

Gotcha, thanks for clarifying! This would be interesting to see and we should be able to make a plot like this for the final version given the results we have on hand. 

> some parts of the VisFIS objectives have been separately studied before

We appreciate you engaging with the other reviews as well here, but just to pitch in one more response to this, we would note that it is still a qualitatively novel contribution that we find a model objective that benefits from feature importance supervision in a way that doesn't suffer the pitfalls which past methods have (i.e. working with random supervision too), and our results have large effect sizes for visual question answering and hold across models and datasets. In particular, improvements in ID acc of up to 2.7 points and OOD acc up to 4.7 points are fairly large as far as VQA-task numbers go (Table 2) (Prior works on VQA tasks like [58] improved OOD acc by 1.5% (1.7% on ‘other’ type questions) and failed to improve ID acc; the improvement in [34] is only up to 1.1%, and it fails to improve ‘other’ type questions acc at all.) Lastly, see the General Response *On main sources of novelty in the paper* paragraph for reiteration of the other three main contributions of our paper besides simply accuracy improvements stemming from the objective terms. Thanks!

 
Thanks for the pointer to table 14 and sorry for the late reply

What I mean is that say you have a VisFIS model with all your objectives, the explanation are classified into 3 categories. You then train a random shuffle model and collect the results corresponding to the question id in the  3 categories according to the VisFIS model and draw the graph.

While I agree to other reviewers that some parts of the VisFIS objectives have been separately studied before and lower my score to 6 or 7, it is still good to see a combination of them works complimentarily. 

 We thank the reviewer for the replies, and we are glad that we have addressed most of your concerns.  

> “However, since measuring RRR and optimizing for RRR is done using the same data (including the same biases and spurious correlations, and other idiosyncrasies), which makes it is unsurprising that a deep network would do well on it.”

Gotcha! Sure this could be unsurprising. I think we would agree this is why it’s important to add in our analysis in Sec. 7.5 which examines what is useful about doing well on these metrics (besides accuracy improvements in Sec. 7.1 and 7.3).

> “RRR as a concept and optimization target is not my critique here, it is the ability of the RRR metric (as proposed and implemented) to actually do its job well and not just capture whatever is necessary to produce better RRR metrics in the dataset rather than actual RRR goals.”

Ok, makes sense! As suggested in a previous comment, “if RRR metrics do not tell us anything more than ID accuracy already does, ultimately we should just measure model quality by ID accuracy and not bother with all these extra metrics (mentioned in L.369-372), which is a more negative conclusion than is presented in past work.” (Also see below in this comment for interpreting this is as a novel negative result in addition to our other main positive results).

> *[W2 weakness.]* “the spurious correlation and other issues can still cause overestimation of ANY skill that is directly optimized on similarly distributed data...still partially unaddressed (esp W2 and how much role it plays)”

We certainly grant that the feature importance annotations (esp. in VQA-HAT) may not be precisely sufficient and comprehensive, but it does not affect three of our main contributions. First, our accuracy improvements (Sec 7.1 & 7.3) are concrete, as we have shown that the improvements do not persist with random supervision, which is itself a sign that the annotations provide useful information to the model (and the effect sizes are large and persist across datasets and models). Second, our discovery of the relationship between accuracy, faithfulness, and plausibility (Sec 7.2) still holds, as issues with annotation quality would only make this relationship harder to see. Lastly, if it helps you think of the contributions in a positive light, please feel free to view all of the RRR results as novel negative results – ultimately we do not find much practical use in them, as even when models do well on RRR metrics, this doesn’t tell us anything new about model generalization (Sec. 7.5). 

At the same time, we want to emphasize that our main RRR improvements in Sec 7.4 are reported on CLEVR-XAI, with additional results on VQA and GQA in the appendix that support the same argument. We are glad that the reviewers can agree CLEVR-XAI has high-quality annotation and is largely immune to the issues you mentioned. Since we see VisFIS improving RRR metrics on CLEVR-XAI (Sec 7.4), we believe the RRR metrics properly reflect the actual RRR goals. Therefore, the improvement on RRR metrics is a meaningful result and not simply a function of dataset biases (though it may not have immediate practical significance, per Sec. 7.5). In short, we think these contributions are trustworthy and are not undermined by the potential deficiencies in annotation quality of VQA or GQA. 

Hopefully, we’ve resolved enough of your questions and clarified our novelty in a way that improves your impression and score for the paper! 


 Thank you for a detailed reply; It was certainly very helpful to read your reiteration of the main points. Since this short discussion cycle is not meant to be an opportunity for substantial revisions, I will just highlight my main impressions:

- Yes, I can see your point about regression analysis being more valuable than correlation analysis (or at least different information). Despite my concerns with the results and my position that this is an important point that can open more ways to do this type of analysis. 

> This issue is less likely to occur in GQA, as it is algorithmically created and has ground-truth important BBs annotation. However, the annotations for CLEVR-XAI are very clean as they are constructed algorithmically based on the data generating process. 

- Cleanliness of labels is not the only source of my concern; Despite the labels being clean, the spurious correlation and other issues can still cause overestimation of *ANY* skill that is directly optimized on similarly distributed data. So, I agree that CLEVR-XAI has a very little issue here, but GQA (despite clean boxes) can still have this issue.

- Overall, I think compared to my original review, I would revise the weights that I assigned to different contributions and results; With higher weights given to the novelty of the results and persistently better results across datasets (not all of which share the deficiencies that I noted) and hence I would assign it a higher overall score than I did earlier. However, some of my other issues, are still partially unaddressed (esp W2 and how much role it plays); I will continue to ponder about this till the end of the discussion period.

 > Sorry if there is a misunderstanding, but we claim that our accuracy improvements are a strong indicator of success precisely because we run the control experiment from [41] with random supervision and find that VisFIS does not benefit from random supervision (L.48-51 & Table 2).

I mean to say that as shown by earlier works as well as well-accepted by this manuscript as well; Is the fact that ID (or OOD) accuracy improvements alone are not an indicator that an algorithm is working for "the right reasons" since it is possible to have a high accuracy without doing any of the other stuff. I think we're both talking about the same thing here just in different words :) 

> Since you agree that the “RRR metrics are sensible,” why is it a weakness to optimize for and achieve good RRR scores?

You are correct. It is not a weakness to optimize for achieving good RRR "abilities". However, since measuring RRR and optimizing for RRR is done using the same data (including the same biases and spurious correlations, and other idiosyncrasies), which makes it is unsurprising that a deep network would do well on it. RRR as a concept and optimization target is not my critique here, it is the ability of the RRR metric (as proposed and implemented) to actually do its job well and not just capture whatever is necessary to produce better RRR metrics in the dataset rather than actual RRR goals. 

 Hi all, thanks for your work in reviewing this submission. As the discussion deadline is fast approaching, we were hoping we could engage in discussion of the paper with each of you regarding the set of comments we posted for each reviewer. We are hoping to clarify some factual details about our paper, particularly around the true novelty of our results relative to the existing literature, and we want to make sure that each reviewers' questions have been fully answered. Thank you and looking forward to the discussion!

 Thank you for your review and comments! We wanted to point your attention to the General Response paragraph *Update to VQA results*, which we hope reaffirms your positive view of our contributions! One more response below:

> “A minor point is that in section 7.2, the authors try to prove the faithful plausible explanation helps improve accuracy by showing that the accuracy positively correlated to the plausibility. As the goal is to show the improvement, it could be better to compare with a random shuffle baseline by showing the accuracy-plausibility curve for the three categories of faithfulness.” 

Thanks for the suggestion! To clarify, do you mean we show acc vs. plausibility & faithfulness curves for VisFIS vs. the baselines? Actually, the relationship between datapoint-level accuracy, faithfulness, and plausibility is very noticeable and consistent across models and datasets, and the plot in 7.2 includes predictions from all of our CLEVR-XAI models, including baselines and supervised models. Models end up with different test accuracies in part because they have different distributions of plausibility & faithfulness for their predictions. So in order to show how these quantities add up to a model’s average test accuracy, we also need to indicate how many points fall into different plausibility & faithfulness categories. We included one table like this in the Appendix (Table 14), but we could include more tables specifically for our unsupervised baseline and for VisFIS. 


 Thank you for your review and comments! We reply to individual points below:

> “Missing some necessary citation, like the claim of ‘early experiments suggest …’.” 

Sorry for the miscommunication here, what is meant is that our early experiments suggested attention was not a helpful explanation for supervision purposes, so we did not explore it further (see also Hyperparameter Tuning in Appendix F - L.780-786). 

> “Also, in Table. 2, the method fails in VQA-HAT and it is not explained well.” 

Please see the General Response paragraph *Update to VQA results*. 

> “The improvement of the method is limited.” 

To our understanding, improvements in ID acc of up to 2.7 points and OOD acc up to 4.7 points are fairly large as far as VQA numbers go (Table 2) (Prior works on VQA like [58] improved OOD acc by 1.5% (1.7% on ‘other’ type questions) and failed to improve ID acc; the improvement in [34] is only up to 1.1%, and it fails to improve ‘other’ type questions acc at all.). With LXMERT (Table 3) our improvement over the baseline is about 1 point, and we find that the second best method from Selvaraju et al. does not achieve statistically significant improvements with LXMERT or UpDn, which is a notable limitation. Following our update to VQA-HAT results, VisFIS is the best method across all datasets and models, as compared to five existing methods and strong unsupervised baselines (see General Response *Update to VQA Results* paragraph). 

> “Bold the Suff-Random OOD result on VQA-HAT in Table. 2.” 

In our original Table 2, we bold numbers that are the best in their group at a statistical significance threshold of p<.05 (described in Table 2 caption), and the Suff-Random OOD result does not meet this threshold. In the new table, Suff-Random clearly does not outperform the baseline on VQA-HAT OOD and also will not need to be bolded (see Table 2 caption for bolding scheme).

> “Please briefly introduce the Suff-human. It is not mentioned until Sec 7.3.”

Thanks for catching this, we will do this!

> “In Table. 4, it is interesting that the entire model cannot beat the ablated ones in the proposed RRR metrics, but no explanation is given.” 

This is a good observation. We were tight on space for elaborating on this result, but we believe that it is due to trade-offs in optimizing for all RRR objectives at once. When optimizing only for RRR-Inv, for instance, it is easier to do well on this metric than when optimizing for all three RRR objectives, where performance on RRR-Inv falls slightly. Evidently, the RRR metrics capture different aspects of model reasoning, since optimizing for them involves some trade-offs in the learned model (but each objective is separately helpful for improving model accuracy).


 > “The paper correctly acknowledges that [conclusion] 1 alone is not a strong indicator of success as shown by prior work (e.g., [41], [51], since a random baseline can also do almost as well)”

Sorry if there is a misunderstanding, but we claim that our accuracy improvements are a strong indicator of success precisely because we run the control experiment from [41] with random supervision and find that VisFIS does not benefit from random supervision (L.48-51 & Table 2). 

> “If [conclusion] #2, was more thoroughly explored, this conclusion would be better supported. However, the paper only presents that the proposed method is slightly better than the random baseline” 

Sorry, we are not sure why random baseline vs. VisFIS performance has anything to do with our explanation of how supervision can improve accuracy. We support this claim in Sec. 7.2, which uses an analysis of prediction accuracy as a function of explanation plausibility and faithfulness using 4 million predictions from 100 unique CLEVR-XAI models, including baseline, unsupervised, and supervised models (trends consistent across models and datasets). 

> “However, the training objectives are identical to what the RRR measures, which makes it unsurprising that they would do better than the baselines” 

Since you agree that the “RRR metrics are sensible,” why is it a weakness to optimize for and achieve good RRR scores?

 Thanks for your thorough comments! Below we respond to individual points:

> “The major parts of [the paper] are already studied by prior work…Please (re)highlight what new and significant conclusions the readers can take from this work that is not found (via a combination) of prior work.” 

We are happy to highlight some of the main points again. Please refer to the General Response paragraph *On main sources of novelty in the paper.*

> “The conclusions and results show only a moderate amount of improvement over existing work” 

To our understanding, improvements in ID acc of up to 2.7 points and OOD acc up to 4.7 points are fairly large as far as VQA numbers go (Prior works on VQA like [58] improved OOD acc by 1.5% (1.7% on ‘other’ type questions) and failed to improve ID acc; the improvement in [34] is only up to 1.1%, and it fails to improve ‘other’ type questions acc at all.). Please also note that VisFIS is significantly better than all five prior methods and strong unsupervised baselines across all three datasets (see the General Response paragraph *Update to VQA results*).

> “the conclusions about lack of correlation of RRR and OOD results are not as significant as they might appear at a first glance and they do not add significantly to information already known previously…[40] shows both improved alignment and improved OOD accuracy; [41] shows that it is possible to have one without having the other”

First, priors works only consider explanation plausibility and related “grounding” metrics, not RRR metrics or faithfulness metrics of the kind we consider. Second, the most important novelty in our analysis of RRR metrics and OOD performance is our use of a regression analysis rather than correlational analysis, which has not been done before. We test whether RRR metrics predict OOD performance while controlling for ID accuracy. Neither of the works cited above use this important control. This control is important because if RRR metrics do not tell us anything more than ID accuracy already does, ultimately we should just measure model quality by ID accuracy and not bother with all these extra metrics (mentioned in L.369-372), which is a more negative conclusion than is presented in past work. Does this help contextualize this analysis relative to what past work has done, or is there part of your concern that we’ve left unaddressed?

> “Why does Suf-Random work so well?” 

First, please note that the Suff-Random method actually only helps with CLEVR-XAI and UpDn. For GQA and VQA-HAT it does not help (Table 2), nor with CLEVR+LXMERT (Table 3). Now to answer your question: in an image classification context, random cropping has long been a popular form of data augmentation to help with generalization, and it is commonly believed that the reason it works well is because visual classes are often highly overdetermined by visual features (see “Data augmentation using random image cropping and patching for deep CNNs”). As a consequence, random feature subsets are usually sufficient for arriving at the label. Random subsetting during training helps with model generalization because it encourages models to pick up on all of these sufficient features, which enables it to perform better on test data when the exact combinations of visual features seen during training may not reappear. The same should hold for VQA tasks where a question like “what object is next to the green triangle” should be answerable with many different feature subsets (as long as they include the sufficient features). 

> “there is a good chance that the metrics are not being measured correctly due to noise in the data in case of GQA and VQA-HAT” 

We certainly grant that the annotations for VQA may not be precisely sufficient and comprehensive, so RRR-Suff, RRR-Inv, and RRR-Unc calculations may have some issues with their inputs. This issue is less likely to occur in GQA, as it is algorithmically created and has ground-truth important BBs annotation. However, the annotations for CLEVR-XAI are very clean as they are constructed algorithmically based on the data generating process. Since we see similar trends in terms of ID acc, OOD acc, and the RRR metric vs. OOD acc relationship across all of our models and datasets, we believe that we are measuring RRR metrics in a reasonable way and our conclusions are trustworthy. We agree that it would be helpful in future work to explore how VisFIS is robust to labeling noise for purposes of improving accuracy, but the analysis components of our paper (Sec. 7.2, 7.4, and 7.5) rely on high quality annotation of FI. 


 We thank the reviewers for their time and effort in reading our paper. Below we highlight some commonly noted strengths and weaknesses that we will reference in individual responses to each reviewer, for the sake of brevity. *NOTE: line numbers mentioned below are from the updated pdf version, but all referenced lines are already present in the original submission.*

First, we are happy that reviewers found that our “proposed objectives are well-defined, sensible and implemented satisfactorily” (Wap4) and that they demonstrate “a clear advantage over the strong random shuffle baseline…and previous work” (jZyx). We also appreciate the observations that the experiments were “thorough and well-presented” (Wap4), include an “ample” number (DBWx), and showed “strong experiment design” (jZyx). 

*On main sources of novelty in the paper*: To resummarize our main novel contributions: (1) we show, for the first time, that feature importance supervision can improve performance on VQA tasks beyond what random supervision can provide (mentioned in L.48-51, Sec 7.1); (2) we offer a novel explanation of how supervision improves performance by looking at both explanation plausibility and faithfulness, which is consistent across different objectives and datasets (mentioned in L.52-54, Sec. 7.2); (3) we provide two new RRR metrics, RRR-Inv and RRR-Unc (mentioned in L.63-65, Sec. 5); and (4) we show that the RRR metrics, which have been suggested to be evidence of model generalizability and OOD performance, do not actually tell us how models generalize beyond what can be known from in-distribution accuracy. This last result is supported by regression analyses rather than purely correlational analyses that prior work has done (mentioned in L.69-76, Sec. 7.5). 

*Update to VQA results*. We have updated our VQA results and are pleased to see that VisFIS continues to be the best method out of all the methods we consider across all three datasets. In our original submission, we reported accuracy results on our entire VQA-HAT test split, but we believe we should update these results according to the best practices outlined by Teney et al. (2020). Originally, we made sure to follow Teney’s recommendations on model checkpointing and other evaluation guidelines, but we reported accuracy results on all question types in VQA-HAT rather than only on “other” type questions, which Teney et al. recommend since these questions are “much less likely [than yes/no questions] to be answered correctly with a naive or malfunctioning methods.” So, we have updated our main results table to report accuracy only on the harder "other" type questions for VQA-HAT according to these guidelines. The results on all questions are in the appendix for reference. Now, we find that feature supervision does improve performance over baselines, making our accuracy results consistent across all three datasets. 

*Additional prior work baselines*: We also added another method from concurrent work, Singla et al. (2022), for further comparison. This method uses a supervised Sufficiency objective based on adding noise to unimportant features during training. This is often the second best method behind VisFIS, but VisFIS outperforms the method for all datasets and model architectures. 

 The paper proposes a new multi-faceted training objective that tries to enforce models to make predictions driven by known "right image regions" while being invariant to features outside of them (The paper names this FI explanation). The first is achieved via enforcing sufficiency (model should make a prediction given limited but sufficient input), and plausibility (produced importance map should align with known importance map); The latter is achieved via uncertainty (enforcing a max-entropy prediction when predictions are made based on "not right" regions), and invariance (when inputs change in the "not right" regions, the predictions should remain the same). The paper also proposes new metrics to measure whether a model is "right for the right reasons (RRR)". The experiments show that the proposed whole pipeline is indeed more successful than a randomly supervised counterpart. However, the authors find that simply improving RRR metrics does not translate to better OOD performance.  # Strengths
- **[S1]:  The proposed objectives are well-defined, sensible, and implemented satisfactorily:** All four objectives used to train the model (+1 for measuring faithfulness) do truly reflect the desiderata from a model that satisfy the RRR property. While portions of this are found in the literature, this work uses a conjunction of all of them and also does a comprehensive study of which of them are more successful, and which translate to better accuracy or a more faithful model. 

- **[S2]: The experiments are thorough and well-presented:** The paper (+ appendix) shows a thorough and well-selected range of experiments that are presented clearly. Especially, the ablations with regard to each of the components in the loss function is well-appreciated by this reviewer. 


# Weaknesses

- **[W1]: Originality and Significance**: My major concern with this submission is the originality and significance of the conclusions reached by the paper. While the paper does a thorough and sensible job of presenting various aspects of enforcing correct grounding to improve OOD performance, the major parts of it are already studied by prior work (This is accurately and adequately reflected in the related section of the submission). The conclusions and results show only a moderate amounts of improvment over existing work and furthermore offer no explanation about why the "random" baselines are still so successful. 
Finally, the conclusions about lack of correlation of RRR and OOD results are not as significant as they might appear at a first glance and they do not add significantly to information already known previously (W3)

- **[W2]: RRR metrics that very closely track actual objectives are not super useful:** The paper reports a greatly improved RRR metrics by use of the objectives described. However, the training objectives are identical to what the RRR measures, which makes it unsurprising that they would do better than the baselines. While this reviewer agrees that the RRR metrics are sensible, there is a good chance that the metrics are not being measured correctly due to noise in the data in case of GQA and VQA-HAT; In VQA-HAT, the "scratch-off" style data collection still allows information from the image to be seen at a lower resolution; Furthermore, humans can make assumptions and the "scrached-off" section might not be a suitable source to determine the sufficiency (the scratched off regions might not be the only region that is enough to answer) or plausibility. Similarly for GQA, the paper uses

>... obtain bounding boxes of relevant objects identified through the question program, which we use as FI supervision.

which can also be noisy in cases of relationships with other objects (e.g., person throwing frisbee). In this case, the same types of correlations present in the "FI" supervision during training, will also exist during measurement. 

In short, RRR metrics themselves are not immune from the types of issues present in data that plague "traditional" in-domain evaluation. A stronger impact could be reached if the paper discussed how the proposed metrics worked in an absence of a strong and clean supervision about and whether the objectives can learn effectively in presence of noise/sparse training signal. This is not adequately addressed. 

- **[W3]: Conclusions reached by the paper are stronger than the degree to which they are actually supported by the paper:** Here are the conclusions copied directly from the paper. 

> In this paper, we show that (1) FI supervision can improve VQA model accuracy via our VISFIS method, (2) accuracy improvements appear to stem from improving explanation plausibility specifically for faithfully explained data, and (3) FI supervision can improve RRR metric performance, and (4) RRR metrics do not actually correlate well with OOD accuracy.

For 1, and 2; The paper correctly acknowledges that 1 alone is not a strong indicator of success as shown by prior work (e.g., [41], [51], since a random baseline can also do almost as well); If #2, was more thoroughly explored, this conclusion would be better supported. However, the paper only presents that the proposed method is *slightly* better than the random baseline, which is still a **very strong** contender (E.g.,2nd best for CLEVR and VQA-HAT among 7 tested). The paper claims that the VQA-HAT is an outlier in some regards, but does not fully explore or offer grounded explanations about why this might be the case.

For 3, It is unsurprising that this is the case (see W2)

For 4,  While the full range of RRR metrics as proposed by this work is not studied by prior work; a combination of prior work already establishes (at least a part of) this conclusion. E.g., [40] shows both improved alignment and improved OOD accuracy; [41] shows that it is possible to have one without having the other. While a thorough approach presented here is welcome, it is not a highly novel or significant observation (See W1 for more information).  Please take a look at the weaknesses above. Specifically, I would like answers to these questions

- Q1: Why does Suf-Random work so well? 

- Q2: Why is the fact that the proposed objectives improve RRR important if 1) they are identical to what was optimized during training and 2) the RRR metrics proposed do not really correlate with OOD performance. 

- Q3: Please (re)highlight what new and significant conclusions the readers can take from this work that is not found (via a combination) of prior work.

 N/A

 The paper proposes VISFIS, which aims to improve the performance with the annotated important feature. The model is trained with multiple losses to meet the different restrictions, like with/without important feature and penalization of unimportant features. The paper also proposed an RRR metrics considering RRR-Suffiency, RRR-Uncertainty, RRR-Invariance, Explanation Plausibility, and Explanation Faithfulness. The method is evaluated on CLEVR-XAI, GQA-100k, and VQA-HAT. Strength:

The idea is straight forward, and the approach is clear and easy to follow. 

ample experiments to show the capability of the proposed methods.

Weakness:

Some descriptions in this paper are a little confusing. 

Missing some necessary citation, like the claim of "early experiments suggest ...". 

The improvement of the method is limited. Line 186: Please cite the paper claimed that experiment result of inefficiency of the attention-based method.

In the Table. 3, compare with Selvaraju et al, the improvement of proposed method is limited. Also, in Table. 2, the method fails in VQA-HAT and it is not explained well. It is not clear what is the meaning of "this seems to be an artifact of the VQA dataset". And if there are "statistical biases" in the VQA, we can expect to see a larger gap between OOD ACCs than ID ACCs since the OOD ACCs may have different "statistical biases". However, I cannot observe such phenomenon in Table. 2. Can you explain it more detailly?

Bold the Suff-Random OOD result on VQA-HAT in Table. 2.

Please briefly introduce the Suff-human. It is not mentioned until Sec 7.3.

In Table. 4, it is interesting that the entire model cannot beat the ablated ones in the proposed RRR metrics, but no explanation is given. Yes. The author discussed about Ethics and Limitations in the paper. I agree that te proposed method does not raise the negative societal impact. 

 The present an approach to utilize human explanations to train the VQA models to be right for the right reason. In particular, the paper proposed five desiderates to help the VQA model by ensuring (1) important objects sufficiently contribute to the right answer; (2) irrelevant objects produce uniform answer scores; (3) model explanations are similar to human explanations, and (4) model explanations for unimportant objects are close to zero. The paper verifies the contribution of these four objectives and demonstrates a clear advantage over the strong random shuffle baseline.  Strengths:
The paper presents the four desiderates for training the VQA model to be right for the right reasons and proposes the four additional objectives to instantiate each desideratum.  Strong experiment designs verify the value of each objective. The comparison with the baseline model (random shuffle ) and previous work is convincing. 

Weakness:
Not much to say. A minor point is that in section 7.2, the authors try to prove the faithful plausible explanation helps improve accuracy by showing that the accuracy positively correlated to the plausibility. As the goal is to show the improvement, it could be better to compare with a random shuffle baseline by showing the accuracy-plausibility curve for the three categories of faithfulness.

The paper is original, of good quality, and well-written. See the weakness part. The authors have adequately addressed the limitations and potential negative societal impact of their work.

Meta-review: 2 out of 3 reviewers are positive about the paper and recommend accept after the author feedback. 

Importantly the paper contributes and the reviewers value 
- the idea and design to make VQA models right for the right reasons
- the experimental validation, including comparison to the recent work Singla et al. [47].
- new RRR-metrics
- analysis of the metrics including exposing limitation its use to estimate for OOD performance

The authors have addressed several of the concerns of the authors, and I recommend acceptance under the expectation that the authors will revise the paper as promised in the author response, including
- adding a plot for a random shuffle model to address reviewer JZyx's concern.
- adding clarifications provided in the author response and discussion with the reviewers to the paper/supplement wherever possible

Decision: Accept