Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces an automatic evaluation framework, called SimEvals, to assess the usefulness of explanations in 3 use cases (forward simulation, model debugging, and counterfactual reasoning). SimEvals mimics user studies by training agents instead of humans. Thus it is cheaper and intended to use before more costly user study. 

=====
After authors' response: My questions have been properly addressed and I am satisfied with the addition of the newly performed user study and the other clarifications being added to the paper. I have increased my score accordingly. 
strengths_and_weaknesses: Strengths
The paper addresses a timely and important topic, that of evaluating the usefulness of explanations in 3 real-world scenarios.
It is well written, easy to read, and clearly states its scope. 

Weaknesses
The 2 main weaknesses I see are: (1) the lack of necessary extensive experiments, and (2) the unconvincing results even among the performed experiments. Since the evaluation method proposed is not of much technical novelty, the main strength of this work was supposed to lie in showing with **extensive experiments** that the results of SimEvals correlate with human results. However, there are very few experiments: only 3 or 4 different explanations are considered for each of the 3 scenarios. And more importantly, only 1 scenario has a comparison with human studies (Table 3). The authors justify the lack of comparison with human studies in footnote 3 by saying that "(a) their interfaces are not publicly available and (b) we do not have access to the same set of participants to compare our study results to theirs.". However, both arguments are extremely shallow: for (a), they could have devised a reasonable interface to do this; for (b), there's absolutely no need to have the exact same participants to draw the conclusion of whether their automatic evaluation correlates with human studies. Finally, the authors claim that "there is a match in the relative rankings" in Table 3, however, we see that the ordering of SHAP, GAM and No explanation are different in SimEvals than in Human studies, and that the only ordering that matches is in the best explanations (from LIME). The authors mention that "an ANOVA test and found a statistically significant difference between the Turkers’ accuracy using LIME vs. all other explanation settings" but they do not mention if there's a statistical difference between SHAP, GAM, and No explanation in both settings, which, if it is, would have indicated different rankings of the explanatory methods between SimEvals and Humans. Also, the authors motivate that a better hyperparameter search for the explanatory methods can be done using SimEvals, but they do not perform any experiments and leave this for future work. Overall, I find the experimental setup very unconvincing, for a work in which this would have been its main focus. 

Minor:
* Tables don't have bold numbers for the best results
L272 evaluates --> evaluate
L280 implements --> implement
Include results from human studies in Table 1
bold results in Table 1
L993 that,
L725&728 ??
L856 Table ??
questions: Maybe the authors could give convincing arguments as to why a comparison with human studies (done by the authors, there's no need of replication of previous results) is not necessary for the 2 scenarios where it has not been performed.
limitations: The authors do not mention anything about potential negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a procedure for evaluating explanations using "simulated evaluations" (SimEvals). The authors demonstrate SimEval using common explanation methods, and demonstrate that their findings align with (human) user responses from a small MTurk survey.
strengths_and_weaknesses: 
This paper clearly presents an idea that I believe is an important contribution to the XAI literature. Their simluations and user study are both relatively small, but their results are convincing. It should be noted that many other researchers have thought about using downstream/user behavior to guide the design of "good" explanations. To my knowledge these studies live mainly in the HCI space. I think it's important for the authors to engage with this literature, because it is very similar in motivation to this work. Some examples:

- https://doi.org/10.1109/TREX51495.2020.00005
- https://doi.org/10.1145/3301275.3302265

It's interesting that the authors chose to frame their method as an alternative or supplement to user studies. I could imagine a version of SimEval that exists completely independently of users, where the purpose of SimEval is to assess how informative an explanation is about a particular model behavior---which is itself a prediction problem. (SimEval does this, but framed in terms of users.)

Since the authors choose to center on users, this raises some important questions:

1) What if the SimEval Agent's behavior differs from the user's behavior? That is, even if an explanation is informative (in a statistical or ML sense) about some aspect of a model, how do we know that this information will also be useful for a human user? I would like to  see the authors engage with this question somewhere in the main body of their paper.

2) The SimEval framework assumes that the evaluator knows what "good" and "bad" downstream behavior looks like. This is not always the case---the three use cases presented here (forward simulation, counterfactual reasoning, and data bugs)are certainly well-motivated, but what about use cases that aren't easily quantified? For example with debugging, we often don't know what the source of a bug is! Is this framework still useful in this case?


questions: See above
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper the authors propose a framework for simulated evaluations (SimEvals) meant to help guide explainable AI (XAI) researchers in terms of what explanations to include in a human subject study. The authors introduce the framework, which involves preparing a dataset with the outputs of the base model (the model that we want to explain) along with any explanations and then training another model as a soft surrogate for human evaluation. The authors go on to show how the performance of the second model on a test set can be seen to roughly correspond to human performance in two existing datasets and then in the results of one novel human subject study. 

strengths_and_weaknesses: The primary strengths of the paper are its formalization of the SimEvals framework, including the guidance and discussion around how and when to apply the framework, and the results section. While other researchers have run experiments that can be understood as instantiations of this framework, to the best of this reviewer’s knowledge, there have been no existing attempts to define this particular framework. That is therefore novel. The results do also demonstrate some value to this approach, though I do have concerns with them that I’ll address below. 

I have two major concerns in terms of the weaknesses of this paper, and they relate to the strengths above. First, there’s the framework. While this is novel experiments like those the framework suggests have been run in the past. As such, the question then turns to the strength of this formalization. In other words, how much is this formalization likely to help other researchers relative to just continuing to have researchers run one-off, domain-dependent experiments? This is unclear to me after reading the paper, particularly because much of the framework is itself domain-dependent. The authors speak to this point in the related work and Appendix A but there’s still no specific argument as to the benefits of this framework in comparison to running domain-dependent versions of these experiments without the framework. The closest we get is that the authors hope that this framework/paper will help future researchers run these experiments, which is certainly possible. But I’m not sure if this means that future researchers will run these experiments faster, be more likely to run this kind of experiment, or so on. 

My second major concern is with the results, which appear to be somewhat weak. At best the results seem to match in terms of the rough ranking (particularly the first place) when comparing user accuracy to agent accuracy. However, there’s cases where the ranking doesn’t match up. For example in Table 3, where the agent is able to get useful information from SHAP and GAM, while these explanations appear actively harmful in the average human case. These results also do not support the claims made by the authors (e.g., “we find strong correspondences between the SimEval agents’ and humans’ accuracy on the use case”). Given the claims, I was expecting something more like a correlation test (e.g., Spearman’s rho) comparing the individual human and agent predictions for the specific individual cases. If there’s some reason such a test would be inappropriate, it would be helpful for the authors to offer an explanation. But the average accuracy doesn’t seem to be particularly representative of human performance, as the authors themselves indicate given the high-variance of LIME results in Table 3. 

Overall, I’m torn between believing there is value to this paper (and more value in the underlying work), but having that tempered by the weak results that do not match up with the claims, and the lack of clarity around what use cases the framework supports over the existing examples. I’ve settled on a weak accept for now but it’s a shaky weak accept.

questions: 1. Why not run a correlation test between the human predictions and the agent’s predictions on the same problems? 
2. What use cases that weren’t already possible or what benefits does the model confer? 

limitations: The authors overall do a good job expressing the limitations of their approach (particularly in the appendices). However, one issue they do not discuss is the possibility of a SimEvals-like approach being used to avoid user studies entirely. While the authors argue against this (“We note that the agent’s performance (i.e. test set accuracy) with a given explanation method is not intended to be interpreted as an anticipated human subject’s performance.”), other claims in the paper would seem to support this use case (“In our experiments, we show that the relative ordering of explanation method performance is consistent for SimEvals and human subject evaluations.”). Some discussion about how this could be avoided in the XAI research field would be appreciated.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a framework called "SimEvals" for conducting use-case grounded algorithmic evaluations of information content presented in a user study. SimEvals involves training an agent that learns to predict the ground truth label given that the same information would be presented to a human user. The test accuracy of the trained agent is used as a measure for the effectiveness of the information content provided. 
strengths_and_weaknesses: Strengths:

1. The paper presents an interesting framework that could potentially aid future user studies in terms of assessing which aspects need actual human user evaluation thereby saving manual efforts and costs

2. The framework also expands the scope of XAI studies in the sense those explanation methods which are often overlooked may also find a place through this scheme

3. The method will offer promising pathways to further research in this area

Weakness:

1. Test accuracy of the agent may not be a wholistic measure of the performance, might need to augment with other metrics as well. 

2. demographics of the AMT workers could have a bearing on the performance validation scheme

3. Might have to consider the influence of cognitive human biases including aspects related to subjectivity, confirmation bias, priming effects among others in evaluation

4. As the study involved human users, information related to institutional ethical clearance should be included in the main paper. 
questions: What were the demographics of the AMT workers who participated in the user study? Would this not have a bearing on the effectiveness of the validation and in turn the proposed framework?

How was it ensured that presenting the AMT workers with the set of scenarios similar to the agent's training and validation phase did not induce a "priming effect"?  If the users were not presented with similar situations, is it possible that the results could have changed?

How does the performance of the framework change with the number of explanations--typically even human users experience fatigue/cognitive overload when presented with a number of explanations. So in the light of this, what kind of advantage or performance benefit the proposed framework can provide?




limitations: Please refer to comments under the last two questions
review_title: null

Meta-review: metareview: The paper proposes simulated evaluations (SimEvals) to guide explainable AI (XAI) researchers about what explanations to include in a user study. All the reviewers agreed that this is a novel contribution to a significant and timely problem. There were common questions around the empirical evaluations that the authors clarified during the feedback phase. 

The reviewers have acknowledged the authors' responses and have confirmed that their questions were adequately addressed.

By adding the new table contrasting prior work that the authors included in their feedback, as well as the clarifications from the reviewer discussion, the paper will be substantially stronger.
confidence: Certain
Decision: Accept