Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the problem of answering queries about "foils,"
i.e., why an alternative plan was not chosen by an agent acting
optimally in a deterministic MDP. The authors describe three broad
classes of responses to this query: (1) one of the actions in the foil
does not satisfy the preconditions, (2) the foil does not achieve the
goal, or (3) the foil has suboptimal cost. Importantly, all responses
are conducted with respect to a pre-specified set of concepts
(predicates; or binary classifiers): the symbolic preconditions and
costs are learned by interaction with the simulator but expressed in
the language of these concepts. The authors discuss extensions to
their basic framework that (1) provide confidence measures along with
the responses and (2) handle noisy/probabilistic concepts.

A strength of this paper, in my opinion, is that it addresses a very
important problem and seems to make a lot of positive strides toward a
solution. I found the motivating paragraphs in the introduction to be
highly compelling, and the related work section to be sufficient for
placing this paper with respect to related literature on
explainability (which I am not very familiar with). Another plus point
is that the authors show that their system is robust to uncertainty
(via the confidence measures) and noise (via the probabilistic
concepts).

However, my main issue with this paper surrounds the assumption of the
pre-specified concepts. The most compelling motivational sentence to
me was, "More often than not...lay user"; however, if we are assuming
that lay users are the target users of the proposed system, where
would these concepts come from? I understand that they could be
specified ahead of time, but then there would be issues if the user
has a concept in mind while specifying a foil that is not covered by
the pre-specified set. The authors do partially address this concern
in the sentences "An empty list...task-related concepts.", but I did
not find this sentence compelling on its own, and it seemed to me that
the empirical studies do not consider this possibility of needing to
add more concepts to be able to answer a query (please do correct me
if I have missed this). To me, one of the most interesting aspects of
this problem is to consider how to automatically learn or improve the
initial set of concepts, perhaps given data of many users' foils
across different problem instances in a particular domain. For
instance, if we notice that users often provide a foil that involves
walking into the skull (we can figure this out with our simulator), we
may be able to learn that a concept NextToSkull is important when
building our symbolic preconditions. Unfortunately, the current paper
does not focus enough on this important aspect of the problem, instead
simply assuming a good set of concepts to be pre-specified, which I
find to be highly unrealistic.

A second major issue is the relatively low quality of the empirical
results. I appreciate that in this line of work, it can be hard to
provide rigorous numbers, since much of the evaluations come from
humans. Nevertheless, I believe that one can obtain much more
illuminating results in this problem setting by improving the
experimental design and reporting. A simple starting point would be to
include confidence measures on the reported numbers, in the paragraphs
discussing the results of H1/H2 and H3. Beyond that, it would be good
to probe deeper, e.g. for H3, cluster (and show us visualizations of)
the different situations where the precondition-based explanations
were useful while the saliency-based explanations were not, and vice
versa, so that we can better understand when each one tends to be
better. Personally, I could imagine many situations (e.g., in
understanding the behavior of an autonomous vehicle) where I would
rather just have the saliency map that highlights a region of my
surroundings, instead of being given a written explanation as to why
the vehicle performed a certain maneuver. So, I think the paper could
be greatly improved if the authors probe more deeply into explaining
the empirical results. By the way, much of the work by Anca Dragan (I
am unaffiliated), see e.g. [1], can be a useful reference in how to set up highly
rigorous user studies.

Looking forward, I have one question for the authors. It is clear that
in real-world applications of AI, our agents will never be able to act
optimally, and we shouldn't expect them to. Given this, would there be
a way to modify the current work to create feedback from the user that
allows the agent to improve its solution? For instance, perhaps the
agent is in the middle of executing the best solution it found after
thinking for 30 minutes, but then the user says "why not do X
instead?", and the agent has to decide between: (a) *the user* missed
something, and I should generate an explanation; or (b) *I* missed
something, and I should think more and revise my current policy.

[1] Bobu, Andreea, et al. "Learning under Misspecified Objective Spaces." Conference on Robot Learning. 2018.
review_title: Interesting problem, but unrealistic assumptions and unclear evaluations

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a novel approach to generate contrastive explanations in a dialogue setting between a human and a planning agent. The setting assumes that the agent generates and offers an optimal plan to the user, and the user in turn challenges the presented plan offering an alternative (i.e. a contrast/foil). The goal of the agent is the denounce the alternative plan by explaining the infeasibility or suboptimality of the plan to the user in concepts they understand.

The explored direction is interesting and relevant as it seems to be a natural addition to the related problem of *generating* contrastive (a.k.a. counterfactual) explanations. I would suggest, however, that the paper more clearly distinguishes between the contrastive explanation generation literature (see, e.g., a survey [1]) with the type of explanation which is offered here, which is to identify the minimal set of preconditions (in an alternative concept space) that describes/explains the difference between two given instances (i.e., a model-proposed fact and a human-generated foil). This motivation is related to such (missing) related work as [2].


Strengths:
- the writing throughout well-polished and the motivation in the abstract and introduction is very well done
- the formulations are sensible and seem to be encapsulating the settings described and generalizations
- the inclusion of a user-study is helpful


Suggestions:
- the overloading of notation is at times difficult to follow
- as a reader not familiar with Montezuma's foils or Sokoban, I had a difficult time understanding the experimental section. 
- a comparison with optimal explanations is lacking (relatedly, statements such as "the searchable to identify *the* expected explanation" is misleading, as, without an infinite budget and exhaustive search, the algorithm can identify an approximate to the optimal explanation)
- re: user study, I am not entirely convinced that the baselines are fair. by default, I would expect that offering more (non-fooling) information would render a higher subjective explainability score. perhaps the experiments would be stronger if tested against other types of information that is provided in addition to the presented baseline (something along the lines of H3; although even here it is not a completely fair comparison because unlike saliency maps, the offered explanations depend on human-annotated concepts which would naturally render the presented explanations are more human-understandable)
- nit; a completeness score of 3.36 / 5 is not possible (this would mean one of the 20 participants voted a non-integer value)
- in the related work, there seems to be an absence of literature on planning; perhaps a differentiation of the scope of the presented paper with this literature would boost the motivation


Summary:
I think this paper explores a fresh direction, which is necessary for enabling humans to contest, challenge, and ultimately trust the decisions of an automated system. I also believe that the presented material still requires a lot of further work and attention, and would be happy to see it accepted so the community can further explore these directions (fair user studies, the relation between approximate and optimal plans, investigations into the properties of foils relative to optimal actions, etc.). If accepted, I would strongly suggest a better integration of the main body and appendix, especially for Sec 3-5).


[1] Karimi et al., https://arxiv.org/abs/2010.04050
[2] Goyal et al., https://arxiv.org/abs/1904.07451

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Edit: 

I have read the other reviews as well as all author responses. The other reviewers noted meaningful concerns, but I believe the authors have clearly addressed most of these points. I still believe this work is an "accept".


Summary: 

The authors present a system for producing explanations in terms of user-specified prerequisite relationships. The authors train a classifier to detect the presence of user-specified concepts. Relationships between these concepts are found by learning a partial symbolic model. With this model, an agent's action can be compared to a user-provided alternative and the first prerequisite-violating action can be identified (or the cost difference can be conveyed). The authors evaluate their approach on two domains with a user study. 

Pros:

-This work addresses a relevant problem in explaining RL (and other SDM) agents and provides an initial solution that provides a solution for a meaningful set of domains. 

-I agree with the authors: to my knowledge, this is the first work to provide explanations in terms of a learned symbolic model separate from the one used by the agent.

-The method for identifying failed preconditions is clearly introduced and motivated. The extension to handle noisy classifiers substantially increases the usefulness of this work. 

Cons:

-This work requires a deterministic domain, as well as access to a simulator from which states can be sampled. This substantially limits the applicability of this approach. (This limitation is not mentioned in the abstract.)

-The authors make a number of assumptions (Section 4, "Confidence over explanations"), but these are not quantitatively evaluated. Adding experiments that measure how well these assumptions hold in practice would improve this work.

-The user study could be improved in a number of ways. It used manual translation of explanations to text. A comparison was made to saliency maps, but not to other explanations (such as causal explanations). 

Questions During Rebuttal Period:

-Please address and clarify the "Cons" above. 

-How important was the change to Montezuma's Revenge (rendering failed actions)? Is this a general requirement for creating explanations in an environment?

-Do you have additional information about the participants? Were they AI practitioners? Were any of the "main study" participants among those who selected the 25/38 concepts? 

Other Comments:

-The authors may be interested in "Distal Explanations for Explainable Reinforcement Learning Agents" (a follow-up to the Madumal et al, 2020 work cited in Section 6). 

-Section 5 appears to be broken into fewer subsections / has fewer line breaks in order to meet page requirements. This leads to a cramped, less organized Section 5. 
Some Typos/etc:

-In general, another editing pass for articles and agreement of subject/verb plurality would help this paper. 

-Line 2 of Introduction: "they ... with its" (change to "with their"). 

-Paragraph 3 of Introduction: The "'" after "questions of this form" is not matched. 

-Background: "We will consider goal-directed agents that any given point is trying to drive" (fix verb agreement and remove extra words). 

-The sentence leading into Definition 4 does not smoothly lead to Definition 4. 

-Section 4: "So we start with" (remove "so").

-Section 4: "we aren't be able" (remove "be"). 

-Section 5: AdaBoost citation is using wrong command. 

review_title: Presents a novel explanation type and does so well

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The authors propose a method of explainable AI for inscrutable blackbox models. The explanations build on a set of user-defined primitives, independently trained on the blackbox representation (e.g., visual frames of an Atari game), and use an increasingly popular method of providing contrastive explanations. Two forms of foil-based responses are provided: (1) indication of action failure from the planning perspective (preconditions unsatisfied); and (2) an explanation of relative sub-optimality that highlights key aspects of action costs that the user may be unaware of.

High-level concepts, particularly those tied to a symbolic description of the world dynamics, is an extremely compelling basis for explanation. It helps build a well-grounded intuition with human users/observers of autonomous systems and arguably is the best way to convey explanations that describe behaviour of an inherently sequential nature.

In addition to the form of explanation primitives, the algorithms are intuitive, and the probabilistic inference seems to be sound.

My concerns with the paper fall into two main categories: the lack of substantial contributions (particularly as related to representation learning) and the strong assumptions placed on the setting.

One of the most significant missed opportunities in this work is to focus on introducing new concepts. Especially given that human studies were conducted, and the setting was identified when algorithms fail, and new or revised concepts are required. Assuming highly accurate binary classifiers for each concept is relatively extreme, and it's only one such overly strong assumption.

Other very strong assumptions include:

(a) The state is memoryless/Markovian: every concept can be determined by looking exclusively at the current frame. This isn't the case in many settings where some memory of the previous actions is required.

(b) The distribution of a fluent across the state space is independent to the distribution of other fluents: this is rarely the case in planning-like domains, and the types of explanations introduced in this work build on planning-like domains a great deal.

(c) There is only one failed precondition: this might be an alright assumption to make given (b), but similarly, I find it unlikely that many domains would have this property.

As pointed out by the authors, assumptions (b) and (c) cause Algorithm 2 to exhaust the entire sampling budget before failing, and I don't believe they are safe assumptions to make.

On the topic of evaluation, there are two further issues. One is the scope of the evaluation (only two domains and a seemingly small number of subjects to test with), and this reduces the significance of the paper's contribution. Another issue is the choice of comparison for H3. The Saliency map is built using different information than that surfaced using the proposed approaches. This makes it challenging to adopt the experiment's conclusion as written since it is also testing the quality of the saliency map in addition to the comparative nature of the explanations.

I am leaning towards rejecting the paper due to the number of assumptions placed on the approach. Combined with the limited evaluation setting and scope of work, the contributions to the field of learning representations seem limited.

Ultimately, my hesitation in recommending acceptance comes from the contribution being on the low-side for the ICLR community. The authors identify key elements that would change this impression -- e.g., refining the concepts when the algorithms fail to find an explanation (italics on pg 5) -- but these do not play a central role in the proposed work.

The H1/H2 results are, in some sense, evident that the proposed explanations are preferred (19/20 for H1). While an important element (it would be very surprising if this weren't the case), they don't serve as a sufficient contribution in their own right. The H3 comparison seems to be somewhat contrived since they come from different sources -- a more accurate comparison would be to engineer the saliency overlay based on the domain-knowledge known. I.e., reflecting the precondition-based information directly. Without that, you conflate both the choice of focus and ability to highlight that choice.


Questions for the authors:

1. How do you remove the (seemingly strong) assumption that the distribution of fluents across the state space is independent among the fluents? Alternatively, why can we expect this to be a reasonable assumption to make?

2. How would you remove the dependence on the fully observable / Markovian assumption on the blackbox output that is used for concept classification? I.e., when the full state cannot be discerned by looking at the screen alone.


Other minor points of improvement for the paper:

o Mind the notation used for your Goal set (near the end of page 2, you are using a different syntax than the one introduced.

o Defn 3 seems to have a random bracket at the end.

review_title: Interesting setting for explanations of sequential decision making, but too many assumptions at play.

final_score: rating: 7: Good paper, accept
review_content: #### Summary
This paper introduces a new method for contrastive explanation of symbolic models on sequential decision-making problems, i.e. explaining why a foil plan is not as good as the system proposed plan. The suboptimality of foil plans is categorized into two types: invalid actions and larger cost.

To make the explanation more understandable to human, the author introduces "concepts" to explanation, which are assumed propositional properties of states. Concept classifiers can be trained on samples to predict the presence of each concept in a state. To explain an invalid action, the method reports a missing precondition concept of the failing action; to explain a larger cost, it reports a set of representative concepts such that the foil actions under these concepts are guaranteed to give a larger cost than the proposed plan. The paper also provides detailed algorithms to find preconditions or representative concepts (using "abstract cost functions") for actions. The authors also introduce simple PGMs to evaluate the confidence of each explanation.

The authors conducted user studies on the proposed explanation method and demonstrated its usefulness.

#### Strength
- Model explanation is indeed a very important field today for any ML models. Sequential decision-making models are popular and widely used, but explanation methods for them are still somewhat underexplored. As reported in the paper, previous work either includes "concepts" directly in model learning (not post-hoc) or use saliency maps (not human concepts); this work introduces human concepts to post-hoc explanation.
- The whole explanation method is novel and makes sense to me. The ideas are intuitive, but the algorithms for finding the explanations and evaluating the confidence are non-trivial.

#### Weakness
- In my opinion, the user study still has room for improvement.
	- For H1 and H2, directly asking the user "which one is more useful" does not look like a good way of comparison. At least to me, it's hard to tell which one is more useful if I have already read both and understood. Maybe it's better to only show one explanation to a participant, and ask how well they understood, or let them perform a task as in H3.
	- For H1, as I see in Figure 8B, the explanation with concepts (CE) has exactly more information than the baseline explanation (BE). In other words, CE is concept + image while BE is only image; BE is a strict subset of CE. Such a comparison does not seem very useful. Would it be better if you, for example, compare concept-only with image-only, such as removing "in the state shown in the image" in CE?
	- For H3, the task setting is interesting, but in the shown example, the concept that the user has to learn is merely to use the switch, which looks somewhat too simple and not very interesting. It would be more attractive if the concept is more complex (e.g. you can't fall off a plane or touch an enemy as in Montezuma).
	- I think the participants' background information (age, gender, how recruited, etc.) is important for a user study and I suggest having it in the main paper or at least making it easier to notice.
- It seems that empirical studies are not provided for confidence score calculation, i.e. how well do the scores correspond with actual explanation accuracy.

#### Questions
- In Section 4 line 16, you mentioned "In cases where we are guaranteed that the concept list is exhaustive..." Could you provide some examples of these cases?

- In Section 5 "Explanation identification", you mentioned "The search was able to identify the expected explanation for each foil." Does that mean all the output explanations are accurate? If so, it will be interesting to see how the search performs on more complex tasks.
- For explanations on larger cost, as in Figure 9B, to me the most ideal explanation would be just *Executing action "push up" when "Box on pink cell" costs at least 10*; other lines are not useful. In other words, it might be better to focus on steps where the foil has larger cost than proposed plan. Do you have any thoughts on this line?

#### Typos
- Section 2: symbol $G$ (set of goal states) font inconsistent
- Appendix A.3 Algorithm 2 line 12: symbol broken

review_title: Interesting explanation method; minor concerns in user study

Meta-review: This paper is an intriguing study of agents that can give explanations (contrastive) of their actions via symbolic representation such as dialog.  Agents can also allow users to argue against the agents' decisions. I am extremely impressed by the quality of the reviewer comments and discussions.  It is also interesting that the reviewers have formed two camps of thought on the paper: One camp consists of R3 and R5 who are in agreement in vociferously criticizing the weak points in the paper.  The other camp consists of R1, R2, and R4, who champion the merits of what they see as strong points.  Notably, all reviewers have fairly high confidence values -- only one confidence score of 3 and all others are 4.

It was a borderline case and not an easy decision. In the end the program committee decided that the paper in its current form does not quite meet the bar, and would benefit from another revision (see e.g., R4 comments).  We think that the work is interesting, and encourage the authors to address the reviewers' comments and resubmit the work to another venue.

Decision: Reject