Venue: ICAPS 2019 Workshop XAIP

Reviews:

final_score: rating: 5: Top 15% of accepted papers, strong accept
review_content: A main idea of the paper, that a contrastive explanation should be answered 'universally' rather than 'existentially' is extremely interesting and novel. 

The capability of this approach to explain how different goal subsets exclude each other is powerful because in any reasonably complex domain, this is could be unknown to the user. Such an understanding could also be helpful for other goal reasoning components if the planner is part of a larger cognitive system. 

The evaluation is quite strong, and the experimental setup is described in detail. For Figure 2, it seems that the only analysis of the results is the last sentence before the conclusion, that " Smaller numbers of goal facts tend to be quite easy, larger ones mostly infeasible, with variance depending on the domain and algorithm. ". Is this the only interesting analysis of this figure? Even though space is tight, I would have appreciated more analysis here.

The paper is highly relevant to the workshop.

minor comments:
	* Section numbers throughout the document are missing


review_title: Excellent paper on universal answers to contrastive questions

final_score: rating: 3: Marginally above acceptance threshold
review_content: The paper proposes an interesting framework for explaining plans in terms of binary plan properties.

> Description versus Explanation: My biggest criticism of the framework would be that it is an explanation framework without any consideration of the explainee in it! What is the model of the person who is receiving the explanation? How do they evaluate plans? Are they optimal planners? Can they arrive at the same plan space given a problem? Are they able to evaluate the entailment property completely and accurately? 

---> 1. What is the definition of an explanation in this framework? 
---> 2. Under what assumptions of the explainee model is this framework capable of producing a valid explanation?

Without these key definitions in the framework, I would say that the proposed work is one of plan description rather than explanation. The need for evaluations is noted only as an afterthought. This is worrying, especially since the framework does not account for the user model at all. As such, I am not sure who has been helped with explanations of this form. I would like to see more discussion on what the evaluation plans are, with respect to the design of experiments that evaluate not the descriptive power but the explanatory power of the framework.

> This also raises the issue that the set of properties are inputs to the algorithm. I see this as an artifact of not having the explainee model in the framework. I think the set of plan properties would follow directly from a complete framework that accounts for the explainee, either from their mental model of the task or from the questions asked. Interestingly, recent work [Sreedharan et al. IJCAI 2018] has looked at estimating this model (in a different context) and answer such why questions.

other: I am not sure that using coverage is an apples-to-apples comparison. Is there a reason to generate the explanation at planning-time? Usually, it is in response to the question (can potentially re-use search data).

other: I would strongly suggest merging (and expanding) the example at the end with the section where plan properties and entailment are introduced. It would be useful to carry this example through in following sections as well. I had a hard time going back and forth making sense of the presentation. Having a running example would really help in readability.

minor: section number depth is probably set to zero, all the section are missing.


review_title: Nice work formalizing the response to the "Why?" question to plans

final_score: rating: 4: Top 50% of accepted papers, clear accept
review_content: This paper describes a general approach to contrastive explanation using plan-property dependencies. The key concept in this paper is plan-property depenency order, in which dependencies between properties over a set of plans are related. This provides a nice conceptual framework for asking contrastive questions over different *types* of properties, rather than just e.g. actions. The idea is instantiated as goal-dependencies, which is in turn implemented as oversubscription planning and the computational cost is evaluated on a large set of IPC benmarks. This demonstrates that the basic idea is computationally expensive, but not outrageously so!

Some more detailed comments:

- I like this idea of plan-space explanations, as outlined in Definition 3 and 4. It was quite a bit of work to get my head around this though because there are a few different concepts (with new syntax) leading up to this. I only really did that work because I was a reviewer and because this is so highly relevant to my own work. I think a nicely crafted example would immediately make this idea clear; even using an abstract example.

- I found Definition 7 difficult to follow. First, it would improve readability if there was some intuition given about when this is about. But more importantly, I could not understand the formalism here: what does A \subseteq G mean when A is a set of actions and G is a set of goals; what does the generalised conjunction mean to conjoin a goal given an action; and what is the set B referring to? 

- I found the remainder of the section then a bit challenging and didn't understand parts of it because I didn't really know what PDO-GE was. I managed to recover into the compilations section.

- The illustrative example does a nice job of making the concepts in the former sections more concrete, but I did find the jump from the MUGS to the textual explanation quite large. Can you just add a brief example or language template from how to get from {2,4,5} to the text?

review_title: Provides a nice conceptual framework for contrastive explanation

Decision: Accept