Venue: ICAPS 2021 Workshop XAIP

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ## Goal of the paper

The paper addresses an important problem of verbalizing plans generated by automated planners such that it is more accessible to non-expert users.

## Significance to the area

It is indeed relevant to the XAIP community as it contributes to making plans comprehensible to experts as well as lay users in natural language.

## Summary

The work proposes the PlanVerb framework which allows for verbalizing plan outputs from automated planners in natural language, while permitting the end user to exert control over various dimensions of verbalization like abstraction, locality, specificity and justifications. The paper, in my opinion, has the following contributions : 

1.  Extension of Verbalization space (Rosenthal et al 2016) to Task Space. They support more domains (other than navigation) across PDDL and RDDL. (Domain - Independent method).

2.  Addition of another dimension : Explanations : They introduce immediate, deferred justifications, and goal explanations, to incorporate causal links between actions in the verbalization. 

3.  Plan Compression : Under "Summary" value of Specificity dimension, plan compressions essentially allows for shorter verbalization by joining descriptions of actions and or subjects. (For example instead of saying Robot R1 went to A. Robot R2 went to A. Plan compression may produce : Robot R1 and R2 went to A.)

The authors also make use of mlconjug3 (Diao 2020) to assign appropriate tense to the verbs.

The key ingredient required to make the verbalization possible is semantic annotation of each action to specify subject, verb, preposition etc. Further synonyms can also be provided to the given verbs along with flags which make those mandatory to be included in the verbalization. This is assumed to be manually input to the system.

Finally, the authors have shown a well formed user study to substantiate that Plan Compression and addition of explanatory utterances and justifications is very helpful. The results of study seems sound and helps in justifying their claims.

## Comments:

Although I'm not an expert in the area, but the authors have have done a good job at covering the related works. However, I would have liked a longer discussion on comparing to works using casual links for verbalization/explanations, specially because a large contribution (and the corresponding study) covers immediate and deferred justifications (which are similar to distal explanations, which again is cited but has not been discussed at length). Another interesting discussion could have been in comparing verbalization with "drill down" operations in Big Data which essentially allows business facing users to expand on and abstract away along various dimensions.

The discussions on verbalization space and individual components of each dimension like levels in abstraction etc are well explained, however they lack accompanying examples (There are a few at the end, but a long running example could have helped understand how each level can affect the verbalization - Question on this at the end).

The PlanVerb algorithm (Algorithm 1) helps understand the flow of how final action selection will be made, which would be further verbalized. The Proposed Work is clearly stated and was mostly easy to understand.

Although I'm not familiar with all the recent works, in my opinion, the extension that this work proposes is very useful since it allows verbalization beyond robot navigation. They have extended all the dimensions - Abstraction, Specificity and Locality from the original work of Rosenthal et al 2016 to make it domain independent. They proposed the additional dimension of Explanations which indeed makes a lot of sense and they have found it to be useful in their studies as well. Finally they have put in efforts to make the final sentence to be as "natural" by leveraging mlconjug3.

## Questions/Concerns
Overall I liked the paper & and its contributions, however I do have the following questions/concerns to the authors & hopefully answers to these will help future readers/authors.

1.  Although as a preliminary work it makes sense to assume manual work in obtaining semantic tags, in the long term, I believe the domain designers should be made separate from experts providing the semantic tags. This is because the tags are being used as it is and delivered to the users by filling up existing templates. Hence, many times terminologies which are easy to understand for domain experts might not be the best choice of terms for lay users.

2.  Authors mention for Level 4 under abstraction dimension of the Verbalization Space that "only the essential parameters of the actions are verbalized". It was unclear to me what these essential parameters are and could be chosen. 

3.  Maybe its straightforward, but it seems the description on "goal-achieving explanations" is missing from Explanation subsection under Verbalization Space. Similarly it is used for Level 4 for Explanations, but I had to assume its meaning from the examples later on.

4.  Authors say "Compression method starts at the beginning of the plan and goes through every pair of actions", it should be consecutive actions. 

5.  In algorithm 1, function role of FILTERGOALS is unclear to me. 

6.  Example 1 : Abstraction uses the following setting (a,l,s,e) = (L3, All Plan, Summary, L4). Since Explanation is set to Level 4, goals should also have been conveyed in the verbalization. But I'm confused if conveying the goals is optional or mandatory under Level 4. If it is optional, how is it decided.

7.  Again, for Level 3 : Abstraction, (take Example 1 which uses L3 abstraction & presents another example with L4 abstraction), the authors say that example with L4 ignores location clause. On what basis did PlanVerb decide to keep details like ask the manager but drop "at the kitchen door", that is, how did it make sure that person of interest is kept while location is secondary. If infact there is a precedence for selection of the parameters across abstraction levels, doesn't this become domain dependent? 

8.  Authors seem satisfied with the support for per-subject plan verbalization from their user study (".. supports the idea of joining plans by subject..") but it seems that the choice of which one should be better is completely dependent on the kind of domain. If the interactions between subjects is very often then it makes sense that per-subject verbalization is ignored, whereas if the actions are mostly independent like robot navigation, then yes, per-subject verbalization could be useful. 

9.  Under PlanVerb Algorithm section the authors say that "the justifications are also considered on a per-subject basis". This means only the actions for each subject are compressed & justifications for each subject is computed. Maybe I'm wrong, but this should mean that justifications across subjects should have been ignored and never appear in the verbalizations. However, Example 1 is "Tomo will locate the manager, which will allow me ..." where the causal link between two subjects is part of the verbalization. This shouldn't be possible unless I'm missing something.

10. Maybe beyond the scope of this work, but it seems the verbalizations are abstracting away the order in which actions should take place across different subjects. Example 2 : Detailed Narrative specificity : takes about Rover3 going to waypoint 7 and rover 2 going to waypoint 7. What if the plan requires that the order is strictly rover 3 then rover 2. Would that still be captured by PlanVerb?

I hope my comments are useful.
review_title: A Useful extension of plan verbalization to Task Plans.

final_score: rating: 7: Good paper, accept
review_content: ## Paper Summary

The paper proposes a domain-independent plan verbalization approach called PlanVerb. The proposed approach is capable of generating natural language descriptions of plans, including explanations of causality between actions of a plan. The authors extend the verbalization space of previous work, and apply it for generating different types of plan descriptions according to user's preferences. PlanVerb can deal with both PDDL and RDDL domains. An evaluation with 42 participants shows that PlanVerb provides understandable plan verbalizations.

## Review
   
The paper is overall very well written, organized, and easy to follow. I think that the authors could try to formalize the Verbalization Space in a more formal way and try to improve the explanation about Algorithm 1 (perhaps add a workflow chart), but apart from that, I really like the paper and the idea of plan verbalization.

Please, consider the following minor points and comments:

### Minor Points and Comments:

1. The authors could have explained the main differences between PDDL and RDDL. Some readers are not familiar with RDDL and its formalism. Please, consider adding a comment about the differences and similarities between PDDL and RDDL.

2. The authors say that PlanVerb does not need pre-writing sentence templates, only tagging the syntactic elements of the actions in the domain model. However, the semantic domain information tagging actually defines some sort of template in the action description (commenting format) to generate sentences. Please, clarify this.

3. As I mentioned before, it would be interesting to see a formal definition of the Verbalization Space. Some symbols are introduced in The PlanVerb Algorithm section without much explanation. Please, try to formalize the Verbalization Space before introducing Algorithm 1.

4. In the PlanVerb Algorithm section, the explanation and notation in the first paragraph seem a bit out of context with respect to Algorithm 1. The notation introduced in that paragraph is not used in Algorithm 1 or in any other part of the paper. Please, check whether such notation is really needed in that section, and if not, please consider removing it.

5. Algorithm 1 (The PlanVerb Algorithm) is somehow useful to understand the plan verbalization process, but it would be good to see it as a workflow chart, illustrating the verbalization process and etc. 

6. I have some questions and comments about your evaluation, as follows.
	- 6.1) You say: "... we have used P = 0.95 as the confidence level.", what does it mean?
	- 6.2) Please, clarify what \tilde{x} means.
	- 6.3) Please, explain the tests you used in your evaluation. You mention t-test and etc, but you don't say what they actually do. 

7. I could not find the supplementary material in the paper submission. 

8. Typos and writing issues:
	- a quick idea of what does each action represent. -> a quick idea of what each action represents.
	- Includes of all the deferred justifications -> Includes all the deferred justifications;
	- different combination of parameters ->  different combinations of parameters;
	- we wanted so see -> we wanted to see;
	- Please, revise this sentence: "For each question, there was between a 20% and a 40% ...";
	- with the highest question being the 97.62% -> with the highest question being 97.62%;
	- More than the 80% -> More than 80%.

## Final Remarks
To conclude, in my opinion, I think that this paper is relevant to the workshop and should be accepted.
Please, try to address the minor points and comments that I pointed out above in the next revision/version of the paper. 
review_title: Interesting approach for plan verbalization

Meta-review: metareview: The paper looks at the problem-generating domain-independent plan verbalizations and builds on previous efforts in this direction in the domain robot path planning. The method also provides the user a certain level of control in the kind of verbalizations that can be generated through updating various parameters (again similar to earlier works). Both reviewers seem to agree that the paper is well suited to the workshop, so I would recommend accepting the paper. 

In terms of comments, the main concerns raised by the reviewers seem to be about the clarity of the paper, with both reviewers pointing at various information missing from the text, including information about the algorithms, notational issues, and the need to include more intuitive descriptions of statistical results and tests. The authors have also pointed out issues, like how the precedence between details to be abstracted are identified and how separating descriptions per agent would affect cases where there are close interactions between the agents.

I would definitely recommend the authors to not only address these comments in future drafts of the paper, but also to respond in the openreview.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept