Venue: ICAPS 2022 Workshop XAIP

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents JEDAI, a framework for robot tasks and motion planning in simulation with an NLP interface for users. Users can suggest a plan for their robot with a simple drag-and-drop workspace built with the Blockly visual programming library. In case that plan is failing, the user gets an explanation of the reason. Explanations in the context of this work are of two types: non-achieved goal conditions and violation of a precondition of an action. 

The paper is in the workshop's scope, and it is clearly written and easy to follow.

The related work section can be explained and written in more detail. I suggest authors check previous XAIP workshop publications and further works of authors who published. There are several essential works presenting frameworks for explainable planning which are related. 

The system seems simple to use, and I am looking forward to user study outcomes, which should be the next step for this work. User satisfaction and explanation ability of the system should be evaluated.

In future work, I suggest defining a complete set of plan failures and covering them, as well as testing outliers.

Additionally, I suggest authors test framework robustness for domain and plan complexities. 
review_title: Recommendation for acceptance. The paper presents an explainable robot planning demo system

final_score: rating: 7: Good paper, accept
review_content: # JEDAI: A system for Skill-Aligned Explainable Robot Planning

The paper starts with the problem of more AIs in everyday life, with a general population that may not understand their limitations.
Without a clear understanding of such limitations they may create dangerous situations.
A system called JEDAI is used to fill the gap between task description and error explanation to the user.
The user uses a drag-and-drop environment to avoid mistakes in high-level programming, and presented with simulation and natural language explanations tailored to the user level.

Prior work focuses on separate elements: visualizations, non-expert programming environments and explanations.
JEDAI uses a more specific approach, focusing on real robots and combining the previous elements in the same interface.
It is made of a Blockly UI, HELM explanations combined with template NLP, and ATM-MDP planner with OpenRAVE.

Five custom domains have been implemented focusing on selected robots.
The end result is a system that shows how to integrate AIs with robots while using predefined abstractions to verify plans.

## Comments
- The paper is well written, but lacks user feedback about how well everything was integrated compared to what is commonly used today for such robots.
It would be interesting to see both if a new user would feel confortable using it, and how complex it would be for an experienced user to add a new robot to it.
This is the most important point: does this tool work for the intended user?

- I would rephrase "we present an AI system JEDAI (...)" to "we present JEDAI (...), an AI system".

- The configurable level of knowledge of the user is not clear, it gives more explanation details for an experienced user?
Such details could also be valuable for a new user, perhaps the new user just needs a simpler description before asking for more details, but this solution should work for everyone.
I only ask this based on experience, many robot projects are programmed by groups of people and they all have different levels of knowledge about each part of the robot.

- Figure 2 lacks the domain and task provided by the user, based on the text description.
I find it odd that "personalized explanation generator" returns a validated high-level plan, it seems that this block is HELM, which both validates and explains.
If this is correct, it validates only high-level plans, is the low-level plan always valid if the high-level one is?

- In Section 3.3 and Conclusion the authors describe hand-written templates for action description, but want to automate this process.
I would ask why not let the user control/provide such templates instead of automating it?
Perhaps their robot has been customized in such a way that a new description is required.

- In Section 3.4 the authors say that failures are reported back, but valid plans are simulated.
Why this work differently for each case, reporting success/failure and simulation would be better to show where/when/how it fails.
Not sure if this is a limitation of the simulator, but it could simulate all previous valid steps until the failure to help the user see the intermediate states/configurations of the robot.
review_title: Review JEDAI

Meta-review: metareview: The paper presents an explainable (robot) planning tool that assists non-experts users in creating plans as well explaining plan errors using natural language. The paper would be a great addition to the program. 

For future work, as suggested by both reviewers, we urge the authors to consider evaluating their system in human user studies. This is of extreme importance as the authors frame the system as an assistive educational tool, and thus, the system’s user satisfaction and explanatory ability should be tested and evaluated properly.

We are looking forward to your presentation.

confidence: 5: The area chair is absolutely certain
Decision: Accept