Venue: ICAPS 2020 Workshop HSDIP

Reviews:

final_score: rating: null
review_content: While I titled my first review "initial assessment", I realize that it already includes everything I had to comment on after reading the paper. Since there was no reaction or a revised submission, I also don't have anything further to add.
review_title: No further comments

final_score: rating: 7: Good paper, accept
review_content: This paper addresses describes an algorithm to generate a planning task with a causal graph that matches a given graph.

The paper is well written and easy to follow and tackles a problem interesting to the planning community. I think it fits HSDIP even though it is not directly related to search or heuristic questions.

Causal graph structures:
The text lists 12 of them, but only explains 7. I would expect that all 12 are explained, even if it is of the form "directed bipartite graph is a special case of the pipartite graph". Also, please make the order of presentation consistent at all 3 places: in the first list in the text, in the detailed list of definitions later in the text, and in Figure 1 (which could/should include more examples).

Translation from SAS+ to PDDL:
Given that any PDDL task can be grounded in many ways (depending on the type of reachability analysis typically performed), I think the inverse translation is non-trivial. At least to me, it is, and the paper does not explain how to derive predicates and action schemata from ground STRIPS representations which only operate on binary variables with no further semantics. What also worries me is that, as the paper states, the translation from PDDL back to SAS+ is far from being unique, and thus resulting planning tasks could have a very different causal structure than the one originally intended.

Algorithm 1:
While fairly easy to follow, I'd suggest to walk the reader through the algorithm while explaining the construction in the text. In the current form, matching pseudocode to text was not always easy.

Construction:
- it was first unclear to me why it is important that SAS+ facts have a meaning, until reading at the end that the task it translated back to PDDL
- prevail conditions vs preconditions: not defined
- why is it guaranteed that an action compatbile with (I)-(III) exists/can be defined?
- how is m chosen? Why is m=0 possible? Why, if m=0, it is guaranteed that all missing edges of E can still be covered through adding new actions (probably the same question as the previous one)?

Evaluation:
- picking two planners seems to be not enough for judging the quality of results: I would suggest to run at least a few IPC 2018 planners (due to being containerized, this should be possible fairly easily)
- what worries me is the generation of unsolvable tasks. I see that there is probably no easy way of preventing this, but this really lowers the quality of the resulting benchmark set. Please report the number of unsolvable tasks (as far as it is possible to know, using other techniques suitable for detecting unsolvability) and the part of it detected by the evaluated planners. Also report the number of tasks not solved by the planners.
review_title: Initial Assessment

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper introduces an algorithm for creating a planning task whose causal graph is isomorphic to a given input graph. It evaluates the method on several types of graphs and two prominent optimal classical planners. While the approach and the analysis could be improved and extended, the idea general approach is certainly interesting. I think the paper is a perfect fit for HSDIP.

Questions:

"[...] various planners performance heavily relies on the various characteristics of the input planning task." Which characteristics (apart from the causal graph structure) do you have in mind?

The paper cites various articles that relate problem hardness to causal graph structure. Wouldn't it be interesting to use your algorithm for confirming the findings experimentally?

I was surprised to read that you generate actions and discard them if they induce unwanted edges in the causal graph. Why can't you just generate "correct" actions?

Theorem 1: is the algorithm still polynomial even though it can be unlucky and generate unwanted actions infinitely often?

Why do you convert the SAS+ formulation to PDDL and then let the two planners convert it back? If the conversion can "destroy" the causal graph structure, why try to obtain the structure in the first place?

Traditionally, AI planning has focused on solvable instances. Do you have an idea on how to guarantee solvability of the generated instances?


Comments:

If I didn't overlook something, the paper leaves open a number of questions: e.g., how do you choose the domain sizes? How do you choose the number of new facts per layer? I think you need to present the algorithm with greater detail, including the variants for all decisions, and possibly make space for this by removing some variants and focusing on the method's core.

The paper uses only two planners to evaluate the new benchmarks. If you are preparing to submit the paper to a conference, I think that the evaluation has to greatly expanded. E.g., you will need to include more than two planners. I recommend using planners from IPC 2018 or planners that build on IPC 2018 planners. This should allow you to make an unbiased selection of state-of-the-art planners (e.g., LM-Cut has been ousted as state-of-the-art by abstraction heuristics a while ago).

It seems overkill to generate 512 instances per class. 30 instances should be enough to show difference between planners.


Minor comments:

"R^{0+}" -> R^+_0

"value accumulating semantic++s++"

[0.1, 0.25, 0.5, 0.75] -> Why not uniform?

"unsolvability is recognize++d++"

"unsolvability is recognized [...] by SymBA* in all 507 cases" -> Do you really know that none of the unsolved tasks is unsolvable?

The text states in twice (in two sentences) that there are tasks unsolved by any of the two planners. You can remove one sentence.

review_title: Interesting line of work. Evaluation could be improved, but good fit for HSDIP.

Decision: Accept