Venue: ICAPS 2021 Workshop HSDIP

Reviews:

final_score: rating: Accept
review_content: The paper introduces preferred operator pruning based on novelty for satisficing classical planning by generalizing the notion of novelty to operators, giving an operator the minimum novelty score of any state that it generated in the search history.
This does an additional pruning of the set of preferred operators obtained from an arbitrary heuristic with which the approach can be combined.

The authors do a thorough evaluation of their approach, showing that the new novelty-based preferred operator pruning is indeed effective and can increase the planner performance. It also shows, though, that it needs some further investigation. The random selection of a subset of the novel operators often outperforms the selection of the top-k novel operators.

The paper is well-written and easy-to-follow. It is technically sound and the topic fits HSDIP.

- Would it make sense to define operator novelty based on the effect facts instead of using the novelty of the generated states?

- The values chosen for b and k are not well motivated. In how many instances is there a difference at all when comparing k=100 and k=1000? Does moving to larger (absolute) values of b change the results? Does it make sense to chose b based on the number of variables?

- Below Def2: ".. of a state where the operator was previously applied during search"
Shouldn't it be the states that were *generated by* the operator?


Minor:
- Above Def.1: "search history that o leadS to"
- Def4: "..is the SET OF k-top b-novel.."
- Evaluation: "number of evaluations was above 100 are emphasizeD"
- "While there is now AN even larger portion.."
review_title: The paper generalizes the concept of novelty to operators and gives a thorough experimental analysis

final_score: rating: Accept
review_content: The paper introduces a method to prune preferred operators based on a novelty criterion similar to the heuristic novelty introduced by Katz et al. (2017).
The authors introduce several definitions of novel operators and evaluate their effectiveness on the IPC benchmarks.
Compared to a baseline using all preferred operators, using only the novel ones shows consistent improvements across most domains.

In my opinion, this is an interesting paper and the topic is a perfect fit for HSDIP.
The paper is well-written, and the contributions are well-motivated in theory and extensively evaluated in practice.
I have a few comments and questions below.

1. In the introduction, you write:
  > "The aforementioned recent partial delete relaxation heuristics, [...] when using preferred operators, use those of FF."

  I am not sure about red-black heuristics, but at least for conjunction-based methods, this statement seems incorrect.
  The preferred operators are extracted following the same methodology as that of FF (taking all applicable operators from the generated plan), but since the partially relaxed plan potentially differs from the fully relaxed plan of FF, so do the extracted preferred operators.

2. In the evaluation for $k$-top operator novelty you use the values $k = \{10, 100, 1000\}$. These values seem very large to me, as I would not expect any state to have that many successors through preferred operators (in most standard planning domains at least). Did you have some statistics on the average number of preferred operators across these domains to come up with these bounds? Have you tried smaller numbers for $k$ as well (e.g., in the range of 1 to 10)?

Minor comments:
- I would suggest using $\bot$ instead of $\emptyset$ for the initial state in the search history notatation (since $\emptyset$ is not an operator).
- The verbal description of the operator novelty (the sentence starting with "In words, ..." following Definition 2) suggests that it considers states in which this operator was applied, but I think it should be states that were reached through this operator.

review_title: Review

Meta-review: The reviewers agreed that the paper should be accepted and mentioned that there are certain points that should be clarified. We ask the authors to do their best to address these comments in the final version.
Decision: Accept