Venue: ICAPS 2020 Workshop HSDIP

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes a scheme for selecting sequences of parameters for
planning problem instance generation, within a specific domain, aimed
at creating a set of instances that better measure the scaling
performance of planning systems.

I see some problems with this proposal, or the way it is presented in
the paper. First, it does not make any consideration for the
variability in instance hardness with equal parameters, which is a
known phenomenon in many domains. TSP-like domains, like Logistics,
Satellite and Rovers, for example often exhibit this characteristic.
The hardness of these problems depends as much on the specific
arrangement of objects in the initial state, roadmap connectivity,
location of goals, etc, as on the number of such, so that the
variation in hardness of different instances that have the same "size"
can be much greater than that resulting from increasing size. The
authors suggest treating the random seed, which implicitly determines
the specific arrangement of an instance for given size parameters, as
an enumerated parameter, and thus keeping it fixed during parameter
sequence generation. However, they also state that the final instance
set "will be generated with a different random seed than the one used
during our optimization process", which implies that there may be no
relation between the hardness of a parameter setting observed during
sequence construction and that of the corresponding instance in the
final set.

The assumption that increasing linear parameters results in harder
problems, even "typically", is also somewhat problematic, since it
ignores relationships between parameters and "phase transition"-like
effects. For example, consider a simple domain that involves
navigating a maze with obstacles, and a parameter that determines the
density of obstacles. At low densities, problems can be easy because
relaxations that ignore obstacles are accurate; at high densities,
problems are easy to solve, or prove unsolvable, because the reachable
state space is very small. It is somewhere in the middle of the range
that problems are the hardest. Note that "density", in this example,
could be defined implicitly, for instance with separate parameters for
maze size and number of obstacles. While this is a made-up example,
examples that have a similar behaviour appear in some IPC domains, for
example the number of instruments in the Satellite domain.

Second, the reliance on a selection of "state of the art" planners in
shaping the benchmark set can also be problematic, because scaling in
different parameters may affect different planners in different ways.
An example of this can be seen by comparing the behaviour of planners
on the IPC 1998 and 2000 Logistics instance sets. The IPC 2000 set
scales up problems mainly by increasing the number of goal (packages);
the number of cities and airplanes grows very little (from 3 to 5, and
1 to 2, respectively). The IPC 1998 set, on the other hand, scales up
the number of all types of objects in the instances. Experimental
comparisons that separate these two instance sets sometimes have shown
a difference in how planners perform on them. If the reference set
includes only planners that, for example, scale the same with
increasing numbers of objects of any type, the resulting benchmark set
will not be able to exhibit the performance of a planner that scales
much better in the number of goals while other parameters are limited
to small values. Another example can be found in the results of the
2016 unsolvability competition, which showed that some planners
although not performing well on average across the set of domains used
in the competition, did very well, and better than the overall
top-performers, in certain domains.

I do not think there is a single best way to select a set of benchmark
problems, except perhaps if you first formulate the hypothesis under
test and select according to that, but in that case the planners or
algorithms would be known, the instances could be tailored to their
capability, and the choice of domains as important as the choice of
instances. Selecting an instance set "blindly" is perhaps a problem
for organisers of the IPC, but even then it does not need to be
suitable for more than the current state of the art, and we know not
to over-interpret the results. If the authors want to propose a method
for selecting benchmark problems for experiments, they should first
clearly delineate the purpose of the experiment that the selected
benchmarks are to be used for - what is the scientific question that
it will answer? - and secondly demonstrate the potential failure modes
of the selection method against that purpose.

review_title: Review (revised paper version)

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper addresses the automated generation of problem instances. The aim is to obtain a benchmark set consisting of instances that make an empirical differentiation of different planners possible. For this purpose, the problem of creating a set of instances is modeled as an optimization problem.

The topic of the paper fits the workshop, since one of the typical topics of the HSDIP workshop is the investigation of “challenging domains for existing combinations of heuristics and search algorithms”.

Overall, I like the paper and I believe that an automated procedure to obtain a benchmark set that is well suited to measure different planners is an interesting and totally relevant topic. While reading the paper, some remarks and questions came to my mind:

1) The sentence “Evaluating planners on different benchmark sets may produce different results, leading to different conclusions about the evaluation”  reminded me of a work by Riddle et al. (2011) which showed that the ranking of the International Planning Competition could be influenced if the representation of a single domain (Blocksworld) is different.

2) Some machine learning techniques require a large amount of data for training. I would be interested to know if this automated technique also makes it possible and is well suited to create a large amount of instances for a domain, i.e. not just 30, but hundreds or thousands?

3) In the conclusion, it is noted that the proposed approach is orthogonal to any approach that can generate new instances. I am not sure if this means that the approach is not compatible with the construction of random instances? For example, if we consider different models for the construction of random instances proposed by Rintanen (2004), it seems that such models also have several parameters that could be optimized in a similar way as the IPC domains.

Minor comments:
- Introduction: aggregated over*y* different 
- Sequence Optimization: in Section _ than those
- Sequence Selection: principles of Section _.

References:
- P. J. Riddle, R. C. Holte, M. W. Barley: Does Representation Matter in the Planning Competition? SARA 2011
- J. Rintanen: Phase Transitions in Classical Planning: An Experimental Study. ICAPS 2004: 101-110

review_title: Early assessment

Decision: Accept