Venue: ICAPS 2020 Workshop HSDIP

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper builds on a recent line of research on using machine learning for portfolio planner selection. In contrast to prior research that used images and graph representations of the planning problem as input to the neural network, the approach proposed here does feature engineering, and extracts a number of features from the planning graph. These features are fed to different classifiers to be trained on: linear regression, random forests, and multilayer perceptron.

The motivation for the need for good portfolio planners is well articulated. I found that the paper does a good job at presenting the work, particularly to these readers that may be familiar with the planning literature, but not so much with the machine learning literature. I liked the extensive experimental section and the different experiments and tests that were performed and evaluated, and I also liked how the results were interpreted.

The topic of the paper is in the scope of the workshop, and in my opinion the quality of the submitted work deserves being presented at HSDIP. 

There are some typos and some parts of the text that need to be rephrased, but in general it is well written. Perhaps for completeness, I recommend the authors to include a sentence of two to explain what is the approach that you follow from Sievers et al. (2019), and that you point out in the beginning of Section 3.

review_title: Early assessment

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The topic of this paper is the selection of algorithms on the basis of features obtained from the problem description graph (abstract structure graph) of a given planning task. For this task, (graph) convolutional neural networks were used in the past. This technique has shown good results, but is difficult to interpret. In other words, it is difficult to infer what a neural network has learned, which is an interesting question to ask in order to infer rules or heuristics when to use a particular planning algorithm. The paper proposes to analyze handcrafted graph features in combination with "traditional" machine learning algorithms. The objective is to determine whether it is possible to achieve the same performance with these features and to perform an importance analysis of the graph features.

The topic of the paper fits to the workshop as the HSDIP workshop explicitly welcomes multidisciplinary work such as the application of machine learning in heuristic search (here: algorithm/heuristic selection). 

I agree with the statement that neural networks are complex function approximators, so it is difficult to understand what features have been learned. However, I think this statement would benefit from a citation.The question of whether interpretable machine learning techniques based on a similar input can achieve the same performance as neural networks is certainly interesting and can provide some insight why the PDG and ASG of a planning task seem to be a good tool to determine which planner to use. But, I am not really sure if the graph features used for the elementary techniques of machine learning can really be learned from a (graph) convolutional neural networks. As in the paper mentioned, a lot of information is lost when the graph is converted to a 128x128 pixel image. My first question is whether information is also lost when the graph is used directly as input to the neural network? In cases where information is lost, one can certainly infer whether the graph is dense or not, but I wonder if features such as radius, diameter or degrees are preserved.  This would indicate that the neural network has no chance to use information used by elementary machine learning?

Regarding the planners used, it is not clear whether seq-opt-symba-1 also uses h2 mutexes. To the best of my knowledge, h2 mutexes are used by default in seq-opt-symba-1. It would be good to mention this.

The sentence: "Like Ma et al. (2020), we select the 145 solvable tasks from the IPC 2018 ..." is confusing. I assume that the tasks that were solved by one of the 17 considered planners within the given time and memory constraints are referred to or do only 145 instances from the IPC 2018 have a solution?

Minor comments:

Abstract:
- convolutional resp. graph convolutional neural networks (abbrev. in abstract) -> respectively? -> (graph) convolutional neural networks

Introduction:
 - of the graph as image *image*

Background:
 - eff -> mathit ($\mathit{eff}$)

Experiments:
 - Instead, *for* we sum up for every feature
 - a feature can be used 170 -> a feature can be used is 170
 - the linear regession are using -> is using ?

review_title: Early assessment

Decision: Accept