Title: Optimize Planning Heuristics to Rank, not to Estimate Cost-to-Goal
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning heuristic functions

deep learning

Immitation learning

planning

A*

best first search
Abstract: In imitation learning for planning, parameters of heuristic functions are optimized against a set of solved problem instances. This work revisits the necessary and sufficient conditions of strictly optimally efficient heuristics for forward search algorithms, mainly A* and greedy best-first search, which expand only states on the returned optimal path. It then proposes a family of loss functions based on ranking tailored for a given variant of the forward search algorithm. Furthermore, from a learning theory point of view, it discusses why optimizing cost-to-goal h* is unnecessarily difficult. The experimental comparison on a diverse set of problems unequivocally supports the derived theory.

Optimize Planning Heuristics to Rank, not to Estimate Cost-to-Goal

Abstract

In imitation learning for planning, parameters of heuristic functions are optimized against a set of solved problem instances. This work revisits the necessary and sufficient conditions of strictly optimally efficient heuristics for forward search algorithms, mainly A* and greedy best-first search, which expand only states on the returned optimal path. It then proposes a family of loss functions based on ranking tailored for a given variant of the forward search algorithm. Furthermore, from a learning theory point of view, it discusses why optimizing cost-to-goal h ∗ is unnecessarily difficult. The experimental comparison on a diverse set of problems unequivocally supports the derived theory.

1 Introduction

Automated planning finds a sequence of actions that will reach a goal in a model of the environment provided by the user. It is considered to be one of the core problems in Artificial Intelligence and it is behind some of its successful applications [39, 27, 46]. Early analysis of planning tasks [31] indicated that optimizing the heuristic function steering the search for a given problem domain towards the goal can dramatically improve the performance of the search. Automated optimization of the heuristic function therefore becomes a central request in improving the performance of planners [6, 13].

Learning in planning means optimizing heuristic functions from plans of already solved problems and their instances. This definition includes the selection of proper heuristics in a set of pattern databases [17, 22, 33, 12], a selection of a planner from a portfolio [25], learning planning operators from instances [32, 56], learning for macro-operators and entanglements [9, 29], and learning heuristic functions by general function approximators (e.g. neural networks) [45, 19, 16, 5].

The majority of research [45, 53, 19, 16] optimizes the heuristic function to estimate the cost-to-goal, as it is well known that it is an optimal heuristic for many best-first heuristic search algorithms including A* or IDA* [21, 28]. These works optimize the heuristic function to solve regression problems . But even a true cost-to-goal h ∗ does not guarantee that the forward search will find the optimal solution while expanding the minimal number of states. This paper defines a stricter version of optimally efficiency [10, 24] as follows. Forward search (or its heuristic) is called strictly optimally efficient iff it expands only states on one optimal solution path returned by the search, i.e., if this optimal solution path has l +1 states, the search will expand only l states.

This work focuses exclusively on a forward search with merit functions. It presents theoretical arguments as to why the order (rank) of states provided by the heuristic function is more important

than the precise estimate of the cost-to-goal and it formalizes the necessary and sufficient condition for strictly optimally efficient search. It also argues why learning to rank is a simpler problem than regression estimating the cost-to-goal from a statistical learning theory point of view. The main motivation of the paper is similar to that in [38], but the derived theory and implementation are much simpler. The sufficiency of optimizing the rank has been already recognized in [59] for a beam-search and in [18] for greedy best-first search (GBFS), though, unlike this work, it cannot guarantee strict optimal efficiency.

We emphasize that this work neither deals with nor questions the existence of a strictly optimally efficient heuristic function with the forward search. It is assumed that the space of possible heuristic functions is sufficiently large, such that there exist one that is sufficiently good. The theoretical results are supported by experimental comparisons to the prior art on eight domains.

The contributions of this paper are as follows.

We state necessary and sufficient conditions for strictly optimally efficient heuristic functions for forward search algorithms in deterministic planning.

We show that when optimizing a heuristic function, it is better to solve the ranking problem.

We argue, why learning to rank is easier than regression used in learning the cost-to-goal.

We instantiate it for A* and GBFS searches leading to two slightly different loss functions.

Weexperimentally demonstrate on eight problems (three grid and five PDDL) that optimizing rank is always better.

2 Preliminaries

A search problem instance is defined by a directed weighted graph Γ = ⟨S , E , w ⟩ , a distinct node s 0 ∈ S and a distinct set of nodes S ∗ ⊆ S . The nodes S denote all possible states s ∈ S of the underlying transition system representing the graph. The set of edges E contains all possible transitions e ∈ E between the states in the form e = ( s, s ′ ) . s 0 ∈ S is the initial state of the problem instance and S ∗ ⊆ S is a set of allowed goal states. Problem instance graph weights (alias action costs) are mappings w : E → R ≥ 0 .

A path (alias a plan) π = ( e 1 , e 2 , . . . , e l ) , e i = ( s i -1 , s i ) ∈ E , of length l solves a task Γ with s 0 and S ∗ iff π = (( s 0 , s 1 ) , ( s 1 , s 2 ) , . . . , ( s l -1 , s l )) and s l ∈ S ∗ . An optimal path π ∗ is defined as a path minimizing the cost of a problem instance Γ , s 0 , S ∗ , its value as f ∗ = w ( π ∗ ) = ∑ l i =1 w ( e i ) . When cost function w ( e ) = 1 optimal path corresponds to the shortest one. π : i = (( s 0 , s 1 ) , ( s 1 , s 2 ) , . . . , ( s i -1 , s i )) denotes subplan of a plan π containing its first i actions. S π = { s i } l i =0 denotes the set of states from the path π = (( s 0 , s 1 ) , ( s 1 , s 2 ) , . . . , ( s l -1 , s l )) , and correspondingly S π : i denotes the set of states in a subplan π : i .

2.1 Forward search algorithm

To align notation, we briefly review the forward search algorithm, of which A* and GBFS are special cases. As we apply full duplicate detection in all of our algorithms, there is a bijection of search tree nodes to planning states. Forward Search for consistent heuristics, where h ( s ) -h ( s ′ ) ≤ w ( s, s ′ ) for all edges ( s, s ′ ) in the w -weighted state space graph, mimics the working of Dijkstra's shortest-path algorithm [11]. It maintains two sets: the first called Open list , O , contains generated but not expanded nodes; the second called Closed list , C , contains already expanded nodes. Parameters α and β of a merit function f ( s ) = αg ( s ) + βh ( s ) used to sort states in Open list allow to conveniently describe different variants of the algorithm as described in detail shortly below. The forward search works as follows.

(a) Select the state s = arg min αg ( s ) + βh ( s )

Initialise Open list as O 0 = { s 0 } . 2. Set g ( s 0 ) = 0 3. Initiate the Closed list to empty, i.e. C 0 = ∅ . 4. For i ∈ 1 , . . . until O i = ∅ i s ∈O i -1

(b) Remove s i from O i -1 , O i = O i -1 \ { s i }

(c) If s i ∈ S ∗ , i.e. it is a goal state, go to 5.

(d) Insert the state s i to C i -1 , C i = C i -1 ∪ { s i }

(e) Expand the state s i into states s ′ for which hold ( s i , s ′ ) ∈ E and for each

set g ( s ′ ) = g ( s i ) + w ( s i , s ′ )

if s ′ is in the Closed list as s c and g ( s ′ ) < g ( s c ) then s c is reopened (i.e., moved from the Closed to the Open list), else continue with (e)

if s ′ is in the Open list as s o and g ( s ′ ) < g ( s o ) then s o is updated (i.e., removed from the Open list and re-added in the next step with updated g ( · ) ), else continue with (e)

add s ′ into the Open list

Walk back to retrieve the solution path.

In the above algorithm, g ( s ) denotes a function assigning an accumulated cost w for moving from the initial state ( s 0 ) to a given state s . During its execution, it always expands nodes with the lowest f ( s ) = αg ( s ) + βh ( s ) . Different settings of α and β give rise to different algorithms: for A*, α = β = 1 ; for GBFS α = 0 and β = 1 .

Consistent heuristics , which are of special interest, are called monotone because the estimated cost of a partial solution f ( s ) = g ( s ) + h ( s ) is monotonically non-decreasing along the best path to the goal. More than this, f is monotone on all edges ( s, s ′ ) , if and only if h is consistent as we have f ( s ′ ) = g ( s ′ ) + h ( s ′ ) ≥ g ( s ) + w ( s, s ′ ) + h ( s ) -w ( s, s ′ ) = f ( s ) . For the case of consistent heuristics, no reopening (moving back nodes from Closed to Open) in A* is needed, as we essentially traverse a state-space graph with edge weights w ( s, s ′ ) + h ( s ′ ) -h ( s ) ≥ 0 . For the trivial heuristic h 0 , we have h 0 ( s ) = 0 and for perfect heuristic h ∗ , we have f ( s ) = f ∗ = g ( s ) + h ∗ ( s ) for all nodes s . Both heuristics h 0 and h ∗ are consistent. For GBFS and other variants, reopening is usually neglected. Even if the heuristic is not consistent, best-first algorithms without the reopening, remain complete i.e., they find a plan if there is one. Plans might not be provably optimal but are often good for planning practice [14].

3 Conditions on strictly optimally efficient heuristic

Let O i be an Open list in the i th iteration of the forward search expanding only states on the optimal path π. Below definition defines a perfect ranking heuristic as a heuristic function where state on some optimal path in the Open list, S π : i ∩ O i , has always strictly lower merit value than other states in the Open list off the optimal path, O i \ S π : i . The intersection S π : i ∩ O i contains always exactly one state.

Definition 1 (Perfect ranking heuristic) . A heuristic function h ( s ) is a perfect ranking in forward search with a merit function f ( s ) = αg ( s ) + βh ( s ) for a problem instance Γ = ( ⟨S , E , w ⟩ , s 0 , S ∗ ) if and only if there exists an optimal plan π = (( s 0 , s 1 ) , ( s 1 , s 2 ) , . . . , ( s l -1 , s l )) such that

g ( s ) is the cost from s 0 to s in a search-tree created by expanding only states on the optimal path π ;

∀ i ∈ { 1 , . . . , l } and ∀ s j ∈ O i \ S π : i we have f ( s j ) > f ( s i ) .

As an example consider a search-tree in Figure 1a with an optimal path (( s 0 , s 1 ) , ( s 1 , s 2 ) , ( s 2 , s 3 )) and states { s 4 , s 5 , s 6 , s 7 } off the optimal path. Then the Open list after expanding s 0 is O 1 = { s 1 , s 4 , s 5 } and that after expanding s 1 is O 2 = { s 2 , s 4 , s 5 , s 6 , s 7 } . The set of states on the optimal sub-path are S π :1 = { s 0 , s 1 } , S π :2 = { s 0 , s 1 , s 2 } . States on the optimal path in the Open lists are S π :1 ∩ O 1 = { s 1 } and S π :2 ∩ O 2 = { s 2 } . States off the optimal path in the Open lists are O 1 \ S π :1 = { s 4 , s 5 } and O 2 \ S π :2 = { s 4 , s 5 , s 6 , s 7 } . Function f is perfectly ranking problem instance in Figure 1a iff following inequalities holds:

Notice that the heuristic values of states on the optimal path are never compared. This is because if the forward search expands only states on the optimal path, then its Open list always contains only

where

s

4

s

5

s

6

s

7

s

0

s

1

s

2

s

3

A

10

B

3

C

8

D

4

0

E

2

8

4

3

4

one state from the optimal path. The definition anticipates the presence of multiple optimal solutions and even multiple goals, which requires the heuristic to break ties and have a perfect rank with respect to one of them.

Theorem 1. The forward search with a merit function f ( s ) = αg ( s ) + βh ( s ) and a heuristic h is strictly optimally efficient on a problem instance ( ⟨S , E , w ⟩ , s 0 , S ∗ ) if and only if h is a perfect ranking on it.

Proof. Sufficiency: If the conditions of a perfect ranking heuristic hold, then there exists an optimal path such that a state on it that is in the Open list always has the strictly lowest heuristic value. Therefore, forward search will never move off the optimal path and is, therefore, strictly optimally efficient.

Necessity: If h is a strictly optimally efficient heuristic, then the forward search always selects the state on the optimal path, which means that the state has the lowest heuristic value of all the states in the Open list, which is precisely the condition of a perfect ranking heuristic.

Theorem 1, despite being trivial, allows certifying that the forward search with a given heuristic in a given problem instance is strictly optimally efficient. This property has been discussed in [59] for Beam search, but the conditions stated there are different because Beam search prunes states in the Open list. Recall that the complexity class of computing a perfect ranking heuristic function is the same as solving the problem because one implies the other. We now remind the reader that the popular cost-to-goal does not always lead to a strictly optimally efficient forward search.

Example 1. While cost-to-goal h ∗ is the best possible heuristic for algorithms like A* (up to tie-breaking) in terms of nodes being expanded, for GBFS, h ∗ does not necessarily yield optimal solutions.

See Figure 2 for a counterexample. The complete proof is in the Appendix. Appendix 8.4 also gives an example of a problem instance, where an optimally efficient heuristic for GBFS returning the optimal solution does not exist.

4 Loss functions

4.1 Ranking loss function

Let's now design a loss function minimizing the number of violated conditions in Definition 1 for a problem instance (Γ , s 0 , S ∗ ) and its optimal plan π. Assuming a heuristic function h ( s, θ ) with parameters θ ∈ Θ , the number of violated conditions can be counted as

O i , S π : i as used in Definition 1, J · K denotes Iverson bracket being one if the argument is true and zero otherwise, and α and β are parameters controlling the type of search, as introduced above.

In imitation learning, we assume that we have a training set T trn consisting of tuples of problem instances and optimal plans T trn = { ( ⟨S i , E i , w i ⟩ , s 0 ,i , S ∗ i , π i ) } n i =1 . To optimize parameters θ of a heuristic function h ( s, θ ) , we propose to minimize the number of wrongly ranked states over all the problem instances in the training set,

In practice, one often wants to solve problem (3) by efficient gradient optimization methods. To do so, the Iverson bracket J · K (also called 0-1 loss) is usually replaced by a convex surrogate such as the hinge-loss or the logistic loss used in the Experimental section below, in which case Equation (3) becomes

While optimization of the surrogate loss might look as if one optimizes a different problem, it has been shown that for certain classes of functions, optimization of surrogate losses solves the original problem (3) as the number of training samples approach infinity [50, 35] .

4.2 Regression loss function

For the sake of comparison, we review a loss function used in the majority of works optimizing the cost-to-goal in imitation learning for planning (some notable exceptions are in the related work section). The optimization problem with L 2 loss function is defined as

where h ∗ ( s i ) is the true cost-to-goal. The optimal solution of the optimization is the heuristic function h ∗ . What is important for the discussion below is that the L 2 loss function uses only states on the optimal path and the heuristic function is optimized to solve the regression problem. Other variants of regression loss, such as asymmetric L 1 in [51], do not solve issues of regression losses discussed in this paper.

4.3 Advantages and disadvantages of loss functions

Cost-to-goal provides a false sense of optimality It has been already mentioned above that A* with h ∗ is strictly optimally efficient up to resolving states with the same value of g ( s ) + h ∗ ( s ) . This means that for problems with a large (even exponential) number of optimal solutions of equal cost (see [23] for examples), A* will be very inefficient unless some mechanism for breaking ties is used. Moreover, since learning is inherently imprecise, a heuristic close but not equal to h ∗ will likely be less efficient than the one close to a perfect-ranking heuristic.

Size of the solution set The set of all functions satisfying Equation (5) is likely smaller than that satisfying Equation (3), because the solution set of (3) is invariant to transformations by stricly monotone functions, unlike the solution set of (5). From that, we conjecture that finding the solution of (5) is more difficult. The precise theorem is difficult to prove since the solution of (5) is not a solution of (3) due to tie resolution in Definition 1.

Optimizing cost-to-goal does not utilize states off the solution path Assume a problem instance ( ⟨S , E , w ⟩ , s 0 , S ∗ ) and an optimal solution path π = (( s 0 , s 1 ) , ( s 1 , s 2 ) , . . . , ( s l -1 , s l )) . Loss function optimized in (5) uses only states { s i } l i =0 on the optimal path. This means that even when the loss is zero, the search can be arbitrarily inefficient if states S\S π off the optimal path will have heuristic values smaller than min s ∈S π h ( s ) . The proposed ranking loss (3) uses states on and off the optimal path and therefore, does not suffer from this problem. This issue can be fixed if the training set is extended to contain heuristic values for all states off the optimal path (states { s 4 , s 5 , s 6 , s 7 } in Figure 1a), as has been suggested in [5, 49]. With respect to the above definitions, this is equal to adding more problem instances to the training set sharing the same graph Γ and set of goal states S ∗ , but differing in initial states. Solving all of them to obtain true cost-to-goal greatly increases the cost of creating the training set .

Heuristic value for dead-end states should be sufficiently high to ensure they are never selected. An infinity in the L 2 loss would always lead to an infinite gradient in optimization. In practice, the ∞ is replaced by a sufficiently large value, but too large values can cause large gradients of L 2 , which might negatively impact the stability of convergence of gradient-based optimization methods. Proposed ranking losses do not suffer this problem.

Goal-awareness The drawback of the heuristic function h ( s, θ ) obtained by optimizing ranking losses is that they are not goal-aware unlike the heuristic optimizing the estimate of cost-to-goal.

Speed of convergence against the size of training set From classical bounds on true error [54] and [7], we can derive (see Appendix for details) that the excess error of ranking loss converges to zero at a rate √ ln n p √ n p , which is slightly faster than that of regression √ ln n s √ n s - √ ln n s , where n s and n p denotes the number of states and state pairs respectively in the training set. But, the number of state pairs in the training set grows quadratically with the number of states; therefore, the excess error of ranking loss for the number of states n s can be expressed as √ 2 ln n s n s , which would be by at least 1 √ n s factor faster than that of regression. Note though that these rates are illustrative since the independency of samples (and sample pairs) is in practice, violated since even samples from the same problem instance are not independent.

Conflicts in training set Unlike regression loss, the proposed family of ranking losses is potentially sensitive to conflicts in the training set, which occur when the training set contains examples of two (or more) different solution paths of the same problem instance. This might prevent achieving a zero loss. The effect on the heuristic depends on the composition of the training set. Either solution paths more frequent in the training set will be preferred, or there will be ties, but their resolution do not affect optimally efficiency. Appendix 8.5 provides an illustrative example.

5 Related Work

The closest work to ours is [36, 38], which adjusts heuristic value by a policy estimated from the neural network. The theory therein has the same motivation, to minimize the number of expanded states, though it is more complicated than the theory presented here. For efficiency, [38] needs to modify the GBFS such that search nodes store policy and probability of reaching the node from the start. Moreover, in some variants (used in the experimental comparison below) the neural network has to estimate both heuristic value and policy.

Inspired by statistical surveys of heuristic functions in [57, 58], [18] proposes to optimize the rank of states preserving order defined by the true cost-to-goal. The formulation is therefore valid only for GBFS and not for A* search. Ties are not resolved, which means the strict optimal efficiency cannot be guaranteed. Similarly, heuristic values for domains where all actions have zero cost cannot be optimized. Lastly and importantly, the need to know the true cost-to-goal for all states in the training set greatly increases the cost of its construction as discussed above. From Bellman's equation [49] arrives at the requirement that the smallest heuristic value of child nodes has to be smaller than that of the parent node. The resulting loss is regularized with L 1 forcing the value of the heuristic to be close to cost-to-goal.

In [5], A* is viewed as a Markov Decision Process with the Q-function equal to the number of steps of A* reaching the solution. This detaches the heuristic values from the cost-to-goal cost, but it does not solve the problem with dead-ends and ties, since the optimization is done by minimizing L 2 loss.

Symmetry of L 2 (and of L 1 ) loss are discussed in [51], as it does not promote the admissibility of the heuristic. It suggests asymmetric L 1 loss with different weights on the left and right parts, but this does not completely solve the problem of estimating cost-to-goal (for example with ties, dead-ends).

Parameters of potential heuristics [41] are optimized by linear programming for each problem instance to meet admissibility constraints while helping the search. On contrary, this work focuses on the efficiency of the search for many problem instances but cannot guarantee admissibility in general.

Combining neural networks with discrete search algorithms, such that they become an inseparable part of the architecture is proposed in [55] and [60]. The gradient with respect to parameters has to be

propagated through the search algorithm which usually requires both approximation and the search algorithm to be executed during every inference.

A large body of literature [46, 20, 15, 2] improves the Monte Carlo Tree Search (MCTS), which has the advantage to suppress the noise in estimates of the solution costs. These methods proposed there are exclusive to MCTS and do not apply to variants of the forward search algorithm. Similarly, a lot of works [45, 53, 61, 19, 8, 16, 5] investigate architectures of neural networks implementing the heuristic function h ( s, θ ) , ideally for arbitrary planning problems. We use [45, 8] to implement h ( s, θ ) , but remind our readers that our interest is the loss function minimized when optimizing θ.

Finally, [1] proposes to construct a training set similarly to pattern databases [12] by random backward movements from the goal state and to adapt A* for multi-core processors by expanding k > 1 states in each iteration. These improvements are independent of those proposed here and can be combined.

6 Experiments

The experimental protocol was designed to compare the loss functions while removing all sources of variability. It therefore uses imitation learning in planning [18, 19] and leaves more interesting bootstrap protocol for future work [3]. The goal was not to deliver state-of-the-art results, therefore the classical solvers were omitted, but their results are in the Appendix for completeness.

The proposed ranking losses are instantiated with the logistic loss function as in Equation (4) for A* search by setting α = β = 1 in (2), called L ∗ , and for GBFS, by setting α = 0 , β = 1 in (2), called L gbfs . L 2 is used as defined in Equation (5). Since the heuristics used in this paper are not admissible, A* search will not provide optimal solutions. It was included in the comparison to demonstrate that heuristics optimized with respect to L gbfs might be inferior in A* search. To observe the advantage of ranking over regression, we defined a L rt loss function

comparing only states on the optimal trajectory. We also compare to the loss function proposed in [49] defined as L be ( h, Γ , π ) = ∑ s ∈S π max { 1 + min s ′ ∈N ( s ) h ( s ′ , θ ) -h ( s, θ ) , 0 } +max { 0 , h ∗ ( s ) -h ( s, θ ) } + max . { 0 , h ( s, θ ) -2 h ∗ ( s ) } , where N ( s ) denotes child-nodes of state s. Furthermore, we compare to the policy-guided heuristic search [38] (denoted as L le ) with GBFS modified for efficiency as described in the paper while simultaneously adapting the neural network to provide the policy and heuristic value.

The heuristic function h ( s, θ ) for a given problem domain was optimized on the training set containing problem instances and their solutions, obtained by SymBA* solver [52] for Sokoban and Maze with teleports and [42] 1 for Sliding Tile. Each minibatch in SGD contained states from exactly a single problem instance needed to calculate the loss. This means that for L 2 , L rt , L le , only states on the solution trajectory were included; for L ∗ , L gbfs , and L be there were additional states of distance one (measured by the number of actions) from states on the solution trajectory. Optimized heuristic functions were always evaluated within A* and GBFS forward searches, such that we can analyze how the heuristic functions specialize for a given search. To demonstrate the generality, experiments contained eight problem domains of two types (grid and general PDDL) using two different architectures of neural networks described below.

Neural network for grid domains States in Sokoban , Maze with teleports, and Sliding-puzzle can be easily represented in a grid (tensor). The neural network implementing the heuristic function h ( s, θ ) was copied from [8]. Seven convolution layers, P 1 . . . P 7 of the same shape 3 × 3 with 64 filters are followed by four CoAT blocks (CoAT block contains a convolution layer followed by a multi-head attention operation with 2 heads and a positional encoding layer). Convolution layers in CoAT blocks have size 3 × 3 and have 180 filters. The network to this stage preserves the dimension of the input. The output from the last block is flattened by mean pooling along x and y coordinates before being passed onto a fully connected layer (FC) predicting the heuristic. All blocks have skip connections.

Neural network for PDDL domains For Blocksworld , Ferry , Spanner , N-Puzzle , and Elevators , where the state cannot be easily represented as a tensor, we have used Strips-HGN [44] and we refer the reader therein for details. Strips-HGN was implemented in Julia [4] using PDDL.jl [62] extending [30]. Importantly, we have used the hyper-graph reduplication proposed in [48] to improve computational efficiency. Since we did not know the precise architecture (unlike in the grid domain), we performed a grid-search over the number of hyper-graph convolutions in { 1 , 2 , 3 } , dimensions of filters in { 4 , 8 , 16 } , and the use of residual connections between features of vertices. The representation of vertices was reduced by mean and maximum pooling and passed to a two-layer feed-forward layer with hidden dimension equal to the number of filters in hyper-graph convolutions. All non-linearities were of relu type. The best configuration was selected according to the number of solved problems in the validation set. Forward search algorithms were given 5s to solve each problem instance chosen in a manner to emphasize differences between loss functions. All experiments were repeated 3 times. All experiments on the PDDL domains including development took 17724 CPU hours. Optimizing heuristic for a single problem instance took approximately 0.5 CPU/h except Gripper domain, where the training took about 8h due to large branch factors. The rest of this subsection gives further details about domains.

Sokoban The training set for the Sokoban domain contained 20000 mazes of grid size 10 × 10 with a single agent and 3 boxes. The testing set contained 200 mazes of the same size 10 × 10 , but with 3, 4, 5, 6, 7 boxes. Assuming the complexity of to be correlated with the number of boxes, we can study generalization to more difficult problems. The problem instances were generated by [40].

Maze with teleports The training set contained 20000 mazes of size 15 × 15 with the agent in the upper-left corner and the goal in the lower-right corner. The testing set contained 200 mazes of size 50 × 50 , 55 × 55 , and 60 × 60 . Therefore, as in the case of Sokoban, the testing set was constructed such that it evaluates the generalization of the heuristic to bigger mazes. The mazes were generated using an open-source maze generator [47], where walls were randomly broken and 4 pairs of teleports were randomly added to the maze structure.

Sliding puzzle The training set contained 20000 puzzles of size 5 × 5 . The testing set contained 200 mazes of size 5 × 5 . These puzzles were taken from [37]. Furthermore, we tested our approach on puzzles of higher dimensions such 6 × 6 and 7 × 7 , all of which were generated with [37]. Thus, in total, the testing set contained 200 sliding tile puzzles for each of the three dimensions.

Blocksworld, N-Puzzle, Ferry, Elevators, and Spanner The problem instances were copied from [44] (Blocksworld, N-Puzzle, and Ferry) and from [34] ( elevators-00-strips ). Problem instances of Spanner were generated by [43]. Blocksworld contained 732 problem instances of size 3-15, N-Puzzle contained 80, Ferry contained 48, Elevators contained 80, and Spanner contained 440. Table 5 in Appendix shows the fraction of -olved mazes by breadth-first search for calibration of the difficulty. The problem instances were randomly divided into training, validation, and testing sets, each containing 50% / 25% / 25% of the whole set of problems. The source code of all experiments is available at https://github.com/aicenter/Optimize-Planning-Heuristics-to-Rank with MIT license.

6.1 Experimental results

Table 1 shows the fraction of solved mazes in percent for all combinations of search algorithms (A* and GBFS) and heuristic functions optimized against compared loss functions. The results match the above theory, as the proposed ranking losses are always better than the L 2 regression loss.

For A* search, the heuristic function optimized against the L ∗ is clearly the best, as it is only once inferior to L rt and L be on the Ferry domain. As expected, heuristic functions optimized with respect to L gbfs behave erratically in A* search, because they were optimized for a very different search.

For GBFS, heuristic functions optimized against the L ∗ or against L gbfs behave the best. It can be shown (the proof is provided in Apendix) that since each action has a constant and positive cost, the heuristic function optimizing L ∗ optimizes L gbfs as well, but the opposite is not true.

Heuristic functions optimized against L 2 estimating cost-to-goal deliver most of the time the worst results. Heuristic functions optimized against other ranking losses L rt and L be frequently perform well, most of the time better than L 2 , but sometimes (Elevators, Sokoban, Sliding puzzle) are much

worse than proposed losses. These experiments further confirm the [57, 18, 59] stating that optimizing the exact estimate of cost-to-goal is unnecessary, but they also show the advantage of including states off the optimal path in the loss function. Finally, the method combining policy and heuristic in GBFS, L le , [38] performs better that L 2 , but worse than the proposed ranking losses. Since both approaches are derived from the same motivation, we attribute this to difficulties in training neural networks with two heads. Ranking losses optimize just the heuristic which is directly used in the search, which seems to us to be simpler. Additional statistics, namely the average number of expanded states (Table 6) and the average length of plans (Table 8) are provided in the Appendix.

7 Conclusion

The motivation of this paper stemmed from the observation that even the cost-to-goal, considered to be an optimal heuristic, fails to guarantee a strictly optimally efficient search. Since a large body of existing research optimizes this quantity, we are effectively lost with respect to what should be optimized. To fill this gap, we have stated the necessary and sufficient conditions guaranteeing the forward search to be strictly optimally efficient. These conditions show that the absolute value of the heuristic is not important, but that the ranking of states in the Open list is what controls the efficiency. Ranking can be optimized by minimizing the ranking loss functions, but its concrete implementation needs to correspond to a variant of the forward search. In case of mismatch, the resulting heuristic can perform poorly, which has been demonstrated when the heuristic optimized for BGFS search was used with A* search. The other benefit of ranking losses is that from the point of view of statistical learning theory, they solve a simpler problem than ubiquitous regression in estimating the cost-to-goal.

The experimental comparisons on eight problem domains convincingly support the derived theory. Heuristic functions optimized with respect to ranking loss L ∗ instantiated for A* search perform almost always the best with A* search (except for one case where it was the second best). In the case of GBFS, heuristic functions optimizing ranking losses L gbfs and L ∗ instantiated for GBFS and A* worked the best. This is caused by the fact that the heuristic optimizing L ∗ for A* optimizes L gbfs for GBFS as well.

We do not question the existence of a strictly optimally efficient heuristic. Given our experiments, we believe that if the space of heuristic functions over which the loss is optimized is sufficiently rich, the result will be sufficiently close to the optimal for the needs of the search.

References

Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence , 1(8):356-363, 2019.

Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5360-5370, 2017.

Shahab Jabbari Arfaee, Sandra Zilles, and Robert C Holte. Learning heuristic functions for large state spaces. Artificial Intelligence , 175(16-17):2075-2098, 2011.

Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review , 59(1):65-98, 2017.

Mohak Bhardwaj, Sanjiban Choudhury, and Sebastian Scherer. Learning heuristic search via imitation. In Conference on Robot Learning , pages 271-280. PMLR, 2017.

Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence. 2001 Jun; 129 (1-2): 5-33. , 2001.

Vladimir Cherkassky, Xuhui Shao, Filip M Mulier, and Vladimir N Vapnik. Model complexity control for regression using vc generalization bounds. IEEE transactions on Neural Networks , 10(5):1075-1089, 1999.

Leah Chrestien, Tomáš Pevný, Antonín Komenda, and Stefan Edelkamp. Heuristic search planning with deep neural network using imitation, attention and curriculum learning. In PRL-IJCAI workshop , 2021.

Lukás Chrpa. Combining learning techniques for classical planning: Macro-operators and entanglements. In ICTAI , pages 79-86. IEEE Computer Society, 2010.

Rina Dechter and Judea Pearl. The optimality of A* revisited. In Michael R. Genesereth, editor, AAAI , pages 95-99. AAAI Press, 1983.

Edsger W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik , 1:269-271, 1959.

Stefan Edelkamp. Automated creation of pattern database search heuristics. In Model Checking and Artificial Intelligence , pages 35-50, 2006.

Stefan Edelkamp and Stefan Schrödl. Heuristic Search - Theory and Applications . Academic Press, 2012.

Ariel Felner, Uzi Zahavi, Robert Holte, Jonathan Schaeffer, Nathan Sturtevant, and Zhifu Zhang. Inconsistent heuristics in theory and practice. Artificial Intelligence , 175(9):1570-1603, 2011.

Dieqiao Feng, Carla P. Gomes, and Bart Selman. Solving hard AI planning instances using curriculum-driven deep reinforcement learning. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 , pages 2198-2205. ijcai.org, 2020.

Patrick Ferber, Malte Helmert, and Jörg Hoffmann. Neural network heuristics for classical planning: A study of hyperparameter space. In ECAI , pages 2346-2353. IOS Press, 2020.

Santiago Franco, Álvaro Torralba, Levi H. S. Lelis, and Mike Barley. On creating complementary pattern databases. In IJCAI , pages 4302-4309, 2017.

Caelan Reed Garrett, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Learning to rank for synthesizing planning heuristics. page 3089-3095, 2016.

Edward Groshev, Aviv Tamar, Maxwell Goldstein, Siddharth Srivastava, and Pieter Abbeel. Learning generalized reactive policies using deep neural networks. 2018.

Arthur Guez, Théophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wierstra, Rémi Munos, and David Silver. Learning to search with mctsnets. In International conference on machine learning , pages 1822-1831. PMLR, 2018.

Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics , 4(2):100-107, 1968.

Patrik Haslum, Adi Botea, Malte Helmert, Blai Bonet, and Sven Koenig. Domain-independent construction of pattern database heuristics for cost-optimal planning. In AAAI , pages 1007-1012, 2007.

Malte Helmert and Gabriele Röger. How good is almost perfect? In AAAI , pages 944-949. AAAI Press, 2008.

Robert C. Holte and Sandra Zilles. On the optimal efficiency of cost-algebraic A. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019 , pages 2288-2295. AAAI Press, 2019.

Michael Katz, Shirin Sohrabi, Horst Samulowitz, and Silvan Sievers. Delfi: Online planner selection for cost-optimal planning. IPC-9 planner abstracts , pages 57-64, 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014.

Donald E Knuth and Ronald W Moore. An analysis of alpha-beta pruning. Artificial intelligence , 6(4):293-326, 1975.

Richard E. Korf. Iterative-deepening-A*: An optimal admissible tree search. In IJCAI , pages 1034-1036. Morgan Kaufmann, 1985.

Richard E Korf. Macro-operators: A weak method for learning. Artificial intelligence , 26(1):3577, 1985.

Šimon Mandlík, Matˇ ej Raˇ cinsk` y, Viliam Lis` y, and Tomáš Pevn` y. Jsongrinder. jl: automated differentiable neural architecture for embedding arbitrary json data. Journal of Machine Learning Research , 23(298):1-5, 2022.

Drew V. McDermott. A heuristic estimator for means-ends analysis in planning. In AIPS , pages 142-149. AAAI, 1996.

David Ménager, Dongkyu Choi, Mark Roberts, and David W. Aha. Learning planning operators from episodic traces. In 2018 AAAI Spring Symposia . AAAI Press, 2018.

Ionut Moraru, Stefan Edelkamp, Santiago Franco, and Moisés Martínez. Simplifying automated pattern selection for planning with symbolic pattern databases. In KI , pages 249-263, 2019.

Christian Muise. Collection of pddl files of classical problems. https://github.com/AI-Planning/ classical-domains, 2023.

XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. On surrogate loss functions and f-divergences. The Annals of Statistics , pages 876-904, 2009.

Laurent Orseau, Levi Lelis, Tor Lattimore, and Théophane Weber. Single-agent policy tree search with guarantees. Advances in Neural Information Processing Systems , 31, 2018.

Laurent Orseau and Levi HS Lelis. Implementation of the algorithms described in "policyguided heuristic search with guarantees" by l. orseau and l. lelis, published at aaai'21. https: //github.com/levilelis/h-levin, 2021.

Laurent Orseau and Levi HS Lelis. Policy-guided heuristic search with guarantees. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12382-12390, 2021.

Arthur L Samuel. Some studies in machine learning using the game of checkers. ii-recent progress. IBM Journal of research and development , 11(6):601-617, 1967.

Max-Philipp B. Schrader. gym-sokoban. https://github.com/mpSchrader/gym-sokoban, 2018.

Jendrik Seipp, Florian Pommerening, and Malte Helmert. New optimization functions for potential heuristics. In ICAPS , 2015.

Jendrik Seipp and Silvan Sievers. Fast downward stone soup 2023.

Jendrik Seipp, Álvaro Torralba, and Jörg Hoffmann. PDDL generators. https://doi.org/10.5281/ zenodo.6382173, 2022.

William Shen. Strips-hgn. https://github.com/williamshen-nz/STRIPS-HGN, 2021.

William Shen, Felipe Trevizan, and Sylvie Thiébaux. Learning domain-independent planning heuristics with hypergraph networks. In ICAPS , volume 30, pages 574-584, 2020.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Kristian Smith. Maze creator / solver. https://github.com/ravenkls/Maze-Generator-and-Solver, 2022.

Gustav Sourek, Vojtech Aschenbrenner, Filip Zelezny, Steven Schockaert, and Ondrej Kuzelka. Lifted relational neural networks: Efficient learning of latent relational structures. Journal of Artificial Intelligence Research , 62:69-100, 2018.

Simon Ståhlberg, Blai Bonet, and Hector Geffner. Learning general optimal policies with graph neural networks: Expressive power, transparency, and limits. In Akshat Kumar, Sylvie Thiébaux, Pradeep Varakantham, and William Yeoh, editors, Proceedings of the Thirty-Second International Conference on Automated Planning and Scheduling, ICAPS 2022, Singapore (virtual), June 13-24, 2022 , pages 629-637. AAAI Press, 2022.

Ingo Steinwart. Consistency of support vector machines and other regularized kernel classifiers. IEEE transactions on information theory , 51(1):128-142, 2005.

Takeshi Takahashi, He Sun, Dong Tian, and Yebin Wang. Learning heuristic functions for mobile robot path planning using deep neural networks. In ICAPS , volume 29, pages 764-772, 2019.

Alvaro Torralba, Vidal Alcázar, Daniel Borrajo, Peter Kissmann, and Stefan Edelkamp. SymBA*: A symbolic bidirectional A* planner. In International Planning Competition , pages 105-108, 2014.

Sam Toyer, Sylvie Thiébaux, Felipe Trevizan, and Lexing Xie. Asnets: Deep learning for generalised planning. Journal of Artificial Intelligence Research , 68:1-68, 2020.

Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems , 4, 1991.

Marin Vlastelica, Michal Rolinek, and Georg Martius. Neuro-algorithmic policies enable fast combinatorial generalization. In International Conference on Machine Learning , pages 10575-10585. PMLR, 2021.

Xuemei Wang. Learning planning operators by observation and practice. In Kristian J. Hammond, editor, AIPS , pages 335-340. AAAI, 1994.

Christopher Wilt and Wheeler Ruml. When does weighted A* fail? In International Symposium on Combinatorial Search , volume 3, 2012.

Christopher Makoto Wilt and Wheeler Ruml. Effective heuristics for suboptimal best-first search. J. Artif. Intell. Res. , 57:273-306, 2016.

Yuehua Xu, Alan Fern, and Sungwook Yoon. Learning linear ranking functions for beam search with application to planning. Journal of Machine Learning Research , 10(7), 2009.

Ryo Yonetani, Tatsunori Taniai, Mohammadamin Barekatain, Mai Nishimura, and Asako Kanezaki. Path planning using neural A* search. In International Conference on Machine Learning , pages 12029-12039. PMLR, 2021.

Ziqi Zhang and Florian Geißer. Extending graph neural networks for generalized stochastic planning. In Bridging the Gap Between AI Planning and Reinforcement Learning . 2021.

Tan Zhi-Xuan. PDDL. jl: An Extensible Interpreter and Compiler Interface for Fast and Flexible AI Planning . PhD thesis, Massachusetts Institute of Technology, 2022.

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems , 33:18795-18806, 2020.

8 Appendix

8.1 Speed of convergence against the size of training set:

We use the classical results of statistical learning theory to show that estimating the cost-to-go converges more slowly with respect to the size of the training set than estimating the rank.

From Equation (1) in the main text it should be obvious that ranking is in its essence a classification problem if one considers a pair of states ( s, s ′ ) as a single sample. For classification/ranking problems, a following bound on true error rate [54] holds with probability 1 -η

where R c denotes the true loss (Equation (1) in the main text) and ˆ R c , its estimate from n p state pairs ( s i , s j ) , and κ , the Vapnik-Chervonenkis dimension [54] of the hypothesis space.

Optimizing h ( s, θ ) with respect to cost-to-goal (Equation (5) in the main text) is a regression problem for which a different generalization bound on prediction error [7] holds with probability 1 -η

where again R r is the error of the estimator of cost-to-go and ˆ R r is its estimate from n s states (Equation (5) in the main text), 2 and [ x ] + is a shorthand for max { 0 , x } .

From the above, we can see that excess error in the ranking case converges to zero at a rate √ ln n p √ n p , which is slightly faster than that of regression √ ln n s √ n s - √ ln n s . But, the number of state pairs in the training set grows quadratically with the number of states; therefore, the convergence rate of the ranking problem for the number of states n s can be expressed as √ 2 ln n s n s , which would be by at least 1 √ n s factor faster than that of regression. We note that bounds are illustrative, since the independency of samples is in practice violated, since samples from the same problem-instance are not independent.

8.2 Suboptimality of perfect heuristic in GBFS

A

10

B

3

C

8

D

4

0

E

2

8

3

4

4

Example 2. While cost-to-goal h ∗ is the best possible heuristic for algorithms like A* (up to tie-breaking) in terms of nodes being expanded, for GBFS, h ∗ does not necessarily yield optimal solutions.

Proof. A* explores all the nodes with f ( s ) < f ∗ ( s ) = g ( s ) + h ∗ ( s ) and some with f ( s ) = f ∗ ( s ) = g ( s ) + h ∗ ( s ) , so nodes can only be saved with optimal f ∗ ( s ) = g ( s ) + h ∗ ( s ) . In fact, when given

h ∗ as the heuristic, only nodes s with f ( s ) = f ∗ ( s ) are expanded. Depending on the strategy of tie-breaking, the solution path can be found in the minimal number of node expansions or take significantly longer (e.g., in lower g -value first exploration of the search frontier). Any heuristic other than h ∗ is either overestimating, and, therefore, may lead to either non-optimal solutions in A*, or weaker than h ∗ , leading to more nodes being expanded.

Even if h ∗ is given, GBFS is not guaranteed to be optimal. Consider the following graph with five nodes A,B,C,D,E , and w ( A,B ) = 8 , w ( B,E ) = 3 , w ( A,C ) = 2 , w ( C, D ) = 4 , w ( D,E ) = 4 , and h ∗ ( A ) = 10 , h ∗ ( B ) = 3 , h ∗ ( C ) = 8 , h ∗ ( D ) = 4 , h ∗ ( E ) = 0 (see Figure 2), initial node s 0 = A , goal node E ∈ S ∗ . The numbers are the actual costs and the red numbers are the exact heuristic function. For finding a path from node A to node E , GBFS would return ( A,B,E ) following the heuristic function. However, the path ( A,C,D,E ) has cost 10 instead of 11 .

8.3 Theory

Below, we prove the claim in made in Experimental section stating that if a heuristic h is strictly optimally efficient for A* search, then it is also strictly optimally efficient for GBFS.

Theorem 2. Let a heuristic h is a perfect ranking for A* search on a problem instance Γ = ( ⟨S , E , w ⟩ , s 0 , S ∗ ) with a constant non-negative cost of actions (( ∃ c ≥ 0) ( ∀ e ∈ E ) ( w ( e ) = c )) . Then h is a perfect ranking for GBFS on Γ .

Proof. Let π = (( s 0 , s 1 ) , ( s 1 , s 2 ) , . . . , ( s l -1 , s l )) be an optimal plan such that ∀ i ∈ { 1 , . . . , l } and ∀ s j ∈ { s j | ∃ ( s k , s j ) ∈ E ∧ s k ∈ S π : i -1 ∧ s j / ∈ S π : i } we have g ( s j ) + h ( s j ) > g ( s i ) + h ( s i ) , where g ( s ) is the distance from s 0 to s in a search-tree created by expanding only states on the optimal path π. We want to proof that if all actions have the same positive costs, then h ( s j ) > h ( s i ) as well.

We carry the proof by induction with respect to the number of expanded states.

At the initialization (step 0) the claim trivially holds as the Open list contains just a root node and the set of inequalities is empty.

Let's now make the induction step and assume the theorem holds for the first i -1 step. We divide the proof to two parts. At first, we prove the claim for ( O ) i \ N ( s i -1 ) and then we proof the claim for N ( s i -1 ) , where N ( s ) denotes child states of the state s.

1) Assume s j ∈ ( O ) i \ N ( s i -1 ) . Since h is strictly optimally efficient for A*, it holds that

where the last inequality is true because g ( s i ) -g ( s j ) ≥ 0 .

̸

Assume ( s j ∈ N ( s i -1 ))( s j = s i ) . Since h is strictly optimall efficient for A*, it holds that

Since g ( s j ) = w (( s i -1 , s j )) = w (( s i -1 , s i )) = g ( s i ) , it holds

which finishes the proof of the theorem.

8.4 Optimally efficient heuristic might not exists for GBFS

Consider the following graph in Figure 3 with four nodes A,B,C,D, , and w ( A,B ) = 1 , w ( B,D ) = 1 , w ( A,C ) = 1 , w ( A,D ) = 9 , w ( B,C ) = 9 , and h ∗ ( A ) = 0 , h ∗ ( B ) = 1 , h ∗ ( C ) = 1 , h ∗ ( D ) = 2 where A is the goal state and D is the initial state.

We can see that for GBFS, the perfect heuristic does not exist for D. On expansion, the GBFS algorithm will put A and B to the open list with heuristic values h(A) = 0 and h(B) = 1. GBFS takes A from the open list and checks if it is a goal state. Since A is goal state, GBFS terminates returning

A

0

B

1

C

1

D

2

A

1

1

9

9

1

path (D,A) as a solution. However, this is not the optimal path as a better (optimal) solution exists (D, B, A). Since the definition of optimal ranking requires the inequalities to hold for this optimal path, in GBFS, the perfect heuristic does not exist for all initial states.

The problem can be fixed if the definition of optimal ranking is changed to consider two cases in the merit function f ( s ) = αg ( s ) + βh ( s ) : α > 0 and β > 0 and α = 0 and β > 0 . In the first case, the optimal ranking should be defined with respect to the "optimal path" (this is the A*); in the latter case, it should be the path with minimal number of expansions. With GBFS, the user simply wants to find a solution but not care about its optimality. With this change, the heuristic function will exist for Figure 3.

8.5 Training set with multiple solution paths with the same length of the plan

The behavior of the learned heuristic function depends on the composition of the training set, which is illustrated below on a simple grid problem. The agent starts at position (4,4) and has to move to the goal position (0,0). There are no obstacles and the agent can only move one tile down or one tile left (the effects on his positions are either (-1,0) or (0,-1)), the cost of the action is one. The number of different solutions path grows exponentially with the size of the grid and all solution path has the same cost (for problem of size 5x5, there are 70 solutions). This problem is interesting, since merit function in A* with optimal heuristic function (cost to goal) is constant, equal to 8, for all states.

For the size of the grid (4,4), the heuristic is determined by a table with 25 values. Below, these values are determined by minimizing the proposed loss for A* algorithm with logistic loss surrogate by BFGS (with all zeros as initial solution). Since the loss function is convex, BFGS finds the solution quickly.

In the first case, the training set contains one solution path ([4, 4], [3, 4], [2, 4], [1, 4], [0, 4], [0, 3], [0, 2], [0, 1], [0, 0]), where the agent first goes left and then down. The learnt heuristic values h is shown in Table 2.

An A* search using these heuristic values will first always take action moving the agent to left and then down. When moving down, the agent does not have another choice. Notice that the heuristic values of many states are not affected by the optimization, because they are not needed to effectively solve the problem.

In the second case, the training set contains two solution paths: the first is in Table 2 and in the second table 3, the agent goes first down and then left. The learnt heuristic values are shown in 3.

An A* search will now face tie at state (4,4), since states (3,4) and (4,3) have the same heuristic value. But the presence of the tie does not affect the optimal efficiency of the search, as A* will always expand states on one of the optimal path from the training set.

Finally let's consider a case, where all 70 possible solutions are in the training set. The learned heuristic values are shown in Table 4.

Rank of heuristic values "roughly" corresponds to cost to goal. The reason, why some states are preferred over the others despite their true cost-to-goal being the same is that they appear in more solution paths. As shown in Table 4, the A* search with learnt heuristic values is strictly optimally efficient.

8.6 Baseline Comparison to breadth-first search

The fraction of solved problems for breadth-first search (5s time limit as used by solvers in the paper) is shown in Table 5.

8.7 Training Details

For the grid domains, ADAM [26] training algorithm was run for 100 × 20000 steps for the grid domains.The experiments were conducted in the Keras-2.4.3 framework with Tensorflow-2.3.1 as the backend. While all solvers were always executed on the CPU, the training used an NVIDIA Tesla GPU model V100-SXM2-32GB. Forward search algorithms were given 10 mins to solve each problem instance.

For the PDDL domains, the training consumed approximately 100 GPU hours and evaluation consumed 1000 CPU hours.All training and evaluation were done on single-core Intel Xeon Silver 4110 CPU 2.10GHz with a memory limit of 128GB. The training algorithm AdaBelief [63] was

allowed to do 10000 steps on the CPU. We emphasize though, that the training time does not include the cost of creating the training set.

independent planners.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper highlights the limitations of the conventional method for learning heuristics in search algorithms, which involves imitating the optimal cost-to-go function. The authors present an alternative approach that focuses on optimizing the correct ranking of state pairs by the heuristic. Because of that characteristic, the authors argue that the resulting heuristic offers both theoretical and practical advantages compared to the standard approach.\n\nSTRENGTHS:\nThe paper presents an original idea that indeed can offer a significan advantage for planning methods. The presentation is mostly clear, with only minor problems.\n\nWEAKNESSES:\nIn my opinion, the claims formulated in the paper should be supported better. I'm not convinced that the reported results stem from the theory. Clearer discussion of theoretical advantages and disadvantages of various approaches, more detailed experiments including analysis of the learned heuristic, and comparison with some other baselines could be a significant improvement to the paper. Details of my concerns are discussed in the comments.\n\nQUESTIONS:\nIn general, I like the idea and the paper, although at this point it may need some improvements. At this point I'm not fully convinced that the improvement you show is due to the theoretical advantages (which should be clarified) or just a fancy heuristic that happens to work better than a certain imitation objective. Below I highlight a few specific comments.\n\n- l.111,117,157 Arguably a central expression, appears in 3 places, and every time is written differently. Correct it (or explain that it's actually not the same).\n\n- l.113-125 This explanation of the notation above is a lifesaver.\n\n- l.131 Does there exist at least one perfect ranging for every problem? Semms that h(s):=[-infty on a fixed optimal path, infty otherwise] works. What happens in quite a common case in which s_0 is not specified and the heuristic should work for any starting state? Does there always exist a perfect ranking in such case?\n\n- l.105-150 While I don't deny the soundness of the theory, it reads a little like explaining quite an intuitive idea (\"the heuristic is strictly optimal iff in every step of the search the next node on a fixed optimal path receives the lowest value\") with very complicated expressions. I'd consider changing the order and starting from intuition, moving to the formalities afterwards. The ultimate conclusion should be the expression in l.111, which I believe is the central object in this section.\n\n- l.161 Quite a strong assumption.\n\n- l.167 The relation between log(1+exp(x)) and [[x>0]] indeed seems loose. On convergence it will have the desired property indeed, but it doesn't exactly optimize the number of violations, rather a sum of violation volumes. How does it relate to your theory? An experimental analysis of the learned function would do.\n\n- l.180 I'd consider rewording the section title. Although after reading the section it makes sense, at first it seems like a general discussion of optimizing neural networks.\n\n- l.181-185 I like the idea, but it's not true that it is easier to learn a function from a bigger class. The core element is the learning signal. In case of cost-to-go, where we have actually a direct supervision, it is of much better quality than in case of a set of inequalities. Consider an example in which we have n states and we want to learn any monotonic function. Although there are much more functions satisfying f(s_{i+1})>f(s_i), learning fixed supervised targets should be faster.\n\n- l.186-196 \"states off the solution path\" -- you mean in the training dataset or in general? As far as I understand, your training dataset consists of solving paths only, so the issue applies to your ranking loss as well, doesn't it? When provided with off-the-path states, standard cost-to-go functions can also utilize them for learning (e.g. bootstrapping for Q-learning). The issue you describe is devoted to the particular training loss you consider, not learning cost-to-go in general.\n\n- l.203-207 I don't understand. Optimizing the proposed losses never considers dead-ends as they don't lie on the optimal paths. Do you consider any hand-modified heuristics? If you are worried about too high gradients, use Huber loss instead of L2. Why should lare values appear in the training? (5) is strictly supervised.\n\n- l.269 That's a shame, it would support your claims well as it makes a middle-ground between (3) and (5).\n\n- l.285 How costly is it to get this dataset? Is your method more efficient than SymBA* afterwards? Gathering optimal paths for training may be very costly, especially in the hardest domains. Thus, can you comment on the applicability of your claims, since you require a costly operation of gathering optimal paths for your training> Can your method, like standard imitation, learn also from sub-optimal data, which is often much easier to get?\n\n- l.286 Why a single problem instance?\n\n- l.322-342 If the test sets are harder than training sets, how does it relate to the theory in which you assume having optimal paths?\n\n- l.343-363 I'm not familiar with the complexity of some instances of the environments. Thus, it would be helpful to see some trivial baselines in the comparison, such as BFS or Dijkstra. Also, to showcase an advantage of your theory-driven approach, I'd consider comparing with a simple objective that learns to output 0 on the solving paths and 1 on other states outside the path. It fits the definition of optimality and doesn't seem to be stable or generalizable (or maybe?). Advantage over such a baseline would highlight the benefit of learning the ranking over heuristics. It's missing now. Yes, I know that you compare already agains many relevant baselines.\n\n- l.343-363 Also, the claims should be supported by some ablations. Is the learned heuristic correct? Does it learn to properly rank states, even in the training domains? What does it converge to, do the values resemble cost-to-go or some other interpretable quantities? What are the mistakes? Does the learned heuristic fit the theory of what it should learn? Are the paths obtained this way optimal? How do they relate to optimal paths in terms of length? How often only states on an optimal path are expanded? Can you learn a useful heuristic for a fixed domain with many possible starting points?\n\n- l.375 I'm not sure whether optimizing set of inequalities is easier than a direct supervision.\n\nAnd a few minors:\n\n- l.31,212,215,271 etc. When you refer to something, either in your paper or in supplementary, refer to the exact section.\n\n- l.40 Do you mean \"state-of-the-art\"?\n\n- l.48 I wouldn't say \"always\", you can demonstrate that it's better \"in those eight environments\".\n\n- l.50-72 I can't find a definition of g(s) or h(s) before you use it to explain your algorithm. Maybe in preliminaries? g(s) is defined below the algorithm, but h(s) is not.\n\n- l.116,135, and others: I feel confused whether \"heuristic\" refers to h or f. Please disambiguate the notation.\n\n- l.305 24000 gradient steps in 100 GPU hours? Seems quite slow, I'd suggest checking it.\n\nLIMITATIONS:\nThe limitations are not clearly stated, there should be a dedicated section for them. Societal impact is not an issue here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPlanning with (possibly learned) heuristic function is important for both planning and learning communities. The paper focuses on the forward search without expanding redundant search states in their forward process. With the development theorem, the authors demonstrated how the proposed concept work in various domains with A* and GBFS algorithms.\n\nSTRENGTHS:\n- The concept of the focusing problem, wheares they seem to be simple, is interesting. The proposed theoretical discussion (e.g., sufficiency and necessity in Theorem 1) is clearly explained; the whole paper is well written and structured.\n- The proposed approach with $L_\\star$ based on the two fundamental search techniques A* and GBFS work efficiently than counterparts.\n\nWEAKNESSES:\n- Although the proposed learning-based approach is promising, it requires much learning time on PDDL domains (e.g., 17724 CPU hours).\n- The scales and variations of compared benchmark domains seem limited (e.g., traditional search problems and mazes).\n\nQUESTIONS:\nThe paper is very clearly organized, and I can focus on the content. The below items are come comments.\n\n- Learning-based heuristic functions could be promising for their generalization ability. However, sometimes they are argued that the required learning times (or resources) are too large. In the reported times with PDDL domains, it requires 17724 CPU hours. These results may cause some questions about the efficiency of the learning-based heuristic itself.\n    - Do you discuss why the learning-based heuristic is promising, even though the training resource is required?\n    - Can we compare planners without learning and planners with learning (i.e., the proposed method) in terms of total required resources?\n    - Can the two solvers (A* and GBFS) have similar characteristics in terms of learning-based methods? The two solvers can be formulated similarly, as noted in L.93. Can we regard these methods as just variants of search using the function of the form in L.92, or can we have some important findings of these methods through the experiments?\n- Mathematical notation issues:\n    - In Def. 1, some mathematical errors seem to occur in the second condition: see L.111 and L.117 to check the condition of $\\wedge$.\n    - In Eq.(6), the log() is not closed.\n\nLIMITATIONS:\nI have no explicit concerns about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigate the computational advantage if a forward-search algorithm utilises ranking rather than optimising cost-to-goal. I think the contribution is good in general.\n\nSTRENGTHS:\n1. The problem is formulated well. \n2. The introduction of the contents is suitable. \n3. The experiment section is good.\n\nWEAKNESSES:\n1. The definition of \"optimal ranking\" is so important to this paper that it needs to be presented before first used. The authors should not use \"(definition is below)\" for this term. \n2. I suggest the authors simplify some expressions. For example, at some points where strict rigorousness is not required, \"minimise the number of expanded states\"--> \"efficient\". \n3. I cannot understand line 7 in the supplementary file. Is it grammatically correct? \n4.Line 10 in the supplementary file, it is not \"lastly\". By the way, what is the connection between the Section 2 in the supplementary and the main text? I don't think there is a need to present this section. As long as \"estimating the cost-to-go converges\" is slower than the proposed scheme, the contribution of the main text is valid. Am I correct? \n5.Can the author find a citation for the claim of Example 1? so that there is no need to present it in detail. I think this example broke the reading flow. \n6. Why is the related work section at Sec 5? Why can't it be Sec 2? \n7. I don't think Sec 4.3 presented rigorous statements. All things are subjective. Did the author write \"conclusive remarks\" here? I suggest the authors move them to the conclusion section. Don't be afraid that the algorithm section is too short. \n\nMinor points: \n\n1. I didn't find the definition for the \\wedge symbol in Def.1.\n\nQUESTIONS:\nPlease explain my comments listed in the weakness block.\n\nLIMITATIONS:\nI don't think the paper has any negative societal impact. But I do suggest the authors polish their Sec 4.3.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of using imitation learning for obtaining planning heuristics suitable for using variations of best-first search with a weighted sum of h() and g(), covering A* and GBFS. The paper gives both theoretical and empirical arguments for predicting a ranking of actions instead of predicting h*, the optimal distance to the goal. One argument in favour of ranking is that regression is a harder problem than regression. Another one is the observation that even with an optimal heuristic, the search can still expand more nodes than necessary. For that, in the case of symmetries, the proposed algorithm would choose a particular optimal solution among all the possible ones. The empirical results show the proposed approach to perform the best in A* in comparison with the others. A heuristic trained for GBFS performs well.\n\nSTRENGTHS:\n- Interesting discussion of ranking vs value estimation.\n- Hard-to-achieve goal of expanding strictly nodes along the path.\n- Discussion on the pros and cons of the approach.\n\t- Including an honest commentary on sample complexity.\n- Promising empirical results.\n\nWEAKNESSES:\n- The proposed approach does not account for flexible goals or multiple goals.\n- Limited discussion on the cases of symmetries and multiple optimal paths.\n\t- This is discussed but it could be analyzed further in the experiments.\n\nQUESTIONS:\n- What happens in the case of symmetry or multiple paths?\n\t- For instance, suppose there are k discrete integer variables, that can be increased and decreased independently. Suppose the goal is to set them to 0. Optimal plans just interleave increasing the variables independently. What would the theoretical analysis and the empirical results look like?\n\t- This example might be unrealistic but it's related to problems with very flexible solutions like the PDDL benchmark Logistics.\n- How sensitive is the method to the distribution of problems and trajectories?\n\t- Is there any reason for it behaving differently across the proposed methods and the others?\n\nLIMITATIONS:\nThe line of work is interesting, and there are certain domains where this method is relevant. The paper might benefit from exploring better those cases, given that the paper admits that there are cases where estimating the value might be better.\n\nIt'd be good to discuss the stability of training for the different algorithms.\n\nThe space used for introducing the search algorithm could be used for further discussion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}