SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Symmetry of  $\mathrm{L}_2$  (and of  $\mathrm{L}_1$ ) loss are discussed in [51], as it does not promote the admissibility of the heuristic. It suggests asymmetric  $\mathrm{L}_1$  loss with different weights on the left and right parts, but this does not completely solve the problem of estimating cost-to-goal (for example with ties, dead-ends).
Parameters of potential heuristics [41] are optimized by linear programming for each problem instance to meet admissibility constraints while helping the search. On contrary, this work focuses on the efficiency of the search for many problem instances but cannot guarantee admissibility in general.
Combining neural networks with discrete search algorithms, such that they become an inseparable part of the architecture is proposed in [55] and [60]. The gradient with respect to parameters has to be
propagated through the search algorithm which usually requires both approximation and the search algorithm to be executed during every inference.
A large body of literature [46, 20, 15, 2] improves the Monte Carlo Tree Search (MCTS), which has the advantage to suppress the noise in estimates of the solution costs. These methods proposed there are exclusive to MCTS and do not apply to variants of the forward search algorithm. Similarly, a lot of works [45, 53, 61, 19, 8, 16, 5] investigate architectures of neural networks implementing the heuristic function  $h(s,\theta)$ , ideally for arbitrary planning problems. We use [45, 8] to implement  $h(s,\theta)$ , but remind our readers that our interest is the loss function minimized when optimizing  $\theta$ .
Finally, [1] proposes to construct a training set similarly to pattern databases [12] by random backward movements from the goal state and to adapt  $\mathrm{A}^*$  for multi-core processors by expanding  $k > 1$  states in each iteration. These improvements are independent of those proposed here and can be combined.
The experimental protocol was designed to compare the loss functions while removing all sources of variability. It therefore uses imitation learning in planning [18, 19] and leaves more interesting bootstrap protocol for future work [3]. The goal was not to deliver state-of-the-art results, therefore the classical solvers were omitted, but their results are in the Appendix for completeness.
The proposed ranking losses are instantiated with the logistic loss function as in Equation (4) for  $\mathrm{A}^*$  search by setting  $\alpha = \beta = 1$  in (2), called  $\mathbf{L}^{*}$ , and for GBFS, by setting  $\alpha = 0$ ,  $\beta = 1$  in (2), called  $\mathrm{L_{gbfs}}$ .  $\mathrm{L}_{2}$  is used as defined in Equation (5). Since the heuristics used in this paper are not admissible,  $\mathrm{A}^*$  search will not provide optimal solutions. It was included in the comparison to demonstrate that heuristics optimized with respect to  $\mathrm{L_{gbfs}}$  might be inferior in  $\mathrm{A}^*$  search. To observe the advantage of ranking over regression, we defined a  $\mathrm{L}_{\mathrm{rt}}$  loss function
$$
\operatorname {L} _ {\mathrm {r t}} (h, \Gamma , \pi) = \sum_ {(s _ {i - 1}, s _ {i}) \in \pi} \log (1 + \exp (h (s _ {i}, \theta) - h (s _ {i - 1}, \theta))) \tag {6}
$$
comparing only states on the optimal trajectory. We also compare to the loss function proposed in [49] defined as  $\mathrm{L_{be}}(h,\Gamma ,\pi) = \sum_{s\in \mathcal{S}^{\pi}}\max \{1 + \min_{s'\in \mathcal{N}(s)}h(s',\theta) - h(s,\theta),0\} +\max \{0,h^{*}(s) - h(s,\theta)\} +\max .\{0,h(s,\theta) - 2h^{*}(s)\}$ , where  $\mathcal{N}(s)$  denotes child-nodes of state  $s$ . Furthermore, we compare to the policy-guided heuristic search [38] (denoted as  $\mathrm{L_{le}}$ ) with GBFS modified for efficiency as described in the paper while simultaneously adapting the neural network to provide the policy and heuristic value.
The heuristic function  $h(s, \theta)$  for a given problem domain was optimized on the training set containing problem instances and their solutions, obtained by SymBA* solver [52] for Sokoban and Maze with teleports and [42] for Sliding Tile. Each minibatch in SGD contained states from exactly a single problem instance needed to calculate the loss. This means that for  $L_2$ ,  $L_{\mathrm{rt}}$ ,  $L_{\mathrm{le}}$ , only states on the solution trajectory were included; for  $L^*$ ,  $L_{\mathrm{gbfs}}$ , and  $L_{\mathrm{be}}$  there were additional states of distance one (measured by the number of actions) from states on the solution trajectory. Optimized heuristic functions were always evaluated within  $A^*$  and GBFS forward searches, such that we can analyze how the heuristic functions specialize for a given search. To demonstrate the generality, experiments contained eight problem domains of two types (grid and general PDDL) using two different architectures of neural networks described below.
Neural network for grid domains States in Sokoban, Maze with teleports, and Sliding-puzzle can be easily represented in a grid (tensor). The neural network implementing the heuristic function  $h(s,\theta)$  was copied from [8]. Seven convolution layers,  $P_{1}\ldots P_{7}$  of the same shape  $3\times 3$  with 64 filters are followed by four CoAT blocks (CoAT block contains a convolution layer followed by a multi-head attention operation with 2 heads and a positional encoding layer). Convolution layers in CoAT blocks have size  $3\times 3$  and have 180 filters. The network to this stage preserves the dimension of the input. The output from the last block is flattened by mean pooling along  $x$  and  $y$  coordinates before being passed onto a fully connected layer (FC) predicting the heuristic. All blocks have skip connections.
${}^{1}$  runs the "LAMA 2011 configuration" of the planner
Neural network for PDDL domains For Blockworld, Ferry, Spanner, N-Puzzle, and Elevators, where the state cannot be easily represented as a tensor, we have used Strips-HGN [44] and we refer the reader therein for details. Strips-HGN was implemented in Julia [4] using PDDL. j1 [62] extending [30]. Importantly, we have used the hyper-graph reduplication proposed in [48] to improve computational efficiency. Since we did not know the precise architecture (unlike in the grid domain), we performed a grid-search over the number of hyper-graph convolutions in  $\{1,2,3\}$ , dimensions of filters in  $\{4,8,16\}$ , and the use of residual connections between features of vertices. The representation of vertices was reduced by mean and maximum pooling and passed to a two-layer feed-forward layer with hidden dimension equal to the number of filters in hyper-graph convolutions. All non-linearities were of relu type. The best configuration was selected according to the number of solved problems in the validation set. Forward search algorithms were given 5s to solve each problem instance chosen in a manner to emphasize differences between loss functions. All experiments were repeated 3 times. All experiments on the PDDL domains including development took 17724 CPU hours. Optimizing heuristic for a single problem instance took approximately 0.5 CPU/h except Gripper domain, where the training took about 8h due to large branch factors. The rest of this subsection gives further details about domains.
Sokoban The training set for the Sokoban domain contained 20000 mazes of grid size  $10 \times 10$  with a single agent and 3 boxes. The testing set contained 200 mazes of the same size  $10 \times 10$ , but with 3, 4, 5, 6, 7 boxes. Assuming the complexity of to be correlated with the number of boxes, we can study generalization to more difficult problems. The problem instances were generated by [40].
Maze with teleports The training set contained 20000 mazes of size  $15 \times 15$  with the agent in the upper-left corner and the goal in the lower-right corner. The testing set contained 200 mazes of size  $50 \times 50$ ,  $55 \times 55$ , and  $60 \times 60$ . Therefore, as in the case of Sokoban, the testing set was constructed such that it evaluates the generalization of the heuristic to bigger mazes. The mazes were generated using an open-source maze generator [47], where walls were randomly broken and 4 pairs of teleports were randomly added to the maze structure.
Sliding puzzle The training set contained 20000 puzzles of size  $5 \times 5$ . The testing set contained 200 mazes of size  $5 \times 5$ . These puzzles were taken from [37]. Furthermore, we tested our approach on puzzles of higher dimensions such  $6 \times 6$  and  $7 \times 7$ , all of which were generated with [37]. Thus, in total, the testing set contained 200 sliding tile puzzles for each of the three dimensions.
Blocksworld, N-Puzzle, Ferry, Elevators, and Spanner The problem instances were copied from [44] (Blocksworld, N-Puzzle, and Ferry) and from [34] (elevators-00-strips). Problem instances of Spanner were generated by [43]. Blocksworld contained 732 problem instances of size 3–15, N-Puzzle contained 80, Ferry contained 48, Elevators contained 80, and Spanner contained 440. Table 5 in Appendix shows the fraction of -olved mazes by breadth-first search for calibration of the difficulty. The problem instances were randomly divided into training, validation, and testing sets, each containing  $50\%$  /  $25\%$  /  $25\%$  of the whole set of problems. The source code of all experiments is available at https://github.com/aicenter/Optimize-Planning-Heuristics-to-Rank with MIT license.
6.1 Experimental results
Table 1 shows the fraction of solved mazes in percent for all combinations of search algorithms (A* and GBFS) and heuristic functions optimized against compared loss functions. The results match the above theory, as the proposed ranking losses are always better than the  $\mathrm{L}_2$  regression loss.
For  $\mathrm{A}^*$  search, the heuristic function optimized against the  $\mathrm{L}^*$  is clearly the best, as it is only once inferior to  $\mathrm{L}_{\mathrm{rt}}$  and  $\mathrm{L}_{\mathrm{be}}$  on the Ferry domain. As expected, heuristic functions optimized with respect to  $\mathrm{L}_{\mathrm{gbfs}}$  behave erratically in  $\mathrm{A}^*$  search, because they were optimized for a very different search.
For GBFS, heuristic functions optimized against the  $\mathrm{L}^*$  or against  $\mathrm{L}_{\mathrm{gbfs}}$  behave the best. It can be shown (the proof is provided in Appendix) that since each action has a constant and positive cost, the heuristic function optimizing  $\mathrm{L}^*$  optimizes  $\mathrm{L}_{\mathrm{gbfs}}$  as well, but the opposite is not true.
Heuristic functions optimized against  $\mathrm{L}_2$  estimating cost-to-goal deliver most of the time the worst results. Heuristic functions optimized against other ranking losses  $\mathrm{L}_{\mathrm{rt}}$  and  $\mathrm{L_{be}}$  frequently perform well, most of the time better than  $\mathrm{L}_2$ , but sometimes (Elevators, Sokoban, Sliding puzzle) are much
problem	complx.	A*	GBFS
L*	Lgbfs	Lrt	L2	Lbe	L*	Lgbfs	Lrt	L2	Lbe	Lle
Blocks	100	100	100	99	100	100	100	100	100	100	99
Ferry	98	98	100	92	100	98	100	100	100	98	98
N-Puzzle	89	87	88	83	89	92	89	89	89	92	88
Spanner	100	89	100	84	92	100	100	100	100	100	100
Elevators	91	85	75	36	66	92	85	79	76	67	58
Sokoban	3 boxes	99	98	96	97	92	98	100	94	95	92	98
4 boxes	89	89	85	81	82	87	91	84	83	84	84
5 boxes	80	75	72	72	73	78	77	74	72	72	73
6 boxes	76	69	59	51	53	73	71	56	51	54	64
7 boxes	55	49	47	42	45	51	49	48	43	45	49
Maze w. t.	50 × 50	92	91	88	87	87	89	90	89	84	85	89
55 × 55	78	75	73	72	74	74	75	74	72	75	74
60 × 60	49	37	35	32	31	42	48	36	34	32	42
Sliding puzzle	5 × 5	88	83	84	80	82	86	87	84	84	84	85
6 × 6	51	48	49	45	46	47	49	45	43	46	48
7 × 7	39	35	36	32	34	35	36	35	32	34	35
Table 1: Fraction of solved problem instances in percent from the testing set. The top row denotes the type of the search and the second top row denotes the loss function against which the heuristic function was optimized. The complexity is explicitly shown for grid domains. Standard deviations are most of the times smaller than one percent and are provided in Table in the Appendix.
worse than proposed losses. These experiments further confirm the [57, 18, 59] stating that optimizing the exact estimate of cost-to-goal is unnecessary, but they also show the advantage of including states off the optimal path in the loss function. Finally, the method combining policy and heuristic in GBFS,  $\mathrm{L_{le}}$ , [38] performs better than the proposed ranking losses. Since both approaches are derived from the same motivation, we attribute this to difficulties in training neural networks with two heads. Ranking losses optimize just the heuristic which is directly used in the search, which seems to us to be simpler. Additional statistics, namely the average number of expanded states (Table 6) and the average length of plans (Table 8) are provided in the Appendix.
The motivation of this paper stemmed from the observation that even the cost-to-goal, considered to be an optimal heuristic, fails to guarantee a strictly optimally efficient search. Since a large body of existing research optimizes this quantity, we are effectively lost with respect to what should be optimized. To fill this gap, we have stated the necessary and sufficient conditions guaranteeing the forward search to be strictly optimally efficient. These conditions show that the absolute value of the heuristic is not important, but that the ranking of states in the Open list is what controls the efficiency. Ranking can be optimized by minimizing the ranking loss functions, but its concrete implementation needs to correspond to a variant of the forward search. In case of mismatch, the resulting heuristic can perform poorly, which has been demonstrated when the heuristic optimized for BGFS search was used with  $\mathrm{A}^*$  search. The other benefit of ranking losses is that from the point of view of statistical learning theory, they solve a simpler problem than ubiquitous regression in estimating the cost-to-goal.
The experimental comparisons on eight problem domains convincingly support the derived theory. Heuristic functions optimized with respect to ranking loss  $\mathrm{L}^*$  instantiated for  $\mathrm{A}^*$  search perform almost always the best with  $\mathrm{A}^*$  search (except for one case where it was the second best). In the case of GBFS, heuristic functions optimizing ranking losses  $\mathrm{L}_{\mathrm{gbfs}}$  and  $\mathrm{L}^*$  instantiated for GBFS and  $\mathrm{A}^*$  worked the best. This is caused by the fact that the heuristic optimizing  $\mathrm{L}^*$  for  $\mathrm{A}^*$  optimizes  $\mathrm{L}_{\mathrm{gbfs}}$  for GBFS as well.
We do not question the existence of a strictly optimally efficient heuristic. Given our experiments, we believe that if the space of heuristic functions over which the loss is optimized is sufficiently rich, the result will be sufficiently close to the optimal for the needs of the search.
7.1 Acknowledgements
This work has been supported by project numbers 22-32620S and 22-30043S from Czech Science Foundation and OP VVV project CZ.02.1.01/0.0/0.0/16_019/0000765 "Research Center for Informatics". This work has also received funding from the European Union's Horizon Europe Research and Innovation program under the grant agreement Tuples No 101070149.
[1] Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence, 1(8):356-363, 2019.
[2] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5360-5370, 2017.
[3] Shahab Jabbari Arfaee, Sandra Zilles, and Robert C Holte. Learning heuristic functions for large state spaces. Artificial Intelligence, 175(16-17):2075-2098, 2011.
[4] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65-98, 2017.
[5] Mohak Bhardwaj, Sanjiban Choudhury, and Sebastian Scherer. Learning heuristic search via imitation. In Conference on Robot Learning, pages 271-280. PMLR, 2017.
[6] Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence. 2001 Jun; 129 (1-2): 5-33., 2001.
[7] Vladimir Cherkassky, Xuhui Shao, Filip M Mulier, and Vladimir N Vapnik. Model complexity control for regression using vc generalization bounds. IEEE transactions on Neural Networks, 10(5):1075-1089, 1999.
[8] Leah Chrestien, Tomás Pevny, Antonín Komenda, and Stefan Edelkamp. Heuristic search planning with deep neural network using imitation, attention and curriculum learning. In PRL-IJCAI workshop, 2021.
[9] Lukás Chrpa. Combining learning techniques for classical planning: Macro-operators and entanglements. In ICTAI, pages 79–86. IEEE Computer Society, 2010.
[10] Rina Dechter and Judea Pearl. The optimality of A* revisited. In Michael R. Genesereth, editor, AAAI, pages 95–99. AAAI Press, 1983.
[11] Edsger W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1:269-271, 1959.
[12] Stefan Edelkamp. Automated creation of pattern database search heuristics. In Model Checking and Artificial Intelligence, pages 35-50, 2006.
[13] Stefan Edelkamp and Stefan Schrödl. Heuristic Search - Theory and Applications. Academic Press, 2012.
[14] Ariel Felner, Uzi Zahavi, Robert Holte, Jonathan Schaeffer, Nathan Sturtevant, and Zhifu Zhang. Inconsistent heuristics in theory and practice. Artificial Intelligence, 175(9):1570-1603, 2011.
[15] Dieqiao Feng, Carla P. Gomes, and Bart Selman. Solving hard AI planning instances using curriculum-driven deep reinforcement learning. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 2198-2205. ijcai.org, 2020.
[16] Patrick Ferber, Malte Helmert, and Jörg Hoffmann. Neural network heuristics for classical planning: A study of hyperparameter space. In *ECAI*, pages 2346–2353. IOS Press, 2020.
[56] Xuemei Wang. Learning planning operators by observation and practice. In Kristian J. Hammond, editor, AIPS, pages 335-340. AAAI, 1994.
[57] Christopher Wilt and Wheeler Ruml. When does weighted A* fail? In International Symposium on Combinatorial Search, volume 3, 2012.
[58] Christopher Makoto Wilt and Wheeler Ruml. Effective heuristics for suboptimal best-first search. J. Artif. Intell. Res., 57:273-306, 2016.
[59] Yuehua Xu, Alan Fern, and Sungwook Yoon. Learning linear ranking functions for beam search with application to planning. Journal of Machine Learning Research, 10(7), 2009.
[60] Ryo Yonetani, Tatsunori Taniai, Mohammadamin Barekatain, Mai Nishimura, and Asako Kanezaki. Path planning using neural A* search. In International Conference on Machine Learning, pages 12029-12039. PMLR, 2021.
[61] Ziqi Zhang and Florian GeiBer. Extending graph neural networks for generalized stochastic planning. In Bridging the Gap Between AI Planning and Reinforcement Learning. 2021.
[62] Tan Zhi-Xuan. PDDL. jl: An Extensible Interpreter and Compiler Interface for Fast and Flexible AI Planning. PhD thesis, Massachusetts Institute of Technology, 2022.
[63] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adbelief optimizer: Adapting step sizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
8.1 Speed of convergence against the size of training set:
We use the classical results of statistical learning theory to show that estimating the cost-to-go converges more slowly with respect to the size of the training set than estimating the rank.
From Equation (1) in the main text it should be obvious that ranking is in its essence a classification problem if one considers a pair of states  $(s, s')$  as a single sample. For classification/ranking problems, a following bound on true error rate [54] holds with probability  $1 - \eta$
$$
R _ {c} \leq \hat {R} _ {c} + \sqrt {\frac {1}{n _ {p}} \left[ \left(1 + \kappa \ln \frac {2 n _ {p}}{\kappa}\right) - \ln \eta \right]},
$$
where  $R_{c}$  denotes the true loss (Equation (1) in the main text) and  $\hat{R}_{c}$ , its estimate from  $n_p$  state pairs  $(s_i, s_j)$ , and  $\kappa$ , the Vapnik-Chervonenkis dimension [54] of the hypothesis space.
Optimizing  $h(s, \theta)$  with respect to cost-to-goal (Equation (5) in the main text) is a regression problem for which a different generalization bound on prediction error [7] holds with probability  $1 - \eta$
$$
R _ {r} \leq \hat {R} _ {r} \left[ 1 - \sqrt {\frac {1}{n _ {s}} \left[ \kappa \left(1 + \ln \frac {n _ {s}}{\kappa}\right) - \ln \eta \right]} \right] _ {+} ^ {- 1}
$$
where again  $R_{r}$  is the error of the estimator of cost-to-go and  $\hat{R}_{r}$  is its estimate from  $n_{s}$  states (Equation (5) in the main text), $^2$  and  $[x]_{+}$  is a shorthand for  $\max \{0, x\}$ .
From the above, we can see that excess error in the ranking case converges to zero at a rate  $\frac{\sqrt{\ln n_p}}{\sqrt{n_p}}$ , which is slightly faster than that of regression  $\frac{\sqrt{\ln n_s}}{\sqrt{n_s} - \sqrt{\ln n_s}}$ . But, the number of state pairs in the training set grows quadratically with the number of states; therefore, the convergence rate of the ranking problem for the number of states  $n_s$  can be expressed as  $\frac{\sqrt{2\ln n_s}}{n_s}$ , which would be by at least  $\frac{1}{\sqrt{n_s}}$  factor faster than that of regression. We note that bounds are illustrative, since the independency of samples is in practice violated, since samples from the same problem-instance are not independent.
8.2 Suboptimality of perfect heuristic in GBFS
Figure 2: Problem instance where perfect heuristic is not strictly optimally efficient with GBFS. Numbers on the edges denote the cost of action and red numbers next to nodes denote the minimal cost-to-go.
Example 2. While cost-to-goal  $h^*$  is the best possible heuristic for algorithms like  $A^*$  (up to tie-breaking) in terms of nodes being expanded, for GBFS,  $h^*$  does not necessarily yield optimal solutions.
Proof. A* explores all the nodes with  $f(s) < f^{*}(s) = g(s) + h^{*}(s)$  and some with  $f(s) = f^{*}(s) = g(s) + h^{*}(s)$ , so nodes can only be saved with optimal  $f^{*}(s) = g(s) + h^{*}(s)$ . In fact, when given
2The formulation in [7] contains constants  $c$  and  $a$ , but authors argue they can be set to  $a = c = 1$  and hasubve been therefore omitted here.
$h^*$  as the heuristic, only nodes  $s$  with  $f(s) = f^{*}(s)$  are expanded. Depending on the strategy of tie-breaking, the solution path can be found in the minimal number of node expansions or take significantly longer (e.g., in lower  $g$ -value first exploration of the search frontier). Any heuristic other than  $h^*$  is either overestimating, and, therefore, may lead to either non-optimal solutions in  $\mathbf{A}^*$ , or weaker than  $h^*$ , leading to more nodes being expanded.
Even if  $h^*$  is given, GBFS is not guaranteed to be optimal. Consider the following graph with five nodes  $A, B, C, D, E$ , and  $w(A, B) = 8$ ,  $w(B, E) = 3$ ,  $w(A, C) = 2$ ,  $w(C, D) = 4$ ,  $w(D, E) = 4$ , and  $h^*(A) = 10$ ,  $h^*(B) = 3$ ,  $h^*(C) = 8$ ,  $h^*(D) = 4$ ,  $h^*(E) = 0$  (see Figure 2), initial node  $s_0 = A$ , goal node  $E \in S^*$ . The numbers are the actual costs and the red numbers are the exact heuristic function. For finding a path from node  $A$  to node  $E$ , GBFS would return  $(A, B, E)$  following the heuristic function. However, the path  $(A, C, D, E)$  has cost 10 instead of 11.
8.3 Theory
Below, we prove the claim in made in Experimental section stating that if a heuristic  $h$  is strictly optimally efficient for  $\mathrm{A}^*$  search, then it is also strictly optimally efficient for GBFS.