SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Leah Chrestien
Czech Technical University in Prague leah.chrestien@aic.fel.czut.cz
Tomáš Pevný
Czech Technical University in Prague, and Gen Digital, Inc.  
pevnytom@fel.czut.cz
Stefan Edelkamp
Czech Technical University in Prague edelkste@fel.czut.cz
Antonín Komenda
Czech Technical University in Prague antonin.komenda@fel.cyut.cz
In imitation learning for planning, parameters of heuristic functions are optimized against a set of solved problem instances. This work revisits the necessary and sufficient conditions of strictly optimally efficient heuristics for forward search algorithms, mainly  $\mathrm{A}^*$  and greedy best-first search, which expand only states on the returned optimal path. It then proposes a family of loss functions based on ranking tailored for a given variant of the forward search algorithm. Furthermore, from a learning theory point of view, it discusses why optimizing cost-to-goal  $h^*$  is unnecessarily difficult. The experimental comparison on a diverse set of problems unequivocally supports the derived theory.
Automated planning finds a sequence of actions that will reach a goal in a model of the environment provided by the user. It is considered to be one of the core problems in Artificial Intelligence and it is behind some of its successful applications [39, 27, 46]. Early analysis of planning tasks [31] indicated that optimizing the heuristic function steering the search for a given problem domain towards the goal can dramatically improve the performance of the search. Automated optimization of the heuristic function therefore becomes a central request in improving the performance of planners [6, 13].
Learning in planning means optimizing heuristic functions from plans of already solved problems and their instances. This definition includes the selection of proper heuristics in a set of pattern databases [17, 22, 33, 12], a selection of a planner from a portfolio [25], learning planning operators from instances [32, 56], learning for macro-operators and entanglements [9, 29], and learning heuristic functions by general function approximators (e.g. neural networks) [45, 19, 16, 5].
The majority of research [45, 53, 19, 16] optimizes the heuristic function to estimate the cost-to-goal, as it is well known that it is an optimal heuristic for many best-first heuristic search algorithms including A* or IDA* [21, 28]. These works optimize the heuristic function to solve regression problems. But even a true cost-to-goal  $h^*$  does not guarantee that the forward search will find the optimal solution while expanding the minimal number of states. This paper defines a stricter version of optimally efficiency [10, 24] as follows. Forward search (or its heuristic) is called strictly optimally efficient iff it expands only states on one optimal solution path returned by the search, i.e., if this optimal solution path has  $l + 1$  states, the search will expand only  $l$  states.
This work focuses exclusively on a forward search with merit functions. It presents theoretical arguments as to why the order (rank) of states provided by the heuristic function is more important
than the precise estimate of the cost-to-goal and it formalizes the necessary and sufficient condition for strictly optimally efficient search. It also argues why learning to rank is a simpler problem than regression estimating the cost-to-goal from a statistical learning theory point of view. The main motivation of the paper is similar to that in [38], but the derived theory and implementation are much simpler. The sufficiency of optimizing the rank has been already recognized in [59] for a beam-search and in [18] for greedy best-first search (GBFS), though, unlike this work, it cannot guarantee strict optimal efficiency.
We emphasize that this work neither deals with nor questions the existence of a strictly optimally efficient heuristic function with the forward search. It is assumed that the space of possible heuristic functions is sufficiently large, such that there exist one that is sufficiently good. The theoretical results are supported by experimental comparisons to the prior art on eight domains.
The contributions of this paper are as follows.
1. We state necessary and sufficient conditions for strictly optimally efficient heuristic functions for forward search algorithms in deterministic planning.
2. We show that when optimizing a heuristic function, it is better to solve the ranking problem.
3. We argue, why learning to rank is easier than regression used in learning the cost-to-goal.
4. We instantiate it for  $\mathbf{A}^*$  and GBFS searches leading to two slightly different loss functions.
5. We experimentally demonstrate on eight problems (three grid and five PDDL) that optimizing rank is always better.
A search problem instance is defined by a directed weighted graph  $\Gamma = \langle S, \mathcal{E}, w \rangle$ , a distinct node  $s_0 \in S$  and a distinct set of nodes  $S^* \subseteq S$ . The nodes  $S$  denote all possible states  $s \in S$  of the underlying transition system representing the graph. The set of edges  $\mathcal{E}$  contains all possible transitions  $e \in \mathcal{E}$  between the states in the form  $e = (s, s')$ .  $s_0 \in S$  is the initial state of the problem instance and  $S^* \subseteq S$  is a set of allowed goal states. Problem instance graph weights (alias action costs) are mappings  $w: \mathcal{E} \to \mathbb{R}^{\geq 0}$ .
A path (alias a plan)  $\pi = (e_1, e_2, \ldots, e_l)$ ,  $e_i = (s_{i-1}, s_i) \in \mathcal{E}$ , of length  $l$  solves a task  $\Gamma$  with  $s_0$  and  $S^*$  iff  $\pi = ((s_0, s_1), (s_1, s_2), \ldots, (s_{l-1}, s_l))$  and  $s_l \in S^*$ . An optimal path  $\pi^*$  is defined as a path minimizing the cost of a problem instance  $\Gamma$ ,  $s_0$ ,  $S^*$ , its value as  $f^* = w(\pi^*) = \sum_{i=1}^{l} w(e_i)$ . When cost function  $w(e) = 1$  optimal path corresponds to the shortest one.  $\pi_{:i} = ((s_0, s_1), (s_1, s_2), \ldots, (s_{i-1}, s_i))$  denotes subplan of a plan  $\pi$  containing its first  $i$  actions.  $S^{\pi} = \{s_i\}_{i=0}^{l}$  denotes the set of states from the path  $\pi = ((s_0, s_1), (s_1, s_2), \ldots, (s_{l-1}, s_l))$ , and correspondingly  $S^{\pi_{:i}}$  denotes the set of states in a subplan  $\pi_{:i}$ .
2.1 Forward search algorithm
To align notation, we briefly review the forward search algorithm, of which  $\mathbf{A}^*$  and GBFS are special cases. As we apply full duplicate detection in all of our algorithms, there is a bijection of search tree nodes to planning states. Forward Search for consistent heuristics, where  $h(s) - h(s') \leq w(s, s')$  for all edges  $(s, s')$  in the  $w$ -weighted state space graph, mimics the working of Dijkstra's shortest-path algorithm [11]. It maintains two sets: the first called Open list,  $\mathcal{O}$ , contains generated but not expanded nodes; the second called Closed list,  $\mathcal{C}$ , contains already expanded nodes. Parameters  $\alpha$  and  $\beta$  of a merit function  $f(s) = \alpha g(s) + \beta h(s)$  used to sort states in Open list allow to conveniently describe different variants of the algorithm as described in detail shortly below. The forward search works as follows.
1. Initialise Open list as  $\mathcal{O}_0 = \{s_0\}$ .
2. Set  $g(s_0) = 0$
3. Initiate the Closed list to empty, i.e.  $\mathcal{C}_0 = \emptyset$
4. For  $i \in 1, \ldots$  until  $\mathcal{O}_i = \emptyset$
(a) Select the state  $s_i = \arg \min_{s\in \mathcal{O}_{i - 1}}\alpha g(s) + \beta h(s)$
(b) Remove  $s_i$  from  $\mathcal{O}_{i - 1}$ ,  $\mathcal{O}_i = \mathcal{O}_{i - 1} \setminus \{s_i\}$
(c) If  $s_i \in S^*$ , i.e. it is a goal state, go to 5.
(d) Insert the state  $s_i$  to  $\mathcal{C}_{i - 1}$ ,  $\mathcal{C}_i = \mathcal{C}_{i - 1} \cup \{s_i\}$
(e) Expand the state  $s_i$  into states  $s'$  for which hold  $(s_i, s') \in \mathcal{E}$  and for each
i. set  $g(s') = g(s_i) + w(s_i, s')$
ii. if  $s'$  is in the Closed list as  $s_c$  and  $g(s') < g(s_c)$  then  $s_c$  is reopened (i.e., moved from the Closed to the Open list), else continue with (e)
iii. if  $s'$  is in the Open list as  $s_o$  and  $g(s') < g(s_o)$  then  $s_o$  is updated (i.e., removed from the Open list and re-added in the next step with updated  $g(\cdot)$ ), else continue with (e)
iv. add  $s'$  into the Open list
5. Walk back to retrieve the solution path.
In the above algorithm,  $g(s)$  denotes a function assigning an accumulated cost  $w$  for moving from the initial state  $(s_0)$  to a given state  $s$ . During its execution, it always expands nodes with the lowest  $f(s) = \alpha g(s) + \beta h(s)$ . Different settings of  $\alpha$  and  $\beta$  give rise to different algorithms: for  $A^*$ ,  $\alpha = \beta = 1$ ; for GBFS  $\alpha = 0$  and  $\beta = 1$ .
Consistent heuristics, which are of special interest, are called monotone because the estimated cost of a partial solution  $f(s) = g(s) + h(s)$  is monotonically non-decreasing along the best path to the goal. More than this,  $f$  is monotone on all edges  $(s, s')$ , if and only if  $h$  is consistent as we have  $f(s') = g(s') + h(s') \geq g(s) + w(s, s') + h(s) - w(s, s') = f(s)$ . For the case of consistent heuristics, no reopening (moving back nodes from Closed to Open) in  $A^*$  is needed, as we essentially traverse a state-space graph with edge weights  $w(s, s') + h(s') - h(s) \geq 0$ . For the trivial heuristic  $h_0$ , we have  $h_0(s) = 0$  and for perfect heuristic  $h^*$ , we have  $f(s) = f^* = g(s) + h^*(s)$  for all nodes  $s$ . Both heuristics  $h_0$  and  $h^*$  are consistent. For GBFS and other variants, reopening is usually neglected. Even if the heuristic is not consistent, best-first algorithms without the reopening, remain complete i.e., they find a plan if there is one. Plans might not be provably optimal but are often good for planning practice [14].
3 Conditions on strictly optimally efficient heuristic
Let  $\mathcal{O}_i$  be an Open list in the  $i^{\mathrm{th}}$  iteration of the forward search expanding only states on the optimal path  $\pi$ . Below definition defines a perfect ranking heuristic as a heuristic function where state on some optimal path in the Open list,  $S^{\pi : i} \cap \mathcal{O}_i$ , has always strictly lower merit value than other states in the Open list off the optimal path,  $\mathcal{O}_i \setminus S^{\pi : i}$ . The intersection  $S^{\pi : i} \cap \mathcal{O}_i$  contains always exactly one state.
Definition 1 (Perfect ranking heuristic). A heuristic function  $h(s)$  is a perfect ranking in forward search with a merit function  $f(s) = \alpha g(s) + \beta h(s)$  for a problem instance  $\Gamma = (\langle S, \mathcal{E}, w \rangle, s_0, S^*)$  if and only if there exists an optimal plan  $\pi = ((s_0, s_1), (s_1, s_2), \ldots, (s_{l-1}, s_l))$  such that
-  $g(s)$  is the cost from  $s_0$  to  $s$  in a search-tree created by expanding only states on the optimal path  $\pi$ ;
-  $\forall i\in \{1,\ldots ,l\}$  and  $\forall s_j\in \mathcal{O}_i\setminus \mathcal{S}^{\pi ;i}$  we have  $f(s_{j}) > f(s_{i})$
As an example consider a search-tree in Figure 1a with an optimal path  $((s_0, s_1), (s_1, s_2), (s_2, s_3))$  and states  $\{s_4, s_5, s_6, s_7\}$  off the optimal path. Then the Open list after expanding  $s_0$  is  $\mathcal{O}_1 = \{s_1, s_4, s_5\}$  and that after expanding  $s_1$  is  $\mathcal{O}_2 = \{s_2, s_4, s_5, s_6, s_7\}$ . The set of states on the optimal sub-path are  $S^{\pi:1} = \{s_0, s_1\}$ ,  $S^{\pi:2} = \{s_0, s_1, s_2\}$ . States on the optimal path in the Open lists are  $S^{\pi:1} \cap \mathcal{O}_1 = \{s_1\}$  and  $S^{\pi:2} \cap \mathcal{O}_2 = \{s_2\}$ . States off the optimal path in the Open lists are  $\mathcal{O}_1 \setminus S^{\pi:1} = \{s_4, s_5\}$  and  $\mathcal{O}_2 \setminus S^{\pi:2} = \{s_4, s_5, s_6, s_7\}$ . Function  $f$  is perfectly ranking problem instance in Figure 1a iff following inequalities hold:
$$
f (s _ {1}) <   f (s _ {4}), \quad f (s _ {2}) <   f (s _ {4}), \quad f (s _ {2}) <   f (s _ {6}),
$$
$$
f (s _ {1}) <   f (s _ {5}), \quad f (s _ {2}) <   f (s _ {5}), \quad f (s _ {2}) <   f (s _ {7}).
$$
Notice that the heuristic values of states on the optimal path are never compared. This is because if the forward search expands only states on the optimal path, then its Open list always contains only
one state from the optimal path. The definition anticipates the presence of multiple optimal solutions and even multiple goals, which requires the heuristic to break ties and have a perfect rank with respect to one of them.
Theorem 1. The forward search with a merit function  $f(s) = \alpha g(s) + \beta h(s)$  and a heuristic  $h$  is strictly optimally efficient on a problem instance ( $\langle S, \mathcal{E}, w \rangle, s_0, S^*$ ) if and only if  $h$  is a perfect ranking on it.
Proof. Sufficiency: If the conditions of a perfect ranking heuristic hold, then there exists an optimal path such that a state on it that is in the Open list always has the strictly lowest heuristic value. Therefore, forward search will never move off the optimal path and is, therefore, strictly optimally efficient.
Necessity: If  $h$  is a strictly optimally efficient heuristic, then the forward search always selects the state on the optimal path, which means that the state has the lowest heuristic value of all the states in the Open list, which is precisely the condition of a perfect ranking heuristic.
Theorem 1, despite being trivial, allows certifying that the forward search with a given heuristic in a given problem instance is strictly optimally efficient. This property has been discussed in [59] for Beam search, but the conditions stated there are different because Beam search prunes states in the Open list. Recall that the complexity class of computing a perfect ranking heuristic function is the same as solving the problem because one implies the other. We now remind the reader that the popular cost-to-goal does not always lead to a strictly optimally efficient forward search.
Example 1. While cost-to-goal  $h^*$  is the best possible heuristic for algorithms like  $A^*$  (up to tie-breaking) in terms of nodes being expanded, for GBFS,  $h^*$  does not necessarily yield optimal solutions.
See Figure 2 for a counterexample. The complete proof is in the Appendix. Appendix 8.4 also gives an example of a problem instance, where an optimally efficient heuristic for GBFS returning the optimal solution does not exist.
4 Loss functions
4.1 Ranking loss function
Let's now design a loss function minimizing the number of violated conditions in Definition 1 for a problem instance  $(\Gamma, s_0, S^*)$  and its optimal plan  $\pi$ . Assuming a heuristic function  $h(s, \theta)$  with parameters  $\theta \in \Theta$ , the number of violated conditions can be counted as
$$
\mathrm {L} _ {0 1} (h, \Gamma , \pi) = \sum_ {s _ {i} \in \mathcal {S} ^ {\pi}} \sum_ {s _ {j} \in \mathcal {O} _ {i} \backslash \mathcal {S} ^ {\pi : i}} [   [ r (s _ {i}, s _ {j}, \theta) > 0 ]   ], \tag {1}
$$
$$
r \left(s _ {i}, s _ {j}, \theta\right) = \alpha \left(g \left(s _ {i}\right) - g \left(s _ {j}\right)\right) + \beta \left(h \left(s _ {i}, \theta\right) - h \left(s _ {j}, \theta\right)\right), \tag {2}
$$
$\mathcal{O}_i, S^{\pi : i}$  as used in Definition 1,  $[\cdot, \cdot]$  denotes Iverson bracket being one if the argument is true and zero otherwise, and  $\alpha$  and  $\beta$  are parameters controlling the type of search, as introduced above.
In imitation learning, we assume that we have a training set  $\mathcal{T}^{\mathrm{trn}}$  consisting of tuples of problem instances and optimal plans  $\mathcal{T}^{\mathrm{trn}} = \{(\langle S_i,\mathcal{E}_i,w_i\rangle ,s_{0,i},\bar{S}_i^*,\pi_i)\}_{i = 1}^n$ . To optimize parameters  $\theta$  of a heuristic function  $h(s,\theta)$ , we propose to minimize the number of wrongly ranked states over all the problem instances in the training set,
$$
\arg \min  _ {\theta \in \Theta} \sum_ {(\Gamma , \pi) \in \mathcal {T} ^ {\mathrm {t r n}}} \mathrm {L} _ {0 1} (h, \Gamma , \pi). \tag {3}
$$
In practice, one often wants to solve problem (3) by efficient gradient optimization methods. To do so, the Iverson bracket  $\llbracket \cdot \rrbracket$  (also called 0-1 loss) is usually replaced by a convex surrogate such as the hinge-loss or the logistic loss used in the Experimental section below, in which case Equation (3) becomes
$$
\arg \min  _ {\theta \in \Theta} \sum_ {(\Gamma , \pi) \in \mathcal {T} ^ {\mathrm {t r n}}} \sum_ {s _ {i} \in \mathcal {S} ^ {\pi}} \sum_ {s _ {j} \in \bar {\mathcal {S}} ^ {i}} \log (1 + \exp (r (s _ {i}, s _ {j}, \theta))). \tag {4}
$$
While optimization of the surrogate loss might look as if one optimizes a different problem, it has been shown that for certain classes of functions, optimization of surrogate losses solves the original problem (3) as the number of training samples approach infinity [50, 35].
4.2 Regression loss function
For the sake of comparison, we review a loss function used in the majority of works optimizing the cost-to-goal in imitation learning for planning (some notable exceptions are in the related work section). The optimization problem with  $\mathrm{L}_2$  loss function is defined as
$$
\underset {\theta \in \Theta} {\arg \min } \sum_ {(\Gamma , \pi) \in \mathcal {T} ^ {\mathrm {t r n}}} \sum_ {s _ {i} \in \mathcal {S} ^ {\pi}} (h \left(s _ {i}, \theta\right) - h ^ {*} \left(s _ {i}\right) ^ {2}, \tag {5}
$$
where  $h^{*}(s_{i})$  is the true cost-to-goal. The optimal solution of the optimization is the heuristic function  $h^*$ . What is important for the discussion below is that the  $\mathrm{L}_2$  loss function uses only states on the optimal path and the heuristic function is optimized to solve the regression problem. Other variants of regression loss, such as asymmetric  $\mathrm{L}_1$  in [51], do not solve issues of regression losses discussed in this paper.
4.3 Advantages and disadvantages of loss functions
Cost-to-goal provides a false sense of optimality. It has been already mentioned above that  $\mathrm{A}^*$  with  $h^*$  is strictly optimally efficient up to resolving states with the same value of  $g(s) + h^{*}(s)$ . This means that for problems with a large (even exponential) number of optimal solutions of equal cost (see [23] for examples),  $\mathrm{A}^*$  will be very inefficient unless some mechanism for breaking ties is used. Moreover, since learning is inherently imprecise, a heuristic close but not equal to  $h^*$  will likely be less efficient than the one close to a perfect-ranking heuristic.
Size of the solution set The set of all functions satisfying Equation (5) is likely smaller than that satisfying Equation (3), because the solution set of (3) is invariant to transformations by strictly monotone functions, unlike the solution set of (5). From that, we conjecture that finding the solution of (5) is more difficult. The precise theorem is difficult to prove since the solution of (5) is not a solution of (3) due to tie resolution in Definition 1.
Optimizing cost-to-goal does not utilize states off the solution path Assume a problem instance  $(\langle \mathcal{S},\mathcal{E},w\rangle ,s_0,\mathcal{S}^*)$  and an optimal solution path  $\pi = ((s_0,s_1),(s_1,s_2),\ldots ,(s_{l - 1},s_l))$ . Loss function optimized in (5) uses only states  $\{s_i\}_{i = 0}^l$  on the optimal path. This means that even when the loss is zero, the search can be arbitrarily inefficient if states  $S\backslash S^{\pi}$  off the optimal path will have heuristic values smaller than  $\min_{s\in S^{\pi}}h(s)$ . The proposed ranking loss (3) uses states on and off the optimal path and therefore, does not suffer from this problem. This issue can be fixed if the training set is extended to contain heuristic values for all states off the optimal path (states  $\{s_4,s_5,s_6,s_7\}$  in Figure 1a), as has been suggested in [5, 49]. With respect to the above definitions, this is equal to adding more problem instances to the training set sharing the same graph  $\Gamma$  and set of goal states  $S^{*}$  but differing in initial states. Solving all of them to obtain true cost-to-goal greatly increases the cost of creating the training set.
Heuristic value for dead-end states should be sufficiently high to ensure they are never selected. An infinity in the  $\mathrm{L}_2$  loss would always lead to an infinite gradient in optimization. In practice, the  $\infty$  is replaced by a sufficiently large value, but too large values can cause large gradients of  $\mathrm{L}_2$ , which might negatively impact the stability of convergence of gradient-based optimization methods. Proposed ranking losses do not suffer this problem.
Goal-awareness The drawback of the heuristic function  $h(s, \theta)$  obtained by optimizing ranking losses is that they are not goal-aware unlike the heuristic optimizing the estimate of cost-to-goal.
Speed of convergence against the size of training set From classical bounds on true error [54] and [7], we can derive (see Appendix for details) that the excess error of ranking loss converges to zero at a rate  $\frac{\sqrt{\ln n_p}}{\sqrt{n_p}}$ , which is slightly faster than that of regression  $\frac{\sqrt{\ln n_s}}{\sqrt{n_s} - \sqrt{\ln n_s}}$ , where  $n_s$  and  $n_p$  denotes the number of states and state pairs respectively in the training set. But, the number of state pairs in the training set grows quadratically with the number of states; therefore, the excess error of ranking loss for the number of states  $n_s$  can be expressed as  $\frac{\sqrt{2\ln n_s}}{n_s}$ , which would be by at least  $\frac{1}{\sqrt{n_s}}$  factor faster than that of regression. Note though that these rates are illustrative since the independency of samples (and sample pairs) is in practice, violated since even samples from the same problem instance are not independent.
Conflicts in training set Unlike regression loss, the proposed family of ranking losses is potentially sensitive to conflicts in the training set, which occur when the training set contains examples of two (or more) different solution paths of the same problem instance. This might prevent achieving a zero loss. The effect on the heuristic depends on the composition of the training set. Either solution paths more frequent in the training set will be preferred, or there will be ties, but their resolution do not affect optimally efficiency. Appendix 8.5 provides an illustrative example.
The closest work to ours is [36, 38], which adjusts heuristic value by a policy estimated from the neural network. The theory therein has the same motivation, to minimize the number of expanded states, though it is more complicated than the theory presented here. For efficiency, [38] needs to modify the GBFS such that search nodes store policy and probability of reaching the node from the start. Moreover, in some variants (used in the experimental comparison below) the neural network has to estimate both heuristic value and policy.
Inspired by statistical surveys of heuristic functions in [57, 58], [18] proposes to optimize the rank of states preserving order defined by the true cost-to-goal. The formulation is therefore valid only for GBFS and not for  $\mathrm{A}^*$  search. Ties are not resolved, which means the strict optimal efficiency cannot be guaranteed. Similarly, heuristic values for domains where all actions have zero cost cannot be optimized. Lastly and importantly, the need to know the true cost-to-goal for all states in the training set greatly increases the cost of its construction as discussed above. From Bellman's equation [49] arrives at the requirement that the smallest heuristic value of child nodes has to be smaller than that of the parent node. The resulting loss is regularized with  $\mathrm{L}_1$  forcing the value of the heuristic to be close to cost-to-goal.
In [5],  $\mathrm{A}^*$  is viewed as a Markov Decision Process with the Q-function equal to the number of steps of  $\mathrm{A}^*$  reaching the solution. This detaches the heuristic values from the cost-to-goal cost, but it does not solve the problem with dead-ends and ties, since the optimization is done by minimizing  $\mathrm{L}_2$  loss.