Title: Sample Complexity of Learning Heuristic Functions for Greedy-Best-First and A* Search
Authors: Shinsaku Sakaue
Taihei Oki
Venue: NeurIPS 2022
Keywords: sample complexity
data-driven algorithm design
heuristic search

Abstract: Greedy best-first search (GBFS) and A* search (A*) are popular algorithms for path-finding on large graphs. Both use so-called heuristic functions, which estimate how close a vertex is to the goal. While heuristic functions have been handcrafted using domain knowledge, recent studies demonstrate that learning heuristic functions from data is effective in many applications. Motivated by this emerging approach, we study the sample complexity of learning heuristic functions for GBFS and A*. We build on a recent framework called \textit{data-driven algorithm design} and evaluate the \textit{pseudo-dimension} of a class of utility functions that measure the performance of parameterized algorithms. Assuming that a vertex set of size $n$ is fixed, we present $\mathrm{O}(n\lg n)$ and $\mathrm{O}(n^2\lg n)$ upper bounds on the pseudo-dimensions for GBFS and A*, respectively, parameterized by heuristic function values. The upper bound for A* can be improved to $\mathrm{O}(n^2\lg d)$ if every vertex has a degree of at most $d$ and to $\mathrm{O}(n \lg n)$ if edge weights are integers bounded by $\mathrm{poly}(n)$. We also give $\Omega(n)$ lower bounds for GBFS and A*, which imply that our bounds for GBFS and A* under the integer-weight condition are tight up to a $\lg n$ factor. Finally, we discuss a case where the performance of A* is measured by the suboptimality and show that we can sometimes obtain a better guarantee by combining a parameter-dependent worst-case bound with a sample complexity bound.

TL;DR: We present upper and lower bounds on the pseudo-dimension for analyzing the sample complexity of learning heuristic functions of greedy-best-first and A* search.

Review discussion:
The paper is relatively straightforward, in the sense that it gives
clear questions and clear answers. It is well written, and explains
the weaknesses of the results, namely the relatively big gap between
the bounds on the pseudodimension of A*, as well as give some
explanation why it is hard to bridge them.
I don't see any major weaknesses. I would suggest that another
interesting direction here is looking at A* for planning. The graph is
obviously exponentially large, so the bounds here are useless, but it
has a compact representation (e.g. the STRIPS model). Could some
heuristics be learned efficiently in that setting?
Typos, etc:
Defn: you use t_1, ..., t_N for the values in the text and z_1...z_N
in the formula
107: disrtibution
154: gaurantees
None
No direct societal impact.
No
We appreciate the reviewer's careful reading and insightful comments. We respond to the following comment.
> I don't see any major weaknesses. I would suggest that another interesting direction here is looking at A* for planning. The graph is obviously exponentially large, so the bounds here are useless, but it has a compact representation (e.g. the STRIPS model). Could some heuristics be learned efficiently in that setting?
Thank you for the constructive suggestion. As the reviewer mentioned, some path-finding instances have extremely many vertices but have compact representations. Such representations are sometimes helpful in reducing the sample complexity. Below is an illustrative example.
&nbsp;
### An example of improving the upper bound with special heuristics
Suppose that a graph with $n$ vertices is given as a STRIPS model, where each vertex $v \in V$ corresponds to a state represented by a binary vector $\boldsymbol{q}\_v \in \\{0, 1\\}^\ell$.
Then, we have $n = 2^\ell$ vertices, i.e., there are exponentially many vertices in $\ell$. As an example of simple heuristic functions, we assume that a heuristic function value $\rho\_v$ for each $v \in V$ is given as $\rho\_v = \boldsymbol{q}\_v^\top \boldsymbol{\theta}$, where $\boldsymbol{\theta} \in \mathbb{R}^\ell$ is a vector of $\ell$ learnable parameters. If we apply GBFS to the graph, a similar discussion to the proof of Theorem 1 implies that the behavior of GBFS is determined by a total order on $n$ values $\\{ \boldsymbol{q}\_v^\top \boldsymbol{\theta} \\}\_{v \in V}$. Such a total order is unique for all $\boldsymbol{\theta} \in \mathbb{R}^\ell$ in an identical region whose boundaries are given by up to $\binom{n}{2}$ hyperplanes of form $(\boldsymbol{q}\_v - \boldsymbol{q}_{v'})^\top \boldsymbol{\theta} = 0$ for $v, v' \in V$.
Thus, Sauer's lemma implies that given any $N$ instances, there are at most $\left( \mathrm{e}\binom{n}{2} N \right)^\ell$ such regions. To shatter the $N$ instances, we need $\left( \mathrm{e}\binom{n}{2} N \right)^\ell = \Omega(2^N)$, implying an $\mathrm{O}(\ell \lg n) \simeq \mathrm{O}(\ell^2)$ upper bound on the pseudo-dimension.
That is, even though there are $n = 2^\ell$ vertices, the upper bound depends only polynomially on $\ell$. A similar result is true for A* if edge weights are integer, but A* for general weights does not enjoy such a $\mathrm{poly}(\ell)$ bound as it incurs an additional $\mathrm{O}(n)$ factor.
&nbsp;
The above example implies that we can sometimes reduce the sample complexity by using compact representations of graphs to design appropriate heuristic functions with fewer parameters. Studying how to design such appropriate heuristic functions would be an interesting and important future direction. We will elaborate more on this point in the final version.
Thank you for your answer. I encourage you to include a version of this in the final version, if accepted.
The paper is fairly well written and organised. The quality of the presentation is overall very good and therefore the paper is relatively easy to follow. Most of the concepts and technical details are introduced and discussed if a fairly clear manner.
I think the paper needs a more detailed running example.
Otherwise it's not very easy to follow the details especially for a reader who's not very familiar with this research area.
Minor comments:
- Definition 1: there is a typo, h(y_i) \geq t_i instead of h(y_i) \geq z_i
1. What are the practical implications of the proposed bounds? Many interesting domains for GBFS/A* search have exponentially large implicit search spaces (e.g., planning, probabilistic inference, games, etc.) which means that 'n' in this case could be extremely large and therefore one would require an exponentially large training dataset.
[Post Rebuttal] Thanks for your answers. They have clarified my concerns.
see above
No
We sincerely thank the reviewer for providing valuable comments. We answer the following question.
> What are the practical implications of the proposed bounds? Many interesting domains for GBFS/A\* search have exponentially large implicit search spaces (e.g., planning, probabilistic inference, games, etc.) which means that 'n' in this case could be extremely large and therefore one would require an exponentially large training dataset.
As the reviewer mentioned, our result is sometimes pessimistic since $n$ can be extremely large (although it is tight for GBFS and A* for integer edge-weight graphs due to the lower bound). If we encounter such huge graphs in practice, what we should consider next is how to design simple heuristic functions to reduce the sample complexity. Roughly speaking, our bounds depend on $n = |V|$ mainly because $n$ heuristic function values $\rho_v$ for all $v \in V$ are independently learnable.
Therefore, using simple heuristic functions with fewer learnable parameters than $n$ would be an effective workaround when $n$ is huge. For an example of such simple heuristic functions, please see the [response to Reviewer zGFs](https://openreview.net/forum?id=FurHLDnmC5v&noteId=87Oj0VKH2dc).
Further discussion on how to design such heuristic functions would go beyond the scope of our work since it requires more instance-specific analysis;
therefore, we have left it for future work. Our result on the fundamental case where all $\rho_v$ ($v \in V$) are learnable will be an important first step to understanding the theoretical aspect of learning-based heuristic search, such as [9, 11, 25, 29, 33].
Strengths are mathematical analysis of the sample complexity for learning heuristic functions for graph search tasks using GBFS/A*.
Weaknesses are that this analysis emphasizes theoretical aspects and missing practical implications of the upper bounds.
I don't know very well about PAC analysis, but following the steps in the paper, the claims all make sense.
Although the assumption is simplified to a very limited case, where the graph has the same terminal node
and the sampling procedure ends up visiting roughly all the nodes in the graph (n and N in the equations),
the upper and lower bounds may characterize the theoretical limits for sample efficiency for learning heuristics.
How to use these bounds when we learn heuristics for solving graph search problems?
Can you provide a concrete example with the theoretical bounds?
I think this work is not relevant to this section.
No
We are grateful to the reviewer for providing thoughtful comments. We answer the following question.
> I don't know very well about PAC analysis, but following the steps in the paper, the claims all make sense.
Although the assumption is simplified to a very limited case, where the graph has the same terminal node and the sampling procedure ends up visiting roughly all the nodes in the graph (n and N in the equations), the upper and lower bounds may characterize the theoretical limits for sample efficiency for learning heuristics.
> How to use these bounds when we learn heuristics for solving graph search problems? Can you provide a concrete example with the theoretical bounds?
We can use our PAC bounds similarly to those in statistical learning theory. In words, PAC bounds guarantee that the expected performance on future instances becomes closer to the empirical performance observed on $N$ sampled instances as $N$ grows.
For example, suppose that path-finding instances on graphs with $n$ vertices are drawn from an unknown distribution. We consider accelerating GBFS/A* applied to those instances by learning good heuristic functions. As is often the case in practice, GBFS/A* with empirically good heuristics runs very fast.
However, this empirical observation alone provides no theoretical guarantee on how fast it can be on future instances drawn from the unknown distribution (particularly because learning of heuristic functions may result in overfitting to sampled instances).
In such situations, we can use our PAC bounds to guarantee the expected running time on future instances. Roughly speaking, with high probability, the expected running time on future instances can be bounded by the observed empirical running time plus about $\mathrm{O}\left(\sqrt{\frac{\mathrm{poly}(n)}{N}}  \right)$, where $\mathrm{poly}(n) = n$ for GBFS and $n^2$ for A*.
Thus, if we are given about $N \gtrsim \frac{\mathrm{poly}(n)}{\epsilon^2}$ instances, the expected deviation from the empirical running time is at most $\epsilon$.
In summary, we can use PAC bounds to translate the empirical performance observed on sampled instances into theoretical bounds on the expected performance on future instances.
Thanks for your answer, and
I look forward to reading "An example of improving the upper bound with special heuristics," which may appear in the final version. Initially, I thought this result was pessimistic, but there seems to be some room for practical impact.

Meta-review:
role: AC
data: metareview: Strong paper studying the sample complexity of learning heuristic functions for GBFS and A*. The reviewers were especially impressed with the theoretical results and find the paper a worthwhile contribution to this conference.
role: PC

Decision: Accept