SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, suppose that we have a problem where there is only one terminal state and sequence of low-level actions that reaches this terminal state, and the search space  $S$  is infinite. Let us denote this trajectory as  $\mathcal{T} = (s_0, a_0, s_1, a_1, \ldots, s_{N-1}, a_{N-1}, s_N)$ . Additionally, assume that the high-level policy  $\pi_{\mathrm{SG}}$  never guides the search towards a solution, that is, for each state  $s \in \mathcal{T}$ , the low-level action sequence for every subgoal  $s_g$  proposed by the generator network contains an incorrect action. Furthermore, assume that the generator network can always generate new, reachable subgoals for all states  $s \in S$ . Hence, only actions selected by the low-level policy lead to the solution, and thus  $\pi(n^*) = \varepsilon^N \prod_{i=0}^{N-1} \pi_{\mathrm{BC}}(a_i | s_i)$ . If  $\varepsilon = 0$  or  $\varepsilon \to 0$ , the search does not converge, and the upper bound in (5) approaches infinity.
Czechowski et al. [4] argue that low-level search suffers more from local noise than subgoal search and using subgoals improves the signal-to-noise ratio of the search, and the results of Kujanpää et al. [18] support this. For complete search, a lower  $\varepsilon$  can increase the use of high-level policy in relation to the low-level actions, thereby shortening the effective planning horizon and making the search more effective. Hence, we hypothesize that a low value of  $\varepsilon$  is generally preferred. However, (6) shows that the worst-case performance of the search deteriorates when the value of  $\varepsilon$  decreases.
If there is no heuristic, the bound can be simplified similarly as in Orseau and Lelis [27]. This following corollary shows that removing the heuristic factor does not affect the inverse dependence of the worst-case search loss on  $\varepsilon$ , and in fact, the worst case of (6), that is, the search failing for  $\varepsilon \rightarrow 0$ , is possible even given a perfect heuristic function.
Corollary 2. (Subgoal-PHS upper bound without heuristic). If there is no heuristic, that is,  $\forall n, \eta(n) = 1$ , the upper bound simplifies to
$$
L \left(P H S - h, n ^ {*}\right) \leq L \left(P H S - h, \hat {n}\right) \leq \frac {g (\hat {n})}{\varepsilon^ {N} \prod_ {i = 0} ^ {N - 1} \pi_ {B C} \left(a _ {i} \mid s _ {i}\right)} \tag {7}
$$
Proof. The first inequality follows as in Corollary 1, and the second inequality follows from the assumptions and  $\sum_{n' \in \mathcal{L}_{\varphi}(n)} \pi(n') \leq 1$  for all  $n$ , see Orseau et al. [28].
The main objective of our experiments is to evaluate whether the proposed complete search algorithm HIPS- $\varepsilon$  can be combined with an existing subgoal search algorithm, HIPS [18], to improve its performance in environments that require long-term planning and complex, object-based relational reasoning. We also analyze the sensitivity of HIPS- $\varepsilon$  to the hyperparameter  $\varepsilon$  and the impact of the
Table 1: The success rates (\%) after performing  $N$  node expansions for different subgoal search algorithms with access to environment dynamics. For HIPS- $\varepsilon$ , we use the value of  $\varepsilon$  that yields in the best performance:  $\varepsilon \rightarrow 0$  for Sokoban,  $\varepsilon = 10^{-5}$  for Sliding Tile Puzzle,  $\varepsilon = 10^{-3}$  for Box-World and  $\varepsilon \rightarrow 0$  for Travelling Salesman Problem. HIPS corresponds to HIPS-env in [18] and uses PHS* as the search algorithm in all environments.
Sokoban	Sliding Tile Puzzle
N	50	100	200	∞	50	100	200	∞
PHS* (low-level search)	0.2	2.4	16.2	100	0.0	0.0	0.0	100
HIPS (high-level search)	82.0	87.8	91.6	97.9	8.7	56.8	86.3	95.0
AdaSubS (high-level search)	76.4	82.2	85.7	91.3	0.0	0.0	0.0	0.0
kSubS (high-level search)	69.1	73.1	76.3	90.5	0.7	79.9	89.8	93.3
HIPS-ε (complete search)	84.3	89.5	93.1	100	18.5	69.5	93.8	100
Box-World	Travelling Salesman Problem
N	5	10	30	∞	20	50	100	∞
PHS* (low-level search)	0.0	0.1	2.2	100	0.0	0.0	0.0	100
HIPS (high-level search)	86.3	97.9	99.9	99.9	19.6	88.1	97.7	100
AdaSubS (high-level search)	0.0	0.0	0.6	21.2
kSubS (high-level search)	0.0	1.5	40.4	87.9
HIPS-ε (complete search)	89.7	98.9	100	100	17.9	87.4	97.9	100
novel heuristic factor and the corresponding  $\mathrm{PHS^{*}}$  evaluation function (4). The OOD-generalization abilities of HIPS- $\varepsilon$  are also discussed. We use the four environments considered in [18]: Box-World [44], Sliding Tile Puzzle [17], Gym-Sokoban [37] and Travelling Salesman (see Fig. 2).
We implement HIPS- $\varepsilon$  as proposed in (2) and the policy  $\pi_{\mathrm{BC}}(a|s)$  as a ResNet-based network [10] and train it using Adam [16]. We use the implementation of HIPS from the original paper and use PHS* (or our complete variant thereof) as the search algorithm for HIPS and HIPS- $\varepsilon$ . In [18], HIPS was evaluated with GBFS and A* on some environments. HIPS- $\varepsilon$  experiments with these search algorithms can be found in Appendix F. We use the versions of HIPS- $\varepsilon$  and HIPS that have access to the environment dynamics (named HIPS-env in [18]) unless stated otherwise.
In Table 1, we report the proportion of solved puzzles as a function of the number of expanded nodes for the low-level search  $\mathrm{PHS}^*$ , high-level search algorithms HIPS [18], kSubS [4], and AdaSubS [45], and our HIPS- $\varepsilon$ , which is HIPS enhanced with the proposed complete subgoal search. We used the kSubS and AdaSubS results from [18] where they were not evaluated on Box-World. The results show that high-level search alone does not guarantee completeness, which means that for some puzzles the solution is not found even without a limit on the number of expanded nodes ( $N = \infty$ ) due to the generator failing to propose subgoals that lead to a valid terminal state, a phenomenon observed in prior work [4, 18, 45]. As expected, HIPS- $\varepsilon$  achieves a  $100\%$  success rate in all environments thanks to augmenting the subgoals with low-level actions. In addition to the number of search node expansions, we analyzed the search cost in terms of wall-clock time and environment steps. HIPS- $\varepsilon$  generally outperforms the baselines also using these metrics. The results of these experiments can be found in Appendices J and K.
We also evaluated HIPS- $\varepsilon$  on Sokoban, STP, and TSP with learned transition models and compared it to HIPS with learned models, behavioral cloning (BC), Conservative Q-learning [CQL, 19], and Decision Transformer [DT, 3]. We omitted Box-World due to HIPS struggling with the learned dynamics [18]. The results are shown in Table 2. HIPS- $\varepsilon$  outperforms the baselines but can fail to find a solution to a solvable problem instance due to the learned dynamics model sometimes outputting incorrect transitions. Completeness can be attained by using the true dynamics for validating solutions and simulating low-level actions, while still minimizing the number of environment steps (see Appendix E). For complete results with confidence intervals, please see Appendix G.
Interestingly, the results in Table 1 show that augmenting the high-level search with low-level actions may have a positive impact on the performance for the same budget of node expansions  $N$ : we observe statistically significantly increased success rates for the same  $N$  in all environments except
Table 2: The success rates (\%) of different algorithms without access to environment dynamics in Sokoban, Sliding Tile Puzzle, and TSP. HIPS- $\varepsilon$  outperforms the baselines and can solve  $100\%$  of the puzzles when the environment dynamics are easy to learn.
HIPS	HIPS-ε	BC	CQL	DT
Sokoban	97.5	100.0	18.7	3.3	36.7
Sliding Tile Puzzle	94.7	100.0	82.5	11.7	0.0
Travelling Salesman	100.0	100.0	28.8	33.6	0.0
Table 3: The success rates  $(\%)$  after  $N$  node expansions in Sliding Tile Puzzle computed only on problem instances solvable by HIPS.  $\langle N\rangle$  is the mean number of expansions needed to find a solution.
N	50	75	100	200	500	〈N〉
HIPS	9.2	38.2	59.8	90.8	99.7	108.9
HIPS-ε, ε = 10-5	17.3	48.4	68.2	93.5	99.9	95.6
in the Travelling Salesman, where the results are approximately equal. Next, we analyze whether the increased number of solved puzzles is only because of the high-level search failing to find a solution to some puzzles. In Table 3, we consider the Sliding Tile Puzzle problem and report the same results as in Table 1 considering only the puzzles solvable by HIPS. The results suggest that the complete search can speed up finding a correct solution.
In Fig. 3, we analyze the effect of the hyperparameter  $\varepsilon$  on the number of solved puzzles. We report the ratio of the number of unsolved puzzles by HIPS- $\varepsilon$  to the number of unsolved puzzles by HIPS as a function of  $N$ , the number of node expansions. We observe that HIPS- $\varepsilon$  outperforms HIPS in all environments except TSP, and extensive hyperparameter tuning to select a good value of  $\varepsilon$  is rarely necessary. The plots also support our hypothesis about a small value of  $\varepsilon$  being often better despite the worse worst-case performance, as  $\varepsilon = 10^{-3}$  is often superior to  $\varepsilon = 0.1$ . The augmented search strategy suggested by Zawalski et al. [45] (which we denote as  $\varepsilon \rightarrow 0$ ) is sometimes but not always the best strategy. We analyze the impact of the value of  $\varepsilon$  in Appendix I.
Generalization to Out-of-Distribution Puzzles In the following experiment, we demonstrate one of the most attractive properties of complete search: its ability to enhance the high-level search policy to transfer to new (potentially more complex) search problems. We evaluate HIPS- $\varepsilon$  on Box-World in which the following changes have been made in comparison to the trajectories in  $\mathcal{D}$ : the number of distractor keys has been increased from three to four, and the colors of distractors changed such that there are longer distractor trajectories that lead into dead-end states, making it more difficult to detect distractor keys. After these changes to the environment, the share of puzzles HIPS can solve drops from over  $99\%$  to approximately  $66\%$  (see Fig. 4). However, the  $100\%$  solution rate can be retained by performing complete search. Note here that higher values of  $\varepsilon$  seem superior to lower values when the generator is worse, which is aligned with our analysis in Section 3.4.
Role of the Heuristic Factor Orseau and Lelis [27] observed that for pure low-level search, using a learned heuristic in addition to a search policy  $\pi$  can lead to significantly improved search performance. We evaluated the performance of HIPS- $\varepsilon$  with different heuristic factors and corresponding PHS evaluation functions. We compared the heuristic factor and evaluation function (4) proposed by us, the Levin-TS inspired evaluation function  $\varphi_{\mathrm{LevinTS}} = g(n) / \pi(n)$  that does not use a heuristic, and
Table 4: The success rates  $(\%)$  of HIPS-  $\varepsilon$  with different PHS evaluation functions. For all environments, we use  $\varepsilon = 10^{-3}$ . Search without heuristic fails in TSP due to running out of memory.
Sokoban	Sliding Tile Puzzle	Box-World	TSP
N	50	100	200	50	100	200	5	10	20	50
\( \hat{\varphi}_{\hat{h}} \) (ours)	82.5	88.8	92.9	16.3	68.6	93.7	89.0	99.1	18.3	82.4
\( \varphi_{LevinTS} \)	66.4	80.1	88.8	0.0	0.0	0.5	27.4	66.3	N/A	N/A
\( \varphi_{depth} \)	71.8	83.6	91.5	0.0	0.2	1.2	37.3	75.8	0.0	0.0
\( \varphi_{dist} \)	68.6	81.3	88.7	0.0	0.1	0.7	30.4	70.0	0.0	0.0
Figure 3: The ratio of the number of unsolved puzzles to the number of unsolved puzzles by HIPS as a function of the number of node expansions  $N$  (x-axis). Values below 1 indicate that the complete search is superior to the high-level search. HIPS- $\varepsilon$  outperforms HIPS in every environment except TSP, where high-level actions are sufficient for solving every problem instance.
Figure 4: The percentage of puzzles remaining unsolved (y-axis) depending on the number of node expansions (x-axis) for complete search with different values of  $\varepsilon$  and only high-level search (left), and the ratio of unsolved puzzles in comparison with HIPS depending on the number of node expansions (right), when the methods have been evaluated on an out-of-distribution variant of Box-World.
naive  $\mathrm{A}^*$ -inspired evaluation functions [9] which correspond to the variant  $\mathrm{PHS}_h$  in [27]:
$$
\varphi_ {\text {d e p t h}} = (g (n) + h (n)) / \pi (n), \quad \varphi_ {\text {d i s t}} = (l (n) + h (n)) / \pi (n).
$$
The results in Table 4 show that a heuristic function is crucial for reaching a competitive search performance in most environments. In particular, on TSP, where it was observed that training a good VQVAE prior (which we use as  $\pi_{\mathrm{SG}}$ ) is very difficult [18], the search heavily relies on the learned heuristic, and the search fails due to running out of memory. Furthermore, naively using A*-inspired evaluation functions fails to be competitive.
Subgoal search algorithms can effectively address complex reasoning problems that require long-term planning. However, these algorithms may fail to find a solution to solvable problems. We have presented and analyzed HIPS- $\varepsilon$ , an extension to a recently proposed subgoal search algorithm HIPS. We achieve this by augmenting the subgoal-level search with low-level actions. As a result, HIPS- $\varepsilon$  is guaranteed to discover a solution if a solution exists and it has access to an accurate environment model. HIPS- $\varepsilon$  outperforms HIPS and other baseline methods in terms of search loss and solution rate. Furthermore, the results demonstrate that augmenting the search with low-level actions can improve the planning performance even if the subgoal search could solve the puzzle without them, and the proposed algorithm is not hyperparameter-sensitive. HIPS- $\varepsilon$  enables using subgoal search in discrete settings where search completeness is critical.
In the future, we would like to tackle some of the limitations of our work. Our search paradigm could be applied to other subgoal search methods than HIPS and search algorithms than PHS. Combining HIPS- $\varepsilon$  with recursive search methods that split the problem into smaller segments could enable scaling to even longer problem horizons. Our analysis of the complete subgoal search assumes deterministic environments and access to transition dynamics. HIPS- $\varepsilon$  can also function without access to the transition dynamics, but the completeness guarantee is lost. Many real-world problems are partially or fully continuous. There are also problems with infinite discrete action spaces that HIPS- $\varepsilon$  is not particularly suited for, and we assume discrete state representations. HIPS- $\varepsilon$  also assumes that a solution exists. Modifying HIPS- $\varepsilon$  for the continuous setting could enable solving real-world robotics tasks. HIPS- $\varepsilon$  also showed potential at transfer learning by efficiently solving more complex Box-World tasks than those seen during training. Applying curriculum learning by learning the initial model offline and then learning to progressively solve harder problem instances with online fine-tuning is a promising direction for future research.
We acknowledge the computational resources provided by the Aalto Science-IT project and CSC, Finnish IT Center for Science. The work was funded by Research Council of Finland (aka Academy of Finland) within the Flagship Programme, Finnish Center for Artificial Intelligence (FCAI). J. Pajärinen was partly supported by Research Council of Finland (aka Academy of Finland) (345521).
[1] Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the Rubik's Cube with Deep Reinforcement Learning and Search. Nature Machine Intelligence, 1 (8):356-363, 2019.
[2] Cameron Allen, Michael Katz, Tim Klinger, George Konidaris, Matthew Riemer, and Gerald Tesauro. Efficient Black-Box Planning Using Macro-Actions with Focused Effects. ArXiv preprint, abs/2004.13242, 2020. URL https://arxiv.org/abs/2004.13242.
[3] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 15084-15097, 2021. URL https://proceedings.neurips.cc/paper/2021/bit/7f489f642a0ddb10272b5c31057f0663-Abstract.html.
[4] Konrad Czechowski, Tomasz Odrzygóźdz, Marek Zbysinski, Michal Zawalski, Krzysztof Olejnik, Yuhuai Wu, Lukasz Kucinski, and Piotr Milos. Subgoal Search for Complex Reasoning Tasks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 624-638, 2021. URL https://proceedings.neurips.cc/paper/2021/bitize/05d8cccb5f47e5072f0a05b5f514941a-AAbstract.html.
[41] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6306-6315, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/7a98af17e63a0ac09ce2e96d03992fbc-Abstraction.html.
[42] Homer Rich Walke, Jonathan Heewon Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't Start from Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning. In Conference on Robot Learning, pages 1652-1662. PMLR, 2023.
[43] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A Survey of Autonomous Driving: Common Practices and Emerging Technologies. IEEE access, 8:58443-58469, 2020.
[44] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational Deep Reinforcement Learning. ArXiv preprint, abs/1806.01830, 2018. URL https://arxiv.org/abs/1806.01830.
[45] Michal Zawalski, Michal Tyrolski, Konrad Czechowski, Damian Stachura, Piotr Piekos, Tomasz Odrzygoźdź, Yuhuai Wu, Łukasz Kuciński, and Piotr Miłos. Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search. ArXiv preprint, abs/2206.00702, 2022. URL https://arxiv.org/abs/2206.00702.
[46] Lunjun Zhang, Ge Yang, and Bradley C. Stadie. World Model as a Graph: Learning Latent Landmarks for Planning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12611-12620. PMLR, 2021. URL http://proceedings.mlr.press/v139/zhang21x.html.