SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we discuss the planning algorithms that can be implemented in the Thinker-augmented MDP. Uninformed search strategies [56, Ch 3.4] that do not require backward search, including Breadth-first search (BFS), Depth-first search (DFS), MCTS can be implemented by the RL agent in the Thinker-augmented MDP. For example, to implement BFS with two actions available, the agent can select  $(a_{1},\text{reset})$ ,  $(a_{2},\text{reset})$ ,  $(a_{1}$ , not reset),  $(a_{1}$ , reset),  $(a_{1}$ , not reset),  $(a_{2}$ , reset), ... as the imaginary actions. If we treat the values as heuristics, then informed search strategies [56, Ch 3.5] that do not require backward search, can also be implemented. For example,  $\mathrm{A}^*$  could be implemented by expanding the unexpanded node that has the highest rollout return in the tree, which can be achieved by keeping a record of unexpanded nodes along with the rollout return in the memories.
Nonetheless, as an MDP does not necessarily involve a goal state as in planning problems and our trained model does not support backward unrolling, any backward search from a goal state, such as bidirectional search [56, Ch 3.4] and partial-order planning [56, Ch 10.4], cannot be implemented by the agent. Agents are also unable to implement planning algorithms that involve heuristics beyond value function, such as GRAPHPLAN [56, Ch 10.3] that employs heuristics based on planning graphs. In summary, an RL agent under the Thinker-augmented MDP can implement any forward-based planning algorithms that do not involve heuristics beyond the value functions.
In practice, the learned planning algorithm is very different from the aforementioned handcrafted planning algorithms, as illustrated in Appendix F and the video visualization. This is because, unlike traditional planning algorithms whose goal is to solve the task in the search, the RL agent's primary incentive is to provide useful information for selecting the next immediate real action.