Strategic Planning: A Top-Down Approach to Option Generation
Authors: Max Ruiz Luyten
Antonin Berthon
Mihaela van der Schaar
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Top-down approach
Strategic Planning
Language-Conditioned RL
Primary area: reinforcement_learning

Abstract: Real-world human decision-making often relies on strategic planning, where *high-level* goals guide the formulation of sub-goals and subsequent actions, as evidenced by domains such as healthcare, business, and urban policy. Despite notable successes in controlled settings, conventional reinforcement learning (RL) follows a *bottom-up* paradigm, which can struggle to adapt to real-world complexities such as sparse rewards and limited exploration budgets. While methods like hierarchical RL and environment shaping provide partial solutions, they frequently rely on either ad-hoc designs (e.g. choose the set of high-level actions) or purely data-driven discovery of high-level actions that still requires significant exploration. In this paper, we introduce a *top-down* framework for RL that explicitly leverages *human-like strategy* to reduce sample complexity, guide exploration, and enable high-level decision-making. We first formalize the *Strategy Problem*, which frames policy generation as finding distributions over policies that balance *specificity* and *value*. Building on this definition, we propose the *Strategist* agent—an iterative framework that leverages large language models (LLMs) to synthesize domain knowledge into a structured representation of actionable strategies and sub-goals. We further develop a *reward shaping methodology* that translates these strategies expressed in natural language into quantitative feedback for RL methods. Empirically, we demonstrate a significantly faster convergence than conventional PPO. Taken together, our findings highlight that *top-down strategic exploration* opens new avenues for enhancing RL on real-world decision problems.

Strategic Planning: A Top-Down Approach to Option Generation
Max Ruiz Luyten Antonin Berthon
Mihaela van der Schaar'
Abstract
1. Introduction
Real-world complexity and top-down planning. Real- world human decision-making hinges on strategic goal- setting and top-down planning (Correa et al., 2023; Collins et al., 2024). From large-scale business decisions to complex healthcare protocols (Strickland & Lizotte, 2021). experts frequently define high-level objectives and then decompose them into increasingly concrete subgoals to guide daily ac- tions. This contrasts sharply with the way conventional reinforcement learning (RL) often proceeds: bottom-up, through extensive trial-and-error in controlled or simulated environments (Mnih et al., 2015; Silver et al., 2016). Al- though bottom-up RL has led to landmark successes in various domains, its reliance on massive exploration and repeated resets conflicts with real-world domains, where errors can be costly time-consuming (Dulac-Arnold et al., 2021), and rewards are sparse and delayed (Rengarajan et al., 2022).
Real-world human decision making often relies strategic planning, where high-level goals guide the formulation of sub-goals and subsequent actions, as evidenced by domains such as health- care, business, and urban policy. Despite notable successes in controlled settings, conventional re- inforcement learning (RL.) follows bottom-up framework, which can struggle adapt to real- world complexities such sparse rewards and limited exploration budgets. While methods like hierarchical RL and environment shaping provide partial solutions, they frequently rely on either ad hoc designs (e.g. choosing the set of high- level actions) or purely data-driven discovery of high-level actions that still requires significant ex- ploration. In this paper, we introduce a top-down RL framework that explicitly leverages human- inspired strategy to reduce sample complexity. guide exploration, and enable high-level decision making. We first formalize the Strategy Problem, which frames policy generation as finding distri- butions over policies that balance specificity and value. Building on this definition, we propose the Strategist agent-an iterative framework that leverages large language models to synthesize do- main knowledge into a structured representation of actionable strategies and sub-goals. We fur- ther develop a reward shaping methodology that translates these strategies expressed in natural lan- guage into quantitative feedback for RL methods. Empirically, demonstrate that our framework significantly enhances the performance of differ- ent underlying RL algorithms, leading to faster convergence and the discovery of more complex behaviors. Taken together, our findings highlight that tow-down strategic exploration opens new avenues to improve RL in real-world decision problems.
Motivation: human-inspired strategies vs. bottom-up RL. Top-down planning is standard for humans operating in complex, uncertain, and ressurce-constrained contexts (Correa et al., 2023). In business, strategists consider al- ternative pathways for growth- such as increasing sales volume, improving operational efficiency, or diversifying product before committing to one. In healthcare, medical teams identify key bottlenecks e.g. availability of caregivers or patient discharge protocols-and define action plans that interleave resource allocation and clinical proce- dures. Such human-inspired strategy formation not only structures the planning process, but also reduces the explo- ration burden by focusing attention on the most promising areas of the decision space (Sutton et al., 1999; Collins et al., 2024).
Challenges in real-world exploration. In many real- world contexts, an unstructured bottom-up search may be neither feasible nor safe (Dulac-Amold et al., 2021). If learning agent suffers significant penalties (like budget overruns in a store or catastrophic patient care failures in hospital), it may never gather enough experience to converge on a truly optimal policy. Even when exploration is feasi- ble, the sheer scale real-world decision spaces and the unpredictability of evolving environments can overwhelm trial-and-error learning.
Equua contribution 'University of Cambridge. Correspondence to: Max Ruiz Luyten <mr971 @ @camaac.uk>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Strategic Planning: A Top-Down Approach to Option Generation
Illustrative Example: Collecting Meat
paired with PPO (Schulman al., 2017) or SOTA meth- ods such EDE (Jiang al., 2023).
Environment: Minecraft
We begin by formally introducing novel top-down RL framework named the Strategy Problem, along with theo- retical grounding for its practical efficacy in $.. Then we discuss the existing work and how they differ from our top- down approach in $ 3. 4 presents a first instantiation of this framework in the Strategist Agent, combining the LLM- based tree search with state-based reward shaping train grrdee RL agents. We demonstrate the efficacy of this method compared to traditional RL methods in $ 5, and we discuss potential future directions in 6.
Goal: Maximize Meat Collection
Bottom-up Approach
Learn low-level actions like movement.
Trial-and-erro explore to locate animals.
Eventually discover hunting for meat.
Top-down Approach
Identify overarching strategies: hunting,
farming, or trading.
crafting tools, building enclosures, breeding livestock.
Outline sub-goals:
2. The Strategy Problem
Plan transitions: switch to farming sta- ble infrastructure place.
Informal Motivation: Why a Strategist? Real-world RL problems often involve vast state and action spaces, sparse rewards, and limited exploration budgets (Dulac-Arnold et al., 2021). In such settings, standard "bottom-up" RL (e.g., Q-learning (Watkins & Dayan, 1992)) without prior knowledge requires prohibitively large number of trials, on the order of |SIAA (1-2)² log } to provide an e-optimal policy with probability å (Strehl al., 2009; Lattimore & Hutter, 2012). In contrast, humans tackle complexity by proposing multiple plausible high-level strategies each narrowing the search in different ways and iteratively re- fining them until one is chosen. Motivated by this, we define a Strategist as an agent that constructs distributions over policies that incorporate expert inductive bias to reduce the search space while encompassing near-optimal policies.
Summary: Bottom-up RL gets drawn to whichever ap- proach yields immediate rewards rewards-bunting. A top-down strategy broadens the horizon to better, complex, approaches like farming.
Recent progress and key limitations. Recent progress in hierarchical RI. (Kulkarni et al., 2016; Liu et al., 2024a), environment shaping (Park et al., 2024), and LLM-assisted RL (Huang et al., 2022; Ahn al., 2022: Singh al., 2022: Pternea et al., 2024; Yan al., 2024) offers partial answers but still leaves key gaps. Many current methods rely manually engineered sub-goals, or purely data-driven dis- covery of structure. Moreover, most approaches often gen- erate a single plan rather than exploring multiple promising strategies. To address these limitations, we propose a new top-down RL framework that uses systematic strategic rea- soning to reduce sample complexity, guide exploration, and elevate deeision-making to more abstract planning layers.
2.1. Formalism Context and Definitions
RL in Nutshell Let M = (S, P. P,R, denote Markov Decision Process (MDP), where S the state space, A the action space, P(s' 5, s, the transition function, R(s, a) the reward function, and y [0,1] 1] the discount factor. A policy maps states to ac- tions (or action distributions), and its expected discounted return is ((T) Ex E>>" (.*R(s, so is the value function and P the initial state distribution. RL provides algorithms M IIM that aim for an optimal policy arg max [TT) with II as the set of policies.
Contributions.
Eoop [VT(s))]. where V*(()
We formalize strategy as a distribution over policies that balances specificity (pruning the search space) and value (encompassing optimal plans).
We instantiate these ideas in the Strategist agent', which uses the based tree search to encode domain knowledge into actionable top-down strategies without prespecifying their components. Crucially, we introduce reward shaping methodology that translates strategies into quantitative feedback for RL.
Distributions over Policies Rather than searching over I from scratch, we propose the use of a strategist:
We empirically validate that this procedure enables ef- feetive exploration in different task specifications in the Crafter environment (Hafner, 2021), leading to faster convergence and improved final performance when
where K. encapsulates high-level or "common sense" knowl- edge about the domain (e.g., a description of the environ- ment natural language). The output ôπ = (MM, is a probability distribution over policies. Intuitively, ón re- stricts the policy space to "plausible" strategies in an option-
2

---

Strategic Planning: A Top-Down Approach to Option Generation
Bottom-up
X Initial policy
Learning algorithm
All policies II
Optimal policies
Sub-optimal policies
Reward
[(m)
Steps
Top-down
X Initial policy
Strategy-guided learning algorithms
3. Train strategy-guided RL agents
Derive strategic options
Reward
Iteratively breakdown into sub-goals
Steps
Figure 1. Comparison between bottom-up up and top-down frameworks when searching for optimal policics given goal to maximize the expected returns ((T).
spirited manner (Precup et al., 1998), focusing learning or planning on smaller and more promising subset.
2.2. Formal Definition of the Strategy Problem
The Strategy Problem
Specificity and Value A good strategy dn must balance two key properties:
Given an MDP M, high-level knowledge KG, and thresholds (H maxx E, Umin). find a distribution
1. Specificity: ōn should focus on a relatively small set of promising policies, thus reducing the effective search space. We quantify the specificity through the entropy H (6n)). "sufficiently specific" strategy satisfies
ôn E A(IM)
such that:
Specificity: H(8n) H Hmax.
H(ōn)= Hunxx
2. Value: [ 2 (1 Umin Umin) (TT*)
for some small threshold Hmax-
2. Value:
Among the policies in ôn's support, there should exist at least one opar-oltimal policy. Formally, define the e-support of ōn
In essence, the Strategy Problem searches for one or multi- ple distributions over policies ôn that maintain low entropy (i.e. high specificity) while ensuring that its support contains at least one high-value policy.
(up = =f IlM Son (T)) > >
where fon (T) is the density of òn. Let ThE arg maxp (8n) J(z) be the best policy within that support. The strategy dn is said to have value at least Unin if
2.3. How Specificity and Value Aid Sample Efficiency
The Strategy Problem aims improve sample efficiency by constraining the policy search to structured distribution ôn that balances specificity and value. In traditional RL, standard probably approximately correct (PAC) arguments show that finding a near-optimal policy requires a number of samples scaling with ([S]][A]) (Strehl et al., 2009), making
J( 2 (1 Umin J(x")
where Umin is small constant.
3

---

Strategic Planning: A Top-Down Approach to Option Generation
learning in large state-action spaces infeasible. In contrast, well-designed dn focuses exploration on smaller, high- potential subset of policies, reducing the effective search space in an options-like spirit (Precup et al., 1998; Fruit & Lazaric, 2017) while still ensuring near-optimality. The following theorem formalizes this advantage, showing that the sample complexity depends on the entropy of ôπ rather than the full size of the policy space.
world dynamics to carry out rollouts. These methods are orthogonal ours and could be used in tandem. While these approaches reduce the search space, they typically do so state- bt-tte, treating the LLM as single-step planner,
rather than seeking multiple overarching strategies or ex- plicitly balancing the value and specificity trade-off's that our top-down method targets. More critically, many such works are only applicable to tasks with rext-format actions (e.g., "open fridge', 'use key') that can be inapplicable for lower-level domains.
Theorem 2.1 (High Specificity & Value = Sample Efficiency). Consider strategy òn E AllM with high specificity, i.e. (ōn) S Hmaxx and high-value, so that J(x²) - [(T S 8/2. Then, with probability at least 1 1-6, identifying policy whose return is e-close to optimal requires at most
Hierarchical RL Several studies have explored hierarchi- cal RL augmented by LLM-generated plans. SayCan (Ahn et al., 2022) and Video-Language Planning (Du et al., 2024) demonstrate that, if given a suitable set of action primitives or a text-to-video planning system, an LLM can sequence sub-goals. Language Planner (Huang et al., 2022) and Prog- Prompt (Singh et al., 2022) adopt similar frameworks but similarly rely on predefined macro-actions or prompts for decomposing tasks. These methods remain fundamentally bottom-up in that LLM suggestions must be hand-alignn to fixed set of learned engineered primitives, which usually requires either text-format action environments extensive model, which partially
o Hmax
1
(
(Hmmer
Hmax ÔT In (+)
In
(+)
episodes, where T determines the lower probability thresh- old for the candidate policies.
The proof is provided in $ A. This theorem underscores the rationale behind the Strategy Problem: the more concen- trated ōn is (i.e., the lower its entropy), the fewer candidate policies, simplifying the search for near-optimal policy. Specificity ensures a reduced search space, while the value condition guarantees that there is at least one near-optimal policy within it. As a result, sample complexity depends on Hmax rather than the full state-action space, reinforeing the key idea that structured policy distributions can signifi- cantly accelerate RL by narrowing the search space while preserving near-optimality. Although this theoretical bound does not extend to our implementation, it provides intuition for the properties that we should strive for in strategy.
training of a ianguage-condtioone defeats the purpose in high-stakes scenarios. In contrast, our framework's 's strategies need not be limited to any single level of abstraction or a subset of primitives; and it flexibly explores multiple possible routes before committing to a refined direction.
Reward shaping Another line of work tackles reward en- gineering using LLMs. (Sarukkai et al., 2024), (Ma et al., 2024), and and
(Xie et al., 2024) show how language models can write code that shapes rewards. Although such approaches mitigate purely sparse rewards, they still rely heavily on trial-and-error adjustments of the reward logic by training models. In contrast, our method embeds reward shaping within a top-down strategizing process, automatically trans- lating strategic knowledge into sub-goal rewards zero-shot without requiring trial and error in the environment or large- scale manual interventions.
3. Related work
Integrating high-level knowledge and structure into RL has long been studied under various guises of hierarchical poli- cies, curriculum learning, and, most recently, LLMM-basd approaches. While these directions offer partial remedies to the challenges of sparse rewards and massive exploration costs, they often adopt a bottom-up perspective that requires either extensive trial-and-eme manor or ad-hoe high-level designs. Our work instead proposes top-down framework that sys- tematically leverages human-inspired strategy formation.
While these prior works demonstrate that LLMs can of- fer valuable inductive biases-either by constraining ac- tion spaces, generating sub-goals, or shaping rewards- they generally omit the notion of strategic exploration that can fundamentally reduce sample complexity. By synthesizing top-down strategies as distributions over policy families and translating them into shaped rewards, our approach strategically narrows the search space without sacrificing expressivity, making a step toward real-world RL where extensive trial and error is not required.
LLM-as-policy A number of recent methods incorpo- rate LLMs to guide RL policies by providing action pri- through bottom-uy perspective. For example, (Yan et al., 2024) view the LLM as a Bayesian prior over ac- tions, enabling more focused exploration by restricting the candidate action space. (Zhang et al., 2024) similarly em- ploy value-based filtering of LLM -roopooeeeed moves. (Liu et al., 2024b) use L.Mss to both propose actions and model
We elaborate on the distinction with these and additional works across RL, Curriculum Learning, Hierarchical RL, L.M planning, as well as LLM reasoning in $ D.
4

---

Strategic Planning: A Top-Down Approach to Option Generation
4. The Strategist Agent
This duality between vertical branching (alternative ap- proaches) and horizontal decomposition (sequential sub- goals) enables the Strategy Tree to perform flexible, multi- level strategic reasoning.
We introduce the Strategist agent as our initial solution to the Strategy Problem. This agent follows the top-down approach to RL, operating in three key phases. First, it generates and explores potential strategies expressed in nat- ural language, following tree-based structure ($ 4.1 and
4.3. Node Attributes: Value, feasibility and specificity
4.2). Second, it evaluates and prioritizes these strategies, iteratively refining them to achieve the necessary level of specificity ($ 4.3). Finally, it uses the best strategies to train RL. agents through reward shaping, effectively guid- ing the policy search process toward the chosen strategies ($ 4.4). While this approach offers promising direction for addressing the Strategy Problem, it represents just one of many possible solutions that merit future investigation.
To select and prioritize exploration in the strategy tree ac- cording to the strategy problem ($ 2). each node h; is as- signed two attributes-value and feasibility, to check that the strategy is realizable in the environment. Note that speci- ficity, the other key property of the Strategy Problem, is implicitly captured by the hierarchical structure of the tree. Hence, by design, the nodes satisfy H (8hh.) < H(
(hi) captures how helpful the node's strategy is for achieving the parent objective. Formally, if (xx) the expected return of a policy æ, then
Value
4.1. Strategy Tree Construction
The Strategist's core component is the Strategy Tree, a hier- archical structure where each node represents a distribution over policies through natural language descriptions.
max ( ((z
v(hh))
maxpesupp, [TT)
Overview of the Strategist Let hg denote the root node corresponding to the overall objective G. The tree ex- pands through refinement process in which each child hig E C(h,) provides a more detailed specification of its parent's 's h, policy distribution. Formally, each node h; is associated with a distribution 8h: E AlII) in the pol- icy space M Child nodes are constrained to represent increasingly specific policy subsets of their parents, with the branching factor reflecting the number of distinct ap- proaches considered for decomposing the node's goal.
Feasibility [(h.) describes the likelihood that policy from 8h can actually be implemented by the RL agent in the enviromment. Formally, [hhs = 8h (Iaggm)) ,, that is, the probability that a random draw from 8h lies within the set of policies agem implementable by our learning system (e.g., limited action space, time constraints, etc.).
We detail aggregation, expansion, and selection in B.1.
4.4. Reward Shaping from Strategies
This recursive refinement process continues until the nodes reach sufficient specificity and value to serve as an effective inductive bias for the learning agent.
A key innovation of our framework is strategy-guided re- ward shaping, which translates textual strategies into quan- fitative reward to guide RI. agents. Concretely, given a strat- egy ht, corresponding to root-to-leaf path of the Strategy Tree, we instantiate a reward The (8, a) that augments the original enviroumental reward. It proceeds in two steps:
4.2. Node Types: Strategic Decomposition
Building on the structure of the Strategy Tree, the nodes h; can be categorized into two fundamental types that enable different forms of strategic reasoning:
1. Node-Specific Scoring:
For the chosen strategy h-, we prompt an LLM with relevant state details } and the textual specification of the node. The LLM outputs a scalar score ut, indicating how "close" the state to satisfying that sub- aob-goal. We normalize ut to obtain the reward R(ss)). For efficiency, we distill R into a student network to reduce LLM calls.
1. Approach Nodes: These nodes branch into (hv ) mutually exclusive strategies to achieve the parent's 's goal (e.g.. "hunt" vs. "farm" to collect meat). Each child distribution ghh,,, narrows the scope of the policy by specifying implementation details. In practice, only the most promising approach branch is pursued.
2. Aggregated Shaping Reward:
We combine the LLM- derived reward R(s,a) EgrwP(s.a) [f('')) with the environment's original reward R(s, a). E [0, 1] con- trols the strength of the strategy signal. In practice, a decays with training, progressively shifting to the reward of the environment:
2. Plan Nodes:
These nodes decompose the parent goal into sequence (or set) of complementary sub-goals that must all be completed (c.g.. "craft sword + lo- cate animals - hunt"). The distribution &h factors in these subgoals, and the overall properties of the node are derived from the composition of its components.
The (St, 04) = (1 - R()) at) + a R(s,a). (1)
5

---

Strategic Planning: A Top-Down Approach to Option Generation
5.1. Environmental Setup
Type
Primitive (Sect 4.2)
Feasibility, Value, Specificity (Sect. 4.3)
Example
We utilize two main environment configurations:
Approach nodes
[(A)) (l
Gathei most
Modified Crafter (Easy & Medium) We modify the task to meat collection, and introduce additional mechanics that allow multiple plausible strategies. Specifically, cows can be bred via saplings, which can in turn be obtained from grass with probability: 1 mobs, and we implement two difficulty tiers:
*(G. (4) (+t) (Z
I (f*) 11 (4)
Humi
Farn
Plan nodes
s(b) (R0)
((hh 11 7I
2- We disable nighttime and hostile
(4') min (d))
1. Crafter-Easy: cows require only one hit to kill and a single sapling to breed.
B
Guloo
2. Crafter-Medium: cows require three hits to kill without a sword (one with a sword) and two saplings to breed.
Trade
Dul
Gread
Fird
Tradk
In these modified versions, rewards are exclusively granted for collecting meat, thereby introducing sparseness: may take many steps for an uninformed agent to discover how to kill or breed cows effectively. Thus, we view this as rich test bed for examining strategic exploration.
Maw
Mack
Reward Shaping from Strategies (Sect. 4.4)
Dovenn Wnowlede
Language Micchl
Strctentt Netwwww
LLM
[(s,, Fs3s
f R(s)
Original Crafter We also evaluate our framework in the original Crafter environment, which features 22 achieve- ments, and the agent receives rewards for unlocking each new achievement. This setting presents different sparse reward challenge and requires strategic prioritization across wider range of potential goals (e.g., tool crafting, combat, resource gathering, survival needs).
Shaped
{ Ennossoonssa states
Alpha Schedule
RL Agent Training Reward
αx
rals, (1 a) R(s,,'))))) + al(s')
Steps
Figure 2. (A) Building blocks of the Strutegy Tree: option nodes are different possible paths to reach the parent goal, while se- quential nodes are sets of sub-goals that need be sequentially completed to reach the parent goal. (B) Example of a possible Strat- egy Tree for the goal maximizing meat collection Minecraft. (C) Illustration of our Strategy-bused Reward Shaping process.
5.2. Experimental Procedure
For each environment, we follow three stages:
1. Strategy Tree Generation. We run the Strategist agent using GPT-4o (OpenAl et al., 2024) to build a Strat- egy Tree (4.1) relevant to the overarching goal of the environment (e.g. maximize meat or achievements).
In environments where actions are sufficiently high-level for the LLM, such as text-based environments, one can also consider using LLo-pooposed actions as a policy regularizer (Yan et al., 2024: Zhang et al., 2024), and condition them on the strategy, thereby further accelerating convergence. However, we prefer to preserve applicability low-level settings; therefore, we opt not to include this in this work. More details about this approach and the reward shaping process are provided in B.
2. Strategy Selection and Reward Shaping. We extract promising strategies (the top two for Modified Crafter and four for Original Crafter, based on feasibility and value). For each strategy, we distill the LLM-based reward with a ResNet-18 (He et al., 2015) on 5,000 states. We combine the learned reward with the original environment one, gradually annealing from 1.0 (or 0.9) to 0 or 0.1 during the course of training.
3. RL Agent Training. For each selected strategy, we train N separate RL agents (typically N = 8 for PPO, N = 3 for EDE due to computational cost) for a set number of steps (e.g., 2 x 10 for Modified Crafter and 1 10c for Original Crafter). We use PPO (Schul- man et al., 2017) as the primary RL backbone. For further validation, we also employ EDE (Jiang et al., 2022) and DreamerV3 (Hafner et al., 2023) as stronger baselines, demonstrating that the Strategist framework can be paired with different underlying RI. algorithms
5. Experiments
We evaluate the Strategist framework on different versions of Crafter (Hafner, 2021) and with different RL back- bones: Proximal Policy Optimization (PPO) (Schulman et al., 2017), and Exploration via Distributional Ensemble (EDE) (Jiang et al., 2022). Our experiments demonstrate how top-down strategy generation and reward shaping sig- nificantly enhance sample efficiency and performance.
6

---

Strategic Planning: A Top-Down Approach to Option Generation
PluntSatisyy Hunger. Satisty Thirst, Eie- sure Adoquite Rest)
Maintaining Health
Survival
Stralegy Surv
Direct Cow Hunting (Seek Cows Direcily)
Breed Cows and Then Them Strategy Breed
Collect much
Strategy Hunt
Strategy Breed
mest as posaible
PlantCollect Saplings;
Breeding Cows
Find Cows; Feed
Collect as much cow possible
Find and Kill Cows
with Saplings)
Hare Hands
Strategy Hunt
Plant Guther Busic Re-
PlantGiuther Resourves for Crafting Tools; Craft Taols; Find and Kill Cows) Strategy Craft
Preparation (Gatteer
sources; Craft Basic Tools;
Craft Wespon Kill
Resources and Tools)
Gather Advinced Resourzes,
Cruft Advanced Tools)
Strategy Craft
Figure 3. Strategy Tree obtained by running the Strategist agent (Section 4) with GPT-4o for the Crafter-Eas (left) and Crafter- Meeiium (right) environments. The strategies are ordered according to the product of feasibility and value estimated by the Strategist at each node,
(c.g., Strategist+PPO, Strategist+EDE) Baseline agents (PPO, EDE, DreamerV3) are trained under the same conditions but without shaped rewards (a 0).
Strategy Breed
Strategy Hunt
PPO
3
Stronegy Hune kaamn
The experimental setup is provided in detail in $ B.
Average Reward
2
5.3. Results on Modified Crafter Environment
Strotegy Breed and then amacc
5.3.1. STRATEGY TREES
oowa
Figure shows the results of the Strategist in the mod- ified environments, illustrating how it identifies multiple high-level approaches before refining them into concrete sub-goals. For instance. in Crafter-Medimm, the Strategist proposes to hunt cows by crafting weapons first, as well as an alternative preeding-centered plan. Each node is anno- tated with an L.M--stimmted, feasibility (likelihood that the agent can achieve the strategy) and value (degree to which it might produce high returns).
Stritegy Breed Icarno gov
0.0M
0.5M
1.0M
1.5M
2.0M
Training Step
5.3.2. PERFORMANCE WITH PPO
Figure 4. Performance of strategy-guidee RL agents (PPO). Top panel: Average episode reward on Crafter-Medumm. Bottom panel: Average number of saplings given to cows per episode (S-C= "Saplings given to Cows"). Each line shows the average of N=8 runs, with error bars showing 95% confidence intervals.
Figure 4 compares the average episode reward of the top two strategy-guided PPO agents (one focusing on hunting, the other on breeding) against a vanilla PPO baseline on Crafter-Medium. Both ttrategy-giided agents achieve faster and more meat collection following their strategies, which underscores the benefit of top-down guidance.
5.3.3. EMPIRICAL STRATEGY METRICS WITH PPO
Table 1 further quantifies this by listing the number of steps required to achieve key milestones. "Hunting" is defined as "a > O average reward"; and "Breeding" as "> 1 average saplings given to a cow".. Both strategy-guided PPO agents achieve hunting in 8/8 runs. typically within 600-900k steps, whereas PPO achieves this in only half of the runs and takes 1.35M steps average. Breeding is achieved only by the breeding- guided agent, indicating that the strategist is effective in eliciting more complex behaviors.
Next, we measure the feasibility and value in deployment when using PPO, shown for Crafter-Easy in Table 2. Feasi- bility operationalized as the maximum normalized shap- ing reward the agent achieves. while value is measured by the maximum environment reward. We observe that for some strategies (e.g., breeding in Crafter-Mediume the re- alized feasibility and value exceed the initial estimates of the Strategist. For others (c.g., crafting- observed performance is lower than anticipated. Figure 5 vi- sually compares LLM- --Meddddeeted metrics with PPO measurements, revealing a strong overall correlation
plans), the
7

---

Strategic Planning: A Top-Down Approach to Option Generation
Achieve Hunking Stralogy
Achiove Breeding Stralegy
To further assess the Strategist's utility, we pair it with EDE, advanced RL algorithm that achieves competitive re- sults the Crafter leaderboard". As shown in Figure 6, Strategist+EDE, particularly with the "Breed" strategy, sig- nificantly outperforms vanilla EDE on Crafter-Mediumm, The increase in episode length for vanilla EDE suggests that its reward gains stem from improved survival. contrast, the Strategist-guider agent consistently learns to breed cows indicated by the "sampling cow" metric-, complex behavior vanilla EDE struggles with, leading to substan- tial increase in episode rewards. This indicates that even ad vavaneed exploration mechanisms benefit from a top-down strategic direction.
Model
Successful runs
Step (M)
Successful runs
Step (M)
4/8
1.35 0.13)
0/8
Strategy Hunt
8/8
0.89 (+/- 0.12)
0/8
Strllogy
8/8
0.59 0.17)
1.22 0.76)
Table 1. Number of steps taken by PPO agents to reach Hunting Breeding stages the Crafter-Medium environment.
but some notable miscalibrations. These results suggest that while the Strategist can effectively propose strategies that boost performance, its heuristic estimates of feasibility and value can be refined through iterative feedback from the environment.
20
Estimated Val.
Observed
Vanilla EDE
Strategy
Feas.
Prod.
R.
Feasibility
Value
Strategist+ [Hunt]
Strategy Hunt
1,00
1,00
1,00
0.74 0.03)
Ep. Reward
15
Strategist EDE [Breed]
Stratogy Breed
0,70
0,80
0.56
0.77 (4/- 0,02)
19.44 (+/- 2.68)
Strategy Craft
0.80
0.60
0.48
0.51 (+/- 0.01)
3.86 (w/-4.00 4.06)
10
Strategy Surv
0.24
0.55 0.01)
0.41)
Table 2. Estimated observed metries for different strategics with PPO on Crafter-Easy. Feas.: Feasibility, Val.: Value, Prod.:
Product, R.: Rank.
Sapling to Cow Ep. Length
750
Estimated
Strategy Hunt Stratagy Breed
Strategy Cralt Stratagy
Observed
500
Feasibility (Observed) 0.60
0.75
40
Stretegy Breed overperforms expactations
20
Value (Estimated)
Value (Observed)
0.8
0.8
0.00
0.25
0.50
0.75
1.00
1.25 1.50
1.75
2.00
Environment Step (M)
Figure 6. Results of EDE and Strategist+EDD on Crafter- Medium. Top: Episode reward. Middle: Episode length. Bottom: Average number of saplings given to cows per episade (Sapling to Cow). The Strategist+EDE outperforms vanilla EDE with both the hunting and breeding strategies. Error bars show the 95% confi- dence intervals across N=3 runs.
Strategy Craft underperfoma espectations
0.4
0.4
0.75 Feasibility (Estimated)
1.00
5.4. Results on Original Crafter Environment
Figure 5. Comparison of the L.M--sstmated vs empirically- observed feasibility and values (PPO) for the Crafter-Rasy environment. Observed values normalized by the maximum observed value.
In the original Crafter environment, the Strategist identifies four overarching strategies. Fight (combat and exploration), Craft (tool making), Resource (resource collection), and Needs (management of survival needs). We evaluate these by pairing them with the EDE agent.
5.3.4. INTEGRATION WITH ADVANCED ALGORITHMS
As stronger baseline RL algorithm, we run Dream- erV3 (Hafner et al., 2023) with 50M parameters in the Modi- fied Crafter environment. DreamerV3 performs slightly bet- ter than vanilla PPO on Crafler-Medium, but Strategist+PPP still outperforms it (see $ C.5 for detailed DreamerV3 re- sults). We also compare to two LLM--bsed baselines: LLM- as-a-policy, and Eureka (Ma et al., 2024) (see $ C.8 in the appendix).
Figure 7 shows that the Strategist+ variants focus- ing on Fight, Resource, and Craft significantly outperform vanilla EDE in terms of overall rewards (cumulative achieve- ments). The Needs-guided strategy performs comparably with vanilla EDE, as basic survival achievements are easily reached by the baseline algorithm.
hhtt:://ghhaacccmdddjjaarraeetee
8

---

Strategic Planning: A Top-Down Approach to Option Generation
6. Key Takeaways and Outlook
Episode Reward
Why Top-Down Matters. Our work reframes exploration in RL strategic design problem: rather than searching the entire policy space, we first search over distributions of policies. The resulting Strategy Problem formalism and its instantiation in the STRATEGIST agent deliver three concep- tual advances:
8
6-
EDE
Strategist-EDE EDE [Fight]
4-
Strategist EDE [Nasource]
Strotegist EDE [Croft]
2
Strwtegist EDE [Needs]
0.0
0.2
0.6 Environment Step (M)
0.4
0.8
1.0
1. A principled objective. By explicitly optimizing the specificity-value trade-off (Thm. 2.1,, we provide a mo- tivating cample-compess entropy of a strategy rather than the state-action space.
result that scales with the
Figure 7. Strategist+EDD on Original Crafter. Episode Re- ward for vanilla EDE and Strategist+EDE with different strategies. Strategist+EDD with Fight, Resource, and Craft significantly out- perform vanilla EDE.
2. LLM-powered structure discovery. LLMs are used offline to inject high-level domain knowledge, producing a Strategy Tree. This separates the quality of strategic guidance from the availability of low-level language- conditioned controllers, potentially expanding its appli- cability robotics or healthcare workflows.
Strategy: Craft
Strategy: Right
ma ss stwoettooeoooo sword
make sto te sward- wwwlstshaattttmiaa
make word sword
40.0%
make, pickas
J013
make B00_0000_9900 sword
-8.1%
deteat somble
100
20
2 S4
15 206
300 50
7 Holasivo Change Stops 186
3. End-to-end integration. Our reward-shaping mecha- nism converts pure text into dense signals that any off- the-shelf RL algorithm can exploit. We demonstrate plug-and-pin gains with PPO and EDE, showing con- sistent improvements in convergence speed, final return, and behavioral diversity.
Change
Strategy: Needs
Stratagy: Ressource
collect_drint
25 0%
collec_coil
collect,
-28.0%
est cow
coplect_popling
0.0%
melee_up
20.0%
collect _wood
0.0%
-100-75 -50 -25 0
Reletive Change in Steps (%5)
25 30 15
-305-75- -50 -22 25 53 15 100
Relative Change Steps (%)
Figure& 8. Relative efficiency of Strategist+E.DE variants across relevant achievements on the original Crafter environment. Each subplot shows the change number of environment steps required to complete an achievement related to each strategy N times (N=100 by default, N 10 for rare achievements marked with *), relative to vanilla EDE. Green bars indicate the baseline failed to complete the achievement within IM steps. Each strategy significantly improves efficiency for its related skills, demonstrat- ing effective specialization.
Empirical Significance. Across three variants of Crafters, Strategist-guided agents (i) master sparse tasks faster than baselines, (ii) unlock qualitatively new behaviors (c.g.. farm- ing) that baselines never discover, and (iii) continue to add value when combined with Mfate-of-the-ar RL methods. These results position strategic guidance as orthogonal axis of progress, complementary to algorithmic model- architecture advances.
Limitations. First, LLM attribute proxies ($ 5.3.3) can be miscalibrated, leading to overexploration of unproductive branches. Second, once a + 0 and the rewards of the en- vironment dominate, agents that are not aligned with the intended strategy. We provide a more detailed discussion of limitations in $ E.
5.4.1. QUALITATIVE BEHAVIOR AND SPECIALIZATION
The benefits of strategic guidance are more evident in the specialized behaviors of the agents. Figure 8 illustrates the relative efficiency gains in different achievements: Strate- gist+EDE agents require significantly fewer steps to com- plete achievements aligned with their given strategy com- pared to vanilla EDE. For instance, the Craft-guided agent is better at creating tools, and the Fight-guided agent excels at achieving combat-related objectives.
revert to simpler strategies
Pathways Forward. To address these limitations, sev- eral promising directions emerge. Although we currently commit to one final strategy during training, allowing the agent to pivot among multiple strategies mid-training is a natural extension, particularly in nonstationary settings. Closing the loop by feeding back agent progress into strat- egy updates may help address model miscalibration. Further opportunities include making the Strategist more scalable and integrating action-level guidance. A detailed discussion of these future work directions can be found in $ E.
detailed visualization of the signature of the achievements reached by each strategy compared to the baseline is shown in Figure 16. Each Strategist+EDE variant focuses and suc- cessfully unlocks achievements relevant to its high-level strategy. This qualitative evidence suggests that agents are indeed adopting behaviors consistent with the abstract strate- gies provided by the Strategist, leading to more effective exploration and task completion in this complex setting.
With top-down RL, we do not learn to plan by acting - we act because we have already planned.

---

Strategic Planning: A Top-Down Approach to Option Generation
Acknowledgements
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint M.XXv:2204.05800 2022a
We extend our gratitude to the anonymous reviewers, area and program chairs, and members of the van der Schaar lab for their valuable feedback and suggestions. ML acknowl- edges sponsorship and support from AstraZeneca. AB ac- knowledges funding from Eedi. This work was supported by Azure sponsorship credits granted by Microsoft's s Al for Good Research Lab. This work was supported by Mi- crosoft's ' Accelerate Foundation Models Academic Research Initiative.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McK- innon, C., Chen, C., Olsson, C., Olah, C., Hernandez. D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J.. Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly. T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dddds Z.., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. Constitu- tional ai: Harmlessness from ai feedback, 2022b. URL https: / httpaaaxivvv...../.. 222/ 08073.
Impact Statement
We propose a top-down RL framework that explicitly uses human-inspired strategies to guide exploration, bridging the gap between purely bottom-up methods and real-world de- cision making in complex domains like healthcare, logistics, and policy. By structuring high-level plans and sub-goals, our approach can reduce sample complexity, mitigate costly mistakes, and uncover beneficial behaviors more quickly.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. Proceedings of the 26th Annual International Conference on Machine Learning. ICML *09, pp. 41-48, New York, NY, USA, 2009. Associa- tion for Computing Machinery. ISBN 9781605585161. doi: 10. 10.11445/5533374.155388.. URL ht https://doii /doi.
However, translating domain knowledge into strategies through LLM brings new considerations. If an LLM's es- timates of feasibility or value are miscalibrated-or if the underlying knowledge contains hidden biases-an agent may be steered toward suboptimal or even harmful plans. Robust oversight and validation of L.M outputs there- fore key, especially in sensitive applications.
org/110.144/155337444 1553333
Botvinick, M. M. Hicrarchical models of behavior and prefrontal function. Trends in Cognitive Sciences, 12(5); 201-208, May 2008. ISSN 1364-6613, doi: 10.1016/j. fics.2008.02.00)
Despite these risks, top-down RL has the potential to im- prove interpretability, safety, and adaptability in settings where traditional trial-and-error is impractical. By incor- porating strategic knowledge from L.MSs (or experts, if deemed necessary), the method can deliver more targeted exploration, potentially enabling faster convergence on high- value solutions in real-world environments that demand rigor and efficiency.
Botvinick, M. M., Niv, Y., and Barto, A. G. Hierar- chically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113 (3)26622280, December 2009. ISSN 0010-0277. doi: 11.1016i.cognniionn200000 200..088...0....000..
References
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforeement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M.. Zhang, C. E., Zhi-Xuan, T., Ho, M., Mansinghka, V., Weller, A., Tenenbaum, and Griffiths, T. L. Building machines that learn and think with people. Nature Human Behaviour, 8(10): 1851-1863, October 2024. ISSN 2397-3374. doi: 10.1038//41562-000 01011//222--2--1991999
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0162) 2022.
B.,
Bacon, P.-L., Harb, J., and Precup, D. The option- critic architecture. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI 17, pp- 1726-1734. AAAI Press, 2017.
Correa, C. G., Ho, M. K., Callaway, F., Daw, N. D., and Griffiths, L. Humans decompose tasks by trading off utility and computational cost. PLoS computational
10

---

Strategic Planning: A Top-Down Approach to Option Generation
biology, 19())=e0110887 June 2023. ISSN 1553-7358. doi: 10. 10.137//joumaa.ppbb.000000
Hafner, D. Benchmarking the spectrum of agent capabilities.
In Deep RL Workshop NeurlPS 2021, 2021.
Craik, J. W. The Nature of Explanation. The Nature of Explanation. University Press, Macmillan, Oxford, England, 1943.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXi::2301.04004 2023.
Dayan, P. and Hinton, learning. In Hanson, S., Cowan, J., and Giles, C. (eds.), Advances in Neural Information Pro- cessing Systems, volume 5. Bargan-Kaffmani 1992. URL https: neurips. cc cc/paper cc/paper_files// hiapee__flles//pprr//99///// d1 (4220066eaee03660933338888ee88 Paper. pdf.
G. E. Feudal reinforcement
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition, 2015. URL https: arxiv. /tgg/bss///eeooooeee/egeeeeeeeee 111.11111.1 03385.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting action- able knowledge for embodied agents, 2022.
Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., and Roiser, the Pavlovian System Sculpts Goal-Directed Choices by Pruning Decision Trees. PLOS Computational Biology, 8(3)xe0002410, March 2012. ISSN 1553-7358. doi: 10. 137 1371/joumaa..pch wwwp:ppppp....... 1002410.
Dietterich, T. G. Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition. Journal of Artificial Intelligence Research, 13:227-303, November 2000. ISSN 1076-9757. doi: 10. 3/3a.......3..
Bonsai Trees in Your Head: How
Du, Y., Yang, S., Florence, P., Xia, F., Wahid, A., brian ichter, Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., Kaelbling, L. P., Zeng, A., and Tompson, J. Video lan- guage planning. In The Twelfth International Conference on Learning Representations, 2024.
Jiang, Y., Kolter, J. Z., and Raileanu, R. Uncertainty- driven exploration for generalization reinforcement learning. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Dulac-Arnold, G., Levine, N., Mankowitz, J., Li, J., Paduraru, C., Gowal, S., and Hester, T. Challenges of real-world reinforcement learning: Definitions, bench- marks and analysis. Machine Learning, 110(9):2419- 2468, September 2021. ISSN 1573-0565. doi: 10.1007/ 110994-021-05961--0
Jiang, Y., Kolter, J. Z.,, and Raileanu, R.
On the Impor- tance of Exploration for Generalization in Reinforcement Learning. In Shirty-Seventh Conference on Neural Infor- mation Processing Systems, November 2023.
Kingma, D. and Ba, Adam: A method for stochastic op- timization, 2017. URL https: 1412.695
Even-Dar, E., Mannor, S., and Mansour, Y. Action climina- tion and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of Machine Learning Research, 7(39):1079-1105, 2006. URL http: //jmlr. org/ https//aperss/vv/e http///evveedae /eeeeeveeddarreeeeeee 6a. html.
Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenen- baum, J. Hierarchical Deep Reinforcement Learning: In- tegrating Temporal Abstraction and Intrinsic Motivation. Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Fruit, R. and Lazaric, A. with Options. In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th International Conference on Artificial Intel- ligence and Statistics, Proceedings of Machine Learn- ing Research, pp. 576-584. PMLR, 20-22 Apr 2017. URL www.hhtpps::: httpppp://pproeeeee /twcc//cceeiicc/oe ngs ml www.s/eesss..... fruit17a. html.
in MDPs
Lai, L. and Gershman, S. J. Human decision making balances reward maximization and policy compression. PLOS Computational Biology, 20(4):e1012077, April 2024. ISSN 1553-7358. doi: 10.1371/journal.ccl 1012057.
Gershman, S. J. and Niv, Y. Learning latent structure: Carv- ing nature at its joints. Current Opinion in Neurobiol- ogy, 20(2):251-256, April 2010. ISSN 0959-4388. doi: (0.10166).omb..0100022.2
Lashley, K. S. The problem of serial order in behavior. In Cerebral Mechanisms in Behavior: the Hixon Symposium, pp.
Wiley, Oxford, England, 1951.
Lattimore, T. and Hutter, M. PAC bounds for discounted mdps. In Bshouty, N. H., Stoltz, G., Vayatis, N., and Zeugmann, T. (eds.), Algorithmic Learning Theory, pp. 320-334, Berlin, Heidelberg, 2012. Springer Berlin Hei- delberg. ISBN 778-3-642-34106-
Gigerenzer, G. and Todd, P. M. Simple Heuristics That Make Us Smart. Simple Heuristics That Make Us Smart. Oxford University Press, New York, NY, US 1999. ISBN 978-0- 195512156-8.

---

Strategic Planning: A Top-Down Approach to Option Generation
Li, G., Cai, C., Chen, Y., Wei, Y., and Chi, Y. Is q-learning minimax optimal? tight sample complexity analysis. Oper. Res., 72(1):222-236, January 2024. ISSN 0030- 364X. doi: 10. 10.12877opre.2023... doi. oat.orgg10.0. 1ww.///rre..// 20.3.2222222
Newell, A. and Simon, H. A. Human Problem Solving.
Human Problem Solving. Prentice-Hall, Oxford, England, 1972.
URL https://
Ng, A. Y., Russell, S., et al. Algorithms for inverse rein- forcement learning. In loml, volume 1, pp. 2000.
Lieder, F. and Griffiths, T. L. Strategy selection as ra- tional metareasoning. Psychological Review, 124(6):
OpenAl al. Gpt-4o system card, 2024. URL https:
762-794, November 2017. ISSN 1939-1471. doi:
/ //aaxiiv.orrg///..o....o att../////b//21..112.......
10.10077/e000000055
Park, Y.. Margolis, G. B., and Agrawal, P. Position: Au- tomatic environment shaping is the next frontier in RL. Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Pro- ceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn- ing Research, pp. 39781 1999199 PMLR, 21-27 Jul 2024.
Liu, D., Kato, S., Gu, W., Ren, F., Yan, J., and Su, G. Inte- grating suboptimal human knowledge with hierarchical reinforcement learning for large-scale multiagent systems. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Pa- quet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp- 102744-102767. Curran Associates, Inc., 2024a.
Payne, J. W., Bettman, J. R., and Johnson, E. J. The Adaptive Decision Maker. Cambridge University Press, Cambridge, 1993. ISBN 978-0-521-41505-7. doi: 10.1017/38097811991199333
Liu, Z., Hu, H., Zhang. S., Guo, H., Ke, S., Liu, B., and Wang, Z. Reason for future, act for now: A principled framework for autonomous IIm agents with provable sam- ple efficiency, 2024b. URL https://aaat http:///axvvvv.ooo org/ abs/ 220s/2309..0 17382.
Pfeiffer, B. E. and Foster, D. J. Hippocampal place-cell sequences depict future paths to remembered goals. Na- ture, 497(7447):74-79 May 2013. ISSN 1476-4687, doi: 10.1038/nature 12112.
Ma. Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A. VIP: Towards Universal Visual Re- ward and Representation via Value-Implicit Pre- Praaniann In The Eleventh International Conference on Learning Representations, September 2022.
Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P.-Y. Automatic Curriculum Learning For Deep RL: A Short Survey, May 2020.
Ma, Y. J., Liang, W., Wang. G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-Level Reward Design via Coding Large Language Models. In The Twelfth International Confer- ence on Learning Representations, 2023. URL htsps: //openreview net htte://orrrmiidddddeedee
Precup, D., Sutton, R. S., and Singh, S. Theoretical results on reinforcement learning with temporally abstract op- tions. In European Conference on Machine Learning, 1998. URL https: / /ppi.. semanticscholal org/ :tss/CorpuTDD::86A4
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Fureka: Human-level reward design via coding large language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview net /oreet///rummm forum? 1ed--=IIEdUSSS5
Pternea, M., Singh, P., Chakraborty, A., Oruganti, Y., Mil- letari, M., Bapat, S., and Jiang, K. The RL/LLM Taxon- omy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models. Journal of Artifi- cial Intelligence Research, 80:1525-1573. August 2024. ISSN 1076-9757. doi: 10.1613/jair. 1.19600.
Miller, G. A., Galanter, E., and Pribram, K. H. Plans and the Structure of Behavior. Plans and the Structure of Behavior. Henry Holt and Co, New York, NY, US, 1960. doi: 10. 10000/0003990000
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http:
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D.,
// //jmlr.or wawlrr.rrg/ao wttp:///apppe//////001133333 htmml.
Rengarajan, D., Vaidya, G., Sarvesh, A., Kalathil, D., and Shakkottai, S. Reinforcement learning with sparse re- wards using guidance from offline demonstration. In International Conference on Learning Representations, 2022. URL https: http:///pppeeeeeeeee net/ forum?
Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518 (7540):529-533 February 2015. ISSN 1476-4687, doi: 10.1038/nature 14236.
id=YJ1 Wzg.WsM..
12

---

Strategic Planning: A Top-Down Approach to Option Generation
Sarukkai, V., Shacklett, B., Majercik, Z., Bhatia, K., Ré, C., and Fatahalian, K. Automated rewards via Ilm- generated progress functions, 2024. URL https:// arxiv.org org// abs/2410. 09187.
Vafa, K., Chen, J. Y., Rambachan, A., Kleinberg, J., and Mullainathan, S. Evaluating the world model implicit in generative model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:: htt.ps::/ppppeeeiiiiiieeee opw.reveeeeeeeeeeeeeeeeee net/ peet/forrue forum? id=aVK4JPpegy.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms, August 2017. arXiv:1707.cm .06347 [cs].
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. Feu- dal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Ma- chine Learning - Volume 70, ICML 17, pp. 3540-3549. JMLR.org, 2017.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via sule-super world models. In International conference on machine learning, pp. 8583-8592. PMLR, 2020.
Siegler, S. Emerging Minds: The Process of Change in Children's Thinking. Oxford University Press, Septem- ber 1996. ISBN 978-0-19-5077885 doi: 10.1093/oso/ 9780195007/872.001.00 10001.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An Open- Ended Embodied Agent with Large Language Models, October 2023, 1rXiv:2305.16291 [es].
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchhrenner, N., Sutskever, I., Lillicrap. T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-48) January 2016. ISSN 1476-4687. doi: 10. 1038/nature 16961.
Watkins, C. J. C. H. and Dayan, P. Q-learning, Machine Learning, 8(3):279 292, May 1992. ISSN 1573-0565. doi: 10.1007/BF00999268.
Wei, X., Kumar, N., and Zhang, H. Addressing bias in generative Al: Challenges and research opportunities in information management. Information & Management, 62(2):104103, 2025. ISSN 0378-7206. doi: 10.1016/j.im 2025.104105.
Simon, H. Rational choice and the structure of the en- vironment. Psychological Review, 63(2):129-138, 1956. ISSN 1939-1471 doi: n0.1037/h0042769.
Wu, Y., Tang, X., Mitchell, T., and Li, Y. Smartplay: A benchmark for ilms as intelligent agents. In ICLR, 2024.
Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q., Zhong, V. Yang, Y., and Yu, T. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: openreview. net/ forum? :M.
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Prog- prompt: Generating situated robot task plans using large language models, 2022.
Strehl, A. L., Li, L.. and Littman, M. L. Reinforce- ment learning in finite mdps: Pac analysis. Jour- nal of Machine Learning Research, 10184):2413-2444 2009. URL http:/ Strehh09a.hhm..
Yan, X., Song, Y., Feng. X., Yang, M., Zhang, H., Ammar, H. B., and Wang, J. Efficient reinforcement learning with large language model priors, 2024. URL https: / //arxiv. hts//bb//2000 22100.410..0 07927.
Strickland, C. and Lizotte, D. Hierarchical Reinforcement Learning for Decision Support in Health Care. Proceed- ings of the Canadian Conference on A Artificial Intelligence, jun 8 2021. http:://caac.pubpub.. htggpub/u8c2799ii
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In Interna- tional Conference on Learning Representations, 2018.
Zhang, S., Zheng, S., Ke, S., Liu, Z., Jin, W., Yuan, J., Yang, Y., Yang, H., and Wang, Z. How can Ilm guide ri? a value-based approach, 2024. URL https: / //axxiv.. org/ abs//bb//at /240222222.. 16181.
Sutton, R. S., Precup, D., and Singh, S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence. 1 112 (1):181-211, 1999. ISSN 0004-3702. doi: 10.1016/ 50094-3702299000052221
Tolman, E. C. Cognitive maps in rats and men. Psychologi- cal Review, 55(4):189-208; 1948. ISSN 1939-1471. doi: 10. 10.1107//0061122
13

---

Strategic Planning: A Top-Down Approach to Option Generation
A. Foundational Bounds for the Policy-Edentification Meta-Algorithm
The results collected this section justify why, under a suitably specific prior over policies and simple probability threshold, one can narrow the search to finite, provably small subset candidate policies. The concrete algorithm we ultimately evaluate in the main paper is different and does not rely directly on the lemmas below; the present derivations trade-offss that motivate our design choices.
serve only to illuminate the
Entropy, probability thresholds, and support size
Let ōn be probability distribution over a (finite or countably- infinite) collection policies {TiJiez All logarithms are natural. Assume bounded (Shannon) entropy
((n))
Σ (m,) (n;)) I ll
where Sy 2 is user- specified specificity budget. For any threshold (0, 1] set
S { :u(() 2 7).
Lemma A.1 (Entropy-threshie theeehhodeo support bound). For every T E (0, 1].
|St & min 2+
Qx In(1//)
}- (+)
The second term in (+). 2 Qar [m[[//]] (where the fraction is taken as oc ifT = 1 and 2M > O, and as 0 if = 0 making the term 2), shows the impact of the entropy budget. Reducing aas lightens this part of the bound. This second term is strictly smaller than 1/T T < 1/2 and Qs < (1-22) (1-2r)
2T) [n(]))n())))
Proof. Since each E S, has mass at least T, 12 ETEE, 6n(T)) 2 21 |s, $ 1/T.
Heavy and light atoms. Partition S, into
H { E Sy 5π((T) 1/e), L Sy \ H.
Because EMEH ÓT(TT= & and 6n(x) > 1/e H, | < and hence | < 2.
Entropy of light atoms (only if L + 2). If L. * Ø, its elements satisfy A 6n 6π(T) S 1/e. The condition 6π((T) S 1/e implies that 7 S 1/e is necessary for L to be non-empty. The function f(x) = - In x has derivative f'(x) = In 1. For x < 1/e, f'(x) > 0, so -r æ is increasing on (0, 1/e]. For every क E L we have + S δπ(() S I/e. Therefore
2 -Tln Il(11/))
I = 1/=11/,77 (l(//) = 1/e. I < 1/e, - In(/T)) is positive. Hence, for E (0, 1/e], +
PM 2 Σ (T) 2 [L|T/(L//T)
so |L]| S PM (7 (1()))) If > 1/e, then = 0. so |[] = 0. In this case, the inequality [L] s aas (nl[/n((1//) also holds (as is less than or equal to non-negative quantity, assuming 7 In(1/r) > 0 for E (0, 1)).
Combine. If L = & (e.g., if 1/c), then |S,| H 2.
If L + 0 (which requires 7 < 1/e), then
|St = H + [L| S 2 2M
Thus, the bound | 2 + PM / /tw//TIin/////// ln(T/)]] holds for all 7 E (0, 1). Taking the minimum with the normalisation bound
1/T completes the proof for E 1). For 7 = 1: (m)) = 1 means n. E H, so L = 0. |S,| = -HH. Since probabilities
sum to I, at most one policy can have probability 1, so S < 1. The bound formula (*) yields min oo) = if 2. > 0. and min(1,2] 2} if Q2 = O.
14

---

Strategic Planning: A Top-Down Approach to Option Generation
Implications. At policies individually hold probability > 1/e, which explains the constant Entropy alone cannot guarantee finite |S-] for arbitrarily small 7 if the set of policies is infinite; but in conjunction with positive floor it yields tractable candidate set whose size is explicitly controlled by Sw and shown in the lemma.
A.1. PAC Identification from Finite Set of Policies
Consider finite collection of candidate policies { Tp). R4 E [0, I]; and since the episodes are independent and each episode starts from the same initial-state distribution, within each policy the returns are i..dd Let
Executing म, for episode of length K yields return
M4 = E[R,),
max Pi- ISISP
Uniform-sampling algorithm. Fix accuracy (0, 1] and confidence å (0, Sample each policy
2 In AP)
m
episodes,
compute the empirical means ii Proposition A.2 (Confidence PAC guarantee). The uniform-sampling algorithm outputs policy ãt such that
R and return the policy म = Targ max, ft..
1=1
Pa 11"
with probability at least The total number of episodes is = Pm= = (n))))
Proof. For any fixed i, Hoeffding's inequality gives
Pr jos - mil x 8) Pr(lis-all)
S 2 2exp mm=²)
2P
where the last step follows from the definition of m. A union bound over all P policies ensures that, with probability at least 1 - 8, the event líz Hul S e/2 holds simultaneously for every i. On this event
A= 2 A.
5. μs 2 A 1 1
The episode count is N = P m, completing the proof.
Instance-dependent refinement. One can significantly tighten the bound by making it gap-dependent, eliminating policies that are clearly suboptimal earlier and not sampling all policies uniformly (Even-Dar et al., 2006).
A.2. From Episodic to Discounted Infinite Horizon
Many sequential-decisionn problems use policy by its discounted return
G =
7 E e [0,
1=0
Because the sum is infinite, we truncate after T steps and control the bias and variance this incurs.
Lemma A.3 (Confidence estimation of discounted value). Let {TI,..., be fintte set of policies in discounted MDP with factor E (0, 1). Fix accuracy 6 E (0, 1] and confidence E (0.
Truncation horizon. Choose
In
T
T (e,7)
2(1
((1-))
2
Sampling rule. For each policy collect independent trajectories of length T, where
2
>
In
4P
7)2
15

---

Strategic Planning: A Top-Down Approach to Option Generation
Guarantee. Let R, Ee=o rrr) be the truncated return on trajectory L for ni and set iis m Ei=1 R.". Then, with probability least ò. 6.
[. (4-14 14| S
for all i I.....P.
where = [ is the true infinite-horizen value.
Proof. Bias. Since rewards lie in [0, I].
S G. R
Σ
(1)
S 8
7 7
2
k=T
so E mil= S E/22.
Variance. Each truncated return satisfies S S (1 -1- (1-))((0-)) 11/11 -7). Hoeffding's ' inequality with range R 1/(11 --- 7) gives, for any
PI(
E[R
> 8) S 2exp 2m (1) -
5
2p
where the last step uses (+). A union bound over the P policies yields line - -E(R( S e/2 for every with probability at least 1 - 5/2.
Combine. On the intersection of the bias and variance events we have lá: م4l for all Because the variance event already occurs with probability
8/2 2 1 - 5, the stated guarantee follows.
Implication for identification. Running allocating the sample budget (+) in place of the episodic bound preserves all guarantees. The gap-independent complexity
(Section A.1) trajectories of length T and
becomes
((1-7)~Pp3
A.3. High Specificity & Near-Optimal Value + Small Sample Complexity
Problem setting. Consider an MDP .M whose optimal value J(x²) is attained by policy x². For the episodic case, episode
returns are normalised to [0, 1]. so (TT)) < 1. For the discounted infinite-horizon case with E [0, [].11, JJT') may be up to 1/(1 - 7). Auxiliary knowledge IC provides a prior distribution 5n E (TTM) Given tolerances (Hmax, Umin; 7) with mmax 2 0 and Umin; 7 E (0, 1], assume
(a) Specificity: (ōn) & Huuxi
(b) Value: there exists a policy Tn such that (n(x)) 2 r and J(r+) 2 J(r) Umin-
Thresholded support.
With probability floor 7 define the candidate set
S = {x 6n())) 2 r)
Lemma A.1 gives the deterministic bound
2 Hmax
SS Pr mink F 7 IIn(1/7)
Since n E S,, it holds that ( ) = max J(T) > ((T!) ((TT)) Vmin Theorem A.4 demonstrates that by applying UNIFORM-SAMPLINE to S, with an algorithm accuracy parameter Ealg = Uninn// /2, policy स can be identified that is (2))mmm--piima relative to (Tx)) (i.c., J(A) 2 ((xT) - (3/2)umin..
Theorem A.4 (Specificity and value yield PAC efficiency). Let ( ) = maxaes, J(T). Under assumption (b), J > J(x²) Umin- Run SELIFOR--SMMLLNG (Proposition A.2) on the set S, with accuracy parameter & = Vonto and confidence
2
E (0,1). 1). The algorithm outputs â such that ( 2 J( E with probability at least 8. Consequently.
J(A) 2 J( 33/2)mmin with probability at least
16

---

Strategic Planning: A Top-Down Approach to Option Generation
Episodic returns. If each episode relurn lies in [0,1] (hence ( values are in [0, 1), the total number of episodes
In(P./0) (where Umin/
N S C.P.
E²
for universal constant C 8.
Discounted infinite horizon. With discount factor 7 E (0, 1) and per-step rewards T E [0, 1] (hence J(x) values can be up to 1/(1 - 7)), let 8tem = s/2 = Umin// /4 be the accuracy largel for Lemma A.3. Using trajectories of length T (Stem 7) (from Lemma A.3, with its accuracy parameter set to Etem) and budget m from Eq. (+) (also with its accuracy parameter set to Etem) yields
S Pr In(P+/8)
(where 22 is the overall algorithm accuracy)
(1 (1-7) ?2222²
for another universal constant C' 2 8.
Proof. (i) Size of the candidate set. Equation (2) gives |r,| S Pr.
(ii) Achieved Optimality. As established, under assumption (b), n! E S, and J( = max J(x) 2 J(T!) 2 J(x*) ) mmin- The UNIFORM-SAMPLING algorithm applied with its accuracy parameter (denoted Ealg in Proposition A.2) set to Umin/ 2. By Proposition A.2, the output satisfies J( Jš, (i.e., (+) 2 Jš.- Umin /2) with probability at least 1 6. Therefore, J(t) 2 (J(x*) Unun)) Uniin/ J(x²)) (3/2))min- Thus, the policy ø is (3/2)u (2))nmnnntpiia
relative to (
(iii) Episodic complexity. Applying Proposition A.2 to S, (of size at most Pt) with an overall algorithm accuracy parameter E = Umin/2 / yields the stated bound. Proposition A.2 itself requires mean estimates to be E/2 = Vmin/ / accurate. The constant accounts for the coefficient in Proposition A.2 (which 2/(())2)2 8 effectively) and any effects of the ceiling function.
(iv) Discounted case. For the UNIFORM-SAMPLING SAMPLING algorithm achieve overall accuracy of Vmin /2, it requires estimates of policy values J(Ts)) that are accurate to s/2 = Uminn/4 /4. Thus, Lemma A.3 is invoked with its accuracy parameter set to Elem
/min/4.
The number of trajectories m per policy from Lemma A.3 is then
2
32 min (1 -7)3
fiem 71 In(4P//))
In(4P,/0)
The total number of trajectories is N = P.m
Interpretation. Low entropy compresses the candidate set through the H hnt////rnn//////// In(1/r) term in Eq. (2). This entropy- dependent component of the bound is smaller than /rr if Himx (1 2r) In(1 /r) (which implies < 1/2); otherwise, the 1/T cap may be tighter or equally constraining, Consequently the episode complexity scales linearly in the entropy- controlled support size Pr, polynomially in 1/E (where : = 2min 2 is the overall algorithm accuracy parameter), and only logarithmically 1/8.
17

---

Strategic Planning: A Top-Down Approach to Option Generation
B. Further Implementation Details
This appendix provides additional technical details to ensure clarity and reproducibility.
B.1. Aggregation for Plan Node Attributes
Recall from the main text that a plan node decomposes a higher-level goal into multiple sub-goals. Let hippan be a plan node with children {hs,1,..., himm) J. each corresponding to a sub-goal.
Feasibility. For plan nodes, the overall feasibility can be estimated as the minimum feasibility among the sub-goals, reflecting the fact that all sub-goals must be achievable:
(he
min JE(I....m)
fha,)).
In practice, to stabilize LLM predictions or incorporate domain knowledge, could weighted combination of sub-goal feasibilitics. However, the minimum operator is the simplest and most intuitive choice.
Value. Because each sub-gote goal contributes to the overall objective, the final value can be viewed as an aggregated measure of sub-goal values. One straightforward strategy to assume independence and multiply sub-goal values:
m
y (hi.plan)
I (ha.).
Alternatively, an average or maximum can be used if the sub-goals are substitutable or if certain sub-gout sobbgoaas are more critical than others. In our experiments, we primarily adopt the multiplicative approach.
Specificity. Plan nodes combine multiple sub-goals that the policy must execute. If H (8% measures the entropy of each sub-goal's policy distribution, approximate the overall entropy by the sum (or another suitable aggregator) of sub-goal entropies:
m2
H H(8 (8hiiplon)
Σ
(6
This ensures that each sub-goal reduces the space of valid policies, consistent with a stricter overall plan.
Nonetheless, these are non-central elements in our work and, fact, the strategy trees we obtain in practice usually place plan nodes leafs.
B.2. Tree Expansion and Selection
As described in the main text, the Strategist builds and refines its Strategy Tree in an iterative manner. Let L be the current set of leaf nodes; each leaf is either an approach node or a plan node that has not yet been fully expanded. To decide which leaf expand next, pick
Anext arg maxx from C
[ ho)) & (him [ho)]
where ho is the root node, and
1 II
m II j=1
(him. ho)
[(b,, big .), x (hem ho) j=1
y (hi
hig .)-
Here, ff {him. ho) represents the root-relative feasibility (likelihood that this path of strategies is implementable). and y (hem. ho) represents the root-relative value (potential return of following that path).
Strategy Decompasition Prompting. Once bast is chosen, we prompt the LLM to refine it. Depending its node type:
Approach Node Expansion. The LLM is asked for a set of mutually exclusive "approaches" that refine the parent node's goal (e.g., different ways to gather resources). We then evaluate each approach in terms feasibility, value, and specificity.
18

---

Strategic Planning: A Top-Down Approach to Option Generation
Plan Node Expansion. The LLM is asked for a list of sub-goals steps that collectively form plan. We then compute aggregated feasibility and value as described in $B.1.
Termination Criteria. Expansion continues until any leaf node meets certain specificity thresholds that are implicitly evaluated by the L.LM:
11 (8tteaat
S H
max.
If multiple leaves reach the threshold, we select the leaf with the highest f(+) V(() product as the final strategy.
Additional stopping conditions. We limit the maximum depth of the tree to avoid too many queries, but the limit is not hit in our runs.
B.2.1. How ENTROPY IS HANDLED AND SPECIFICITY ACIIEVEE
In our framework, specificity is key property of a strategy distribution ôn, ensuring that the strategy concentrates on a relatively small set of promising policies, thereby reducing the search space for the RL agent. We quantify specificity via the entropy (ōn) of the policy distribution associated with strategy node. A strategy is considered "sufficiently specific" if its entropy H (Ōn) is below a certain threshold Hmar, as defined in $ 2.
Within the STRATEGIST agent's implementation, an explicit numerical calculation of entropy (5n) for each node using the LLM is not performed during the tree construction phase. Instead, specificity is primarily achieved by design through the hierarchical nature of the Strategy Tree. As described in $ 4.3, child nodes are intended to represent increasingly specific policy subsets of their parents. This means that by construction, the policy distribution 8th.,, associated with child node ha. should have lower entropy than that of its parent h4 i.., H (8 (8).)..
The LLM is prompted (see $ B.3) to generate sub-goals that are more concrete and narrower in scope than their parent goal. The threshold Hmaz serves as a conceptual guide in the Strategy Problem formulation ($ 2). While not directly calculated and checked against Hmax by the LLM for each node, the process of decomposing goals into more detailed sub-goals implicitly drives the entropy down.
B.2.2. How VALUE ESTIMATED
In the context of the Strategy Tree ($ 4.3), the value (h(4 of node h, is estimate of how much of the parent node's s potential optimal return is preserved by focusing on the strategy 8hh, associated with node h; It's defined as the ratio of the maximum expected return achievable within the support of 8hh to the maximum expected return achievable within the support of its parent's 's strategy distribution 8hyrren The term Umin in the Strategy Problem formulation ($ 2) relates to this by
ensuring that J( 2 (1 "min) meaning the best policy within the strategy's 's support is at least (1 Umin) times as good as the true optimal policy n².
During the construction of the Strategy Tree by the STRATEGIST agent, the value attribute for each node is initially estimated by the LLM. The LLM is prompted (see $ B.3) to assess this based on its domain knowledge and the textual description of the goal associated with the node. This is a heuristic estimate. The STRATEGIST agent does not perform rollouts or direct calculation of (ir) during the tree generation phase to determine these values. Instead, it relies on the LLM's zero-shot estimation capabilities. As discussed in & 5.3.3, these initial LLM estimates can be refined with empirical data.
B.2.3. HOW FEASIBILITY ESTIMATED
The feasibility f(h,) of a strategy node h4 in the Strategy Tree quantifies the likelihood that the set of policies 8h associated with that node actually be implemented by the learning agent within the given environment and its constraints. As formally defined in $ 4.3, ((h+) = 8h. (Magent). which is the probability mass of 8h. that falls within the set of policies Uagent implementable by the RL agent. This probability represents how much of the current strategy aligns with policies achievable by the agent's learning hypothesis class.
Similar to V. the feasibility attribute for each node is estimated by the LLM (see B.3) based on its understanding of the task and the environment.
19

---

Strategic Planning: A Top-Down Approach to Option Generation
B.2.4. HOw INITIAL LLM ESTIMATES CAN BE UPDATED
The initial estimates feasibility and value for each node in the Strategy Tree are provided by the LLM in zero-shot manner. These are heuristic estimates and can be imperfect.
As suggested in 5.3.3, these initial LLM estimates can and ideally should be dynamically updated based on feedback from the agent's actual interactions with the environment. As the RL agent attempts to execute strategies and sub-goals, its performance (e.g., achieved rewards, completion rates) can be used to refine the "feasibility" and "value" attributes stored in the tree. This "closing the loop" by feeding back agent progress into strategy updates is highlighted as an important area for future work. The experimental setup presented primarily uses the initial LLM estimates for strategy selection but then empirically measures the realized feasibility and value compare against these initial heuristics.
B.2.5. BUILDING THE STRATEGY TREE
We instantiate the Strategist agent using a GPT-4o model (OpenAl et al., 2024), using the prompts provided in $ B.3. After building the entire Tree following the process detailed above and in $ 4, we run another instance of GPT-4o to refine the estimates of feasibility and value using the prompt in Listing 2 below.
B.3. LLM Prompts
LLM Prompting Details.
Below are the different prompts used to query LLMs in our experiments.
strategy'_context: I
You are going to previde the high-level strategy for goal by proposing different options and breaking the problem down into progressively Lower-level subgoals.
The structure goals and sub-goals will be organised in a tree strueture, where
the root node is the overall goal, and children of node are different strategies, ie options of subgoals that can each help fulfill the parent node. It is important to note that the children of node represent different independent** strategies. Each branch should be viable way of getting to the overall without needing any other branch of the tree to be completed.
Each node has two key measures: #*Feasibilityee: Confidence that this goal be achieved, assessed independently of the parent.
- A feasibility of 1 means the goal is certain to be achieved.
A feasibility of 0.5 reflects 5018 confidence in the goal's achievability.
**value**: How restrictive the present goal is compared to its parent in terma of their potential to maximi se expected returns. sspecially between possible sub- goals, the higher value subgoal should correspond to the one that expected to maximise expected rewards for the overall objectve.
value of means achieving this geal is as good as it gets for the parent goal.
12
value of 0.5 suggests that achieving this goal expected to yield half parent's potential returns.
13
14
The tree will be built iteratively, starting with the root node. At each step, the most promising leaf node will be selected based on the highest product of feasibility and value, and you'll be provided with its root-to-leat path as a 1ist of nodes.
15
You one of the following but not both in the same response:
17
1. Break down the leaf node into children (use this only if you think the node should be broken down further, avoid unnecessary complexity), or
18
2. Leave it as leaf node if it's specific enough to represent clear, lox-level actions that can be easily executed.
20
the root-to-lcaf path includes sequential nodes, you will receive multiple leaf nodes. Apply the same process to each leaf, one after the other.
The root node (id=1) will already be in the tree. You will add children nodes to break it down. Before making decisions, reflect on:
23
1. Whether to break down the node,
20

---

Strategic Planning: A Top-Down Approach to Option Generation
2. which nodes add, and
25
3. How you assess their feasibility and value.
26
Use the appropriate command only when you are confident in choice of subgoals and feasibility/valu estimates.
28
Finish your response after giving your commands. Do not rehcarse your commands. Once you give them it should be the end of your response.
29
32
tree _comandds:
33
### Command Structure for Modi fying the Tree
To execute command, write it plain line without quotes or symbols. The available commands are:
36
37
++ADD| [goal description] [parent id (int) ] [feasibility] [value]**: **: Add a new node.
38
AAMODIPYI [id [ feasibilityl [new value] **: Modify the feasibility and value of an existing node.
39
**REMOVE| [id (int) +*: Delete a node and all its children.
**ADD\_ SEQUENTTAL "Goal 1;Goal :..al N" [parent id] [feasibilityl [valuel [feasibilityN] [valueN] **: Add multiple sequential subgoals to the specified parent node. Goals are separated by semicolons. Make sure there are as many feasibility and values as there are goads.
[feasibilityy]] [value2]
41
**RKAD\_TREEe++ read the current tree
*MEND'_TREEE*: Use this if the current strategy cannot be broken down any further Use this command only you are **absolutely.. sure that you generation process.
end the
43
Note: do not use the character "I" inside the goal description.
Once you have executed command or a set of command, finish your response. Do not write anything clse after using series of any of the commands above. You will have the opportunity to read the tree make further modifications at the next iteration.
45
providel_ber L___ettt_t_dddtee
Here is the current mest gromising root-to-leaf path the tree:
48
49
Loo
Please evaluate this path and choose either:
52
1. Break down the leaf node into children (use this only you think the node should be broken down further, avoid unnecessary complexity), or
53
2. Leave it as leaf node if it repzesents clear, low-level actions that an agent can easily execute.
55
If you break cown the node, explain your reasoning and use the ADD command add new children.
56
If you leave it as leaf node, justify why it's specific enough. Then, you can ask to read the tree with READ\_TRER, freely modify the nodes of your choice. you
any further changes the tree, terminate the process with KNDY_TRRR.
want
58
Ensure you reflect carefully about the factors that can influence the feasibility and value of any nodes you add, and provide final estimation.
Listing I. Base Prompt for building the Strategy Tree
will provided with a tree showing strategic options towards an overall goal located in the root node. You will be asked to evaluate the foasibility and value of each node, based on the following definitions:
Feasibility: How likely is the current node's a strategy can be successfully implemented in the rovmmntttt? score between 0 (very unlikely) and 1 (very likely).
21

---

Strategic Planning: A Top-Down Approach to Option Generation
Value:
If the current node's s strategy was achieved, how good would achieving **its parent goal**, relative to other potential strategies? A score between 0 (very bad strategy, e.g., even if achieved, achieving it does not imply fulfilling the parent goal) and (optimal way to reach the parent goal and maximise the reward) A high-value node the parent goal. maximum value goal should correspond to the strategy that can yield the highest value for the parent goal.
towards
be sufficient its own to achieve
* **Make sure to evaluate each node strictly part of the node description**.
the basis of the information provided
Do not evaluate nedes downstream of node that has flag 'sequential: True'. Evaluate the sequential node as usual but ignore its downst nodes.
For the root node (id-1), just set feasibility to 1 and value to 1.
For each node, reason step by step to reach your estimated feasibility and value. Please format the final outcome between <answer> and </answer> feasibility' :.., :.., 'value':... ...
dict: (rid':
Listing 2. Value and Feasibility Metrics
Given the goal ' (goall', analyze each of the (n\_states) states provided by describing how close far it is from achieving the goal.
Hased on this analysis, assign a score between 0 and 5 to each state, where:
0: The state is very far from the goal.
1: The state is far from the goal.
2: The state is sonewhat far from the goal.
3: The state is scmewhat close to the goal.
4: The state is close to the goal.
5: The state is very close to the geal.
11
**Examples of State Characteristics:e.
12
S*ooee 0 or 1:**
13
- No cows in sight.
No saplings for breeding.
15
Low health due to unmet survival needs.
16
17
Score 2:**
Survival needs are stable.
19
Maybe saplings inventory but no cows sight.
20
21
**Soore 3 or 4:**
One or mere cows in sight.
Health and survival metrics not critical.
24
25
+*Score 5:*+
26
- A cow adjacent to the agent and saplings in inventory for breeding.
28
Only take into account characteristics that are directly related the goal. For example, if the strategy invoives breeding, take saplings into account in your scoring; if the goad is simply hunting, ignore saplings in your evaluation; only consider health and vitals it they are explicitly stated in the goal.
Do NOT repeat parts of the game descripticn such as the different objects in the game or the game objective.
30
Only reflect on things that are directly related to the states and the goal.
31
Once you reach a conclusion, always give final answer with the following format: scORES:<scons for state 1>, <score for state 2>,..., <score for state (n\_states)>
34
35
For example, if scores of 1, 5 and 4 are assigned to state 1, 2 and 3 respectively, your answer would finish with: 5, 4"" Make sure to finish your response with this final 1ine: "SCORRS:"
Listing 3. Reward Shaper
22

---

Strategic Planning: A Top-Down Approach to Option Generation
goal'Lcontext
The environment is: Crafter
The overall goal is: "Collect as much cow neat as possible. Bach item of meat collected by killing cow counts for 1 point. Your goal is to maximise the number of points collected within episode of the environment."
game :
Crafter is open-werld survival game used reinforcement learning (RL) benchmark. The game focuses on evaluating an agent' general capabilities by presenting ng a procedurally generated, 20 envi rorment where agents interact through vi visual inputs. Crafter is designed for deep exploration, long-term reasoning, and generalization, making it rich environment for Rhi research.
Environment Overview:
10
World Generation: Each episode takes place in a procedurally generated 64x64 grid world. Terrain types include:
Grasslands
12
Forests
13
Lakes
14
Mountains
Caves
The terrain determines the distributicn of creatures and resources.
17
18
Game Mechanics:
19
1. Survival Needs:
Health:
The player/agent has health points (HP), which are depleted by hunger, thirst, fatigue, or attacks from monsters. regenerate when hunger, thirst, and fatigue are satisfied.
21
Hunger: Decreases over time; satisfied by esting meat (from cows) or plants.
Thirst: Decreases over time; satisfied by drinking from lakes.
Rest: Decreases
time; replenished by sleeping in safe areas.
24
If any of these values (hunger, thirst, rest) reach zero, player starts losing health.
25
2. Resources:
Resources can be collected and used craft tools. They include:
28
Wood (frem trees)
29
Stone
Coal
Iron
32
Dianosds
33
Saplings (plants that can be grown for food)
34
3. Crafting and Technology Tree:
Players
crafting table and furnace craft tcols: Wood Pickaxe and Wood sword Stone Pickaxe and Stone Sword Iron Pickaxe and Iron Sword
37
38
39
Higher-tier tocls like the iron pickaxe and sword require more advanced resources (iron, coal) and improve efficiency in collecting resources and defeating enemies.
41
4. Creatures:
Cows: Source meat (food) Can be found in grass areas.
44
45
Game Interface:
45
Observation: Agents receive 64x64 pixel images representing a top-down view of the world, including 8x8 grid area surrounding the player and inventory display (health, hunger, thirst, rest, collected materials). Action Space: A flat categorical space with 17 actions, including movenent, interacting with objects (e.g., gathering resources or attacking), crafting tools, and placing objects.
47
Other important cemments:
23

---

Strategic Planning: A Top-Down Approach to Option Generation
50
Blocks that can be freely moved through are: grass, path, sand.
51
Blocks that can be placed and easily destroyed are: stone, tree, table, furnace. Blocks that can be destroyed are: coal, iron, diamond, water. Blocks that can neither be moved through ner destroyed are: lava.
52
Meat can only collected by killing cow killed by hitting it time. No weapons or tools are required to kill
Cows be found in grassy areas.
55
Cow.
Cows can be bred by feeding them 1 sapling. In such case, new cow will spawn next 56
the first
57
The player can move in four directions: up, down, left, right.
Listing 4. Crafter Easy Environment
goal conntext:
The environment is: Crafter
The overall goal is: "Collect much cow neat as possible. Each item of meat collected by killing
counts for point. Your goal is to maximise the number of points collected within one episode the environment."
game Lemet__ontext:
Crafter is open-werld survival game used reinforcement learning (RL) benchmark. The game focuses evaluating agent's general capabilities by presenting a procedurally generated, 20 envi envmentment where agents interact through visual inputs. Crafter is designed for deep exploration, long-term reasoning, and generalization, making it rich environment for RL research.
Environment Overview:
World Generation: Each episode takes place in a procedurally generated 64x64 grid
world. Terrain types include:
Grasslands
11
Forests
12
Lakes
13
Mountains
14
Caves
The terrain determines the distributicn of creatures and resources.
16
17
Game Mechanics:
18
1. Survival Needs:
Health:
The player/agent has health points (HP), which are depleted by hunger, thirst, fatigue, or attacks from monsters. regenerate when hunger thirst, and fatigue are satisfied.
20
Hunger: Decreases over time; satisfied by eating meat (from cows) or plants.
Thirst: Decreases over time; satisfied by drinking from lakes.
Rest: Decreases
time; replenished by sleeping in safe areas. If any of these values (hunger, thirst, rest) reach zero, player starts losing health.
23
24
2. Resources:
Resources can be collected and used craft tools. They include:
27
Wood (frem trees)
28
Stone
Coal
rrnn
amonds
32
Sapling ings (plants that can be grown for food)
33
3. Crafting and Technology Tree:
Players
crafting table and furnace craft tcols: Wood Pickaxe and Wood Sword Stone Pickaxe and Stone Sword Iron Pickaxe and Iron Sword
36
37
38
Higher-tier tocls like the iron pickaxe and sword require more advanced resources (iron, coal) and improve efficiency in collecting resources defeating enemies.
24

---

Strategic Planning: A Top-Down Approach to Option Generation
41
4, Creatures:
42
Cows: Source of meat (food). Can be found in grass areas.
Game Interface:
45
Observation: Agents receive 64x64 pixel images representing a top-down view of the world, including an 8x8 grid area surrounding the player and an inventory display (health, hunger, thirst, rest, collected materials). Action Space: A flat categorical space with 17 actions, including movement, interacting with objects (e.g., gathering resources or attacking), crafting tools, and placing objects.
47
Other important comments:
Blocks that can be freely moved through are: grass, path, sand. Blocks that can be placed and easily destroyed are: stone, tree, table, furnace. Blocks that can be destroyed are: coal, iron, diamond, water. Blocks that can neither be moved through nor destroyed are: lava.
50
51
52
Meat can only be collected by killing By default, a cow killed by hitting it times. When using a sword however, only hit is required.
Cows can be found in grassy areas.
54
55
Cows can be bred by feeding them 2 saplings. In such case, a new cow will spawn next to the first one.
The player move in four directions: up, down, left, right.
Listing 5. Crafter Medium Environment
B.4. Reward Shaping Details
Reward Shaping Network. We use ResNet-18 (He et al., 2015) to predict sub-goal progress scores for each environment
state. Key architectural choices:
Input shape: 64 x 64 image frames.
Output dimension: continuous score R(s) E [-1, 1]. using tanh as the final activation function.
Data Augmentation:
We augment the labeled dataset by a factor of 10 using mixup (Zhang et al., 2018).
Loss function: Mean-Squared Error (MSE) loss.
Training setting: We train the model for 10 epochs on the augmented dataset.
We backpropagate using the Adam optimizer (Kingma & Ba, 2017) with learning rate 1- 10
4, 10-4,
0.9, 33 0.999.
Data collection pipelines: We collect 5000 frames: 4000 from 200 trajectories of PPO agents pretrained on the environ- ment, and 1000 from the expert human demonstration dataset provided with the original Crafter environment (Hafner, 2021). Observations are converted textual data using Smartplay (Wu et al., 2024). We iteratively sample states, provide their textual descriptions GPT-4o-mini model, and the prompt for reward shaping (provided above) generate reward signals.
Combining Rewards. The final reward at time t
R((s)) B
The (st, ax) (1 ((()).)) + a
with 3 a normalization constant that we set to 10 to balance the shaped reward with the original reward R. We use an inverse square-root decay schedule for a, either from to between 50k and 1M ("short schedule") or from 0.9 to 0 between 500k and 1.5M ("long schedule"). piecewise or adaptive schedule could be employed if the environment exhibits dynamic difficulty sub-goals are naturally learned in stages.
25

---

Strategic Planning: A Top-Down Approach to Option Generation
B.5. RL Algorithm Details
PPO Hyperparameters. In our experiments, use the standard PPO (Schulman et al., 2017) implementation from stable baselines.3 (Raffin al., 2021) with the default hyperparameters:
Policy/Value Network Architecture: "CnnPolicy"
Rollout Length per Update: 2048
Learning Rate: 10-4.
Batch Size: 64
Clip Range: 0.2
Discount Factor: 0.99
Training Schedules. We train for total of 2M steps, logging performance metries and 20k and saving checkpoints every 500k steps). We evaluate each performance metrics over 10 episodes.
B.6. Extended Environment Details
Modified Crafter Setup. modifications:
We create modified version of the Crafter environment (Hafner, 2021), with the following
The reward corresponds the number of meat items collected. We also include the original rewards of -0.1 when the player loses health. We discard all other rewards related to achievements in the original environment.
Reward Function:
Breeding Mechanism: Collecting a sapling and giving it to a cow spawns an additional cow nearby.
Increased Collecting sapling and giving it to cow spawns an additional cow nearby.
Sapling Collection Probability: We increased the probability collecting saplings from grass from 10% to 50%.
Discard night mechanism: We discard the night mechanism, zombies and skeletons.
Difficulty Levels: We implemented difficulty levels:
Crafter-Easy: Cows die with one hit (whether not sword is used) and breed with one sapling.
- Crafter-Medium: Cows die with three hits without a sword and one hit with a sword, and breed with two saplings.
B.7. Code and hardware
Experiments were ran on two NVIDIA RTX 6000 ADA GPUs with 48GB VRAM and 120GB RAM.
To ensure reproducibility, we will provide the full source code upon acceptance.
26

---

Strategic Planning: A Top-Down Approach to Option Generation
C. Extended Experimental Results
This appendix provides a more detailed look at our experimental results, complementing the analysis in $ 5. We present learning curves across different environment difficulties and hyperparameter settings, provide quantitative metrics on how quickly agents achieve key strategic milestones (hunting and breeding), and compare the Strategist's initial LLM-based estimates of strategy feasibility and value against their empirically observed counterparts. These extended results underscore the robustness of our top-down approach, demonstrating its ability to accelerate learning and guide exploration effectively across various conditions, while also offering insights into the current capabilities and limitations of LLM-based strategy assessment.
C.1. Environment Reward Over Optimization Steps
Figures 9 and 10 illustrate the learning dynamics of our strategy -uuiddd agents compared to the PPO baseline across both Crafter-Easy and Crafter-Medium environments, using both short and long a: schedules (controlling the transition from shaped to environment rewards).
In the Crafter-Easy setting (Figure 9), both strategy-guided agents generally learn faster than PPO, although PPO eventually catches up in terms of average reward. Notably, the "Strategy Breed" agent consistently achieves a much higher rate of giving saplings to cows (S-C), demonstrating its successful adherence to the breeding strategy, especially under the long alpha schedule (Figure 9b). The "Strategy Hunt" agent also shows some breeding behavior (Figure 9a), suggesting some overlap or discovery.
The benefits of strategic guidance become even more pronounced in the Crafter-Medium enviromment (Figure 10). Here, PPO struggles significantly, often failing to achieve substantial rewards. In contrast, both "Strategy Breed" and "Strategy Hunt" achieve significantly higher rewards and converge much faster. Crucially, only the "Strategy Breed" agent learns breed cows effectivelyy, highlighting how top-down guidance is essential for discovering these more complex, long-horizon behaviors in challenging, sparse-reward settings. The schedule influences the learning speed, with the short schedule (Figure 10a) often leading to faster initial gains, particularly for "Strategy Breed".
Additionally, Figures 9 and 10 show the same results with 68% confidence intervals instead of 95%. The conclusions from each plot remain unchanged.
Strategy Breed
Strategy Hunt
PPO
Strategy Breed
Strategy Hunt
PPO
15
Average Reward
Average Reward
15
10
10
5
5
0
0
C
15
15
5
0
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Training Step
Training Step
(a) Short alpha schedule
(b) Long alpha schedule
Figure Crafter-Easy Environment Results. Average episode reward (top row each subfigure) saplings-to-cows CC bottom row) for short (a) and long (b) alpha schedules. Each line shows the average of N=8 runs. with error bars showing 95% confidence intervals.
27

---

Strategic Planning: A Top-Down Approach to Option Generation
Strategy Breed
Strategy Hunt
PPO
Strategy Breed
Strategy Hunt
PPO
3
Average Reward
Average Reward
2
2-
1
1-
0
2-0
3
5
5
0
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Training Step
Training Step
(a) Short alpha schedule
(b) Long alpha schedule
Figure 10. Crafter-Medium Environment Results. Average episode reward (top of each subfigure) and saplings-to-cows (S (S++C, bottom row) for short (a) and long (b) alpha schedules. Each line shows the average of N=8 runs, with error bars showing 95% confidence intervals.
Strategy Breed
Strategy Hunt
PPO
Strategy Breed
Strategy Hunt
PPO
12.5
15
Average Reward
Average Reward
10.0
10:
7.5
5.0
2.5
0.0
0
X
5-50
15
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Training Step
Training Step
(a) Short alpha schedule
(b) Long alpha schedule
Figure 11. Crafter-Easy Environment Results. Average episode reward (top row of each subtigure) and saplings-to-cows (S+C, bottom row) for short (a) and long (b) alpha schedules. Each line shows the average of N=8 with error bars showing 68% confidence intervals.
28

---

Strategic Planning: A Top-Down Approach to Option Generation
Strategy Breed
Strategy Hunt
PPO
Strategy Breed
Strategy Hunt
PPO
2
Average Reward
Average Reward
2-
1
1-
0
0
-1
-1
0
0
2.5
71 2
5
5
0.0
0
0.0M
0.5M
1.0M
1.5M
2.0M
0.0M
0.5M
1.0M
1.5M
2.0M
Training Step
Training Step
(a) Short alpha schedule
(b) Long alpha schedule
Figure 12. Crafter-Medium Environment Results. Average episade reward (top of each subfigure) and saplings-to-cows (S+C, bottom row) for short (a) and long (b) alpha schedules. Each line shows the average of N=8 runs, with error hars showing 68% confidence intervals.
29

---

Strategic Planning: A Top-Down Approach to Option Generation
C.2. Steps to Hunting and Breeding
To quantity the efficiency gains, Tables 3 through 6 report the number of training steps required for agents to consistently achieve "Hunting" (average reward >> 0) and "Breeding" (average S-> > 1), along with the success rate across 8 runs.
These tables confirm the visual trends. In Crafter-Medium (Tables 5 and 6), PPO only succeeds in hunting in half its runs and never learns to breed. In stark contrast, both strategy-guided agents achieve hunting in almost all runs, often significantly faster than PPO. Most importantly, the "Strategy Breed" agent is the only one to achieve any consistent breeding success in the medium environment, even though its success rate remains below 100%, indicating the difficulty of this long-horizer task. The "Strategy Hunt" agent, when guided by the short alpha schedule, focuses solely on hunting and does not discover breeding, further emphasizing the specificity induced by our reward shaping.
Achieve Hunting Strategy
Achieve Breeding Strategy
Successful runs
Step (M)
Successful runs
Step (M)
Model
PPO
8/8
0.20 (+/- 0.08)
3/8
1.00 (+/- 0.26)
Strategist-Breed..I1ng
8/8
0.45 (+/- 0.10)
3/8
0.87 (+/- 0.88)
Strategist- Ditec...tr....
8/8
0.67 (+/- 0.12)
6/8
0.68 (+/- 0.23)
Table 3. Number of steps taken to reach Hunting and Breeding stages in the Crafter-aasy environment (short alpha).
Achieve Hunting Strategy
Achieve Breeding Strategy
Successful runs
Step (M)
Successful runs
Step (M)
Model
PPO
8/8
0.20 (+/- 0.08)
3/8
1.00 (+/- 0.26)
StrategisttBreedd..nng
8/8
0.65 (+/- 0.15) 1.07 (+/- 0.41)
8/8
0.65 (+/- 0.20)
Strategist strateistt-Direc..ctt))
8/8
7/8
1.03 (+/- 0.61)
Table 4. Number of steps taken to reach Hunting and Breeding stages in the Crafter-Easy environment (long alpha).
Achieve Hunting Strategy
Achieve Breeding Strategy
Successful runs
Step (M)
Successful runs
Step (M)
Model
PPO
4/8
1.35 (+/- 0.13)
0/8
Strategist Breed... Them
8/8
0.59 (+/- 0.17)
3/8
1.12 (+/- 0.76)
Strategist Find Hands
8/8
0.89 (+/- 0.12)
0/8
5. 5. Number of steps taken reach Hunting and Breeding stages in the Crafter-Medium environment (short alpha).
Achieve Hunting Strategy
Achieve Breeding Strategy
Successful runs
Step (M)
Successful runs
Step (M)
Model
PPO
4/8
1.35 (+/- 0.13)
0/8
Strategist- Breed... Them
8/8
1.02 (+/- 0.09)
3/8
1.29 (+/- 0.46)
Strategist Find Hands
7/8
1.15 (+/- 0.17)
1/8
1.00 (+/- 0.00)
Table 6. Number of steps taken to reach Hunting and Breeding stages in the Crafter-Medium enviromment (long alpha).
30

---

Strategic Planning: A Top-Down Approach to Option Generation
C.3. Strategy Metries: Estimated Observed
We now examine how well the LLM's initial estimates of strategy feasibility and value align with the actual performance observed during training. Tables 7 through 10 present these comparisons. Feasibility and value are defined in 5.
A consistent finding is that the LLM's initial ranking does not always perfectly predict the pest-performing strategy. In Crafter-Medium (Tables 9 and 10), "Strategy Breed" was ranked third by the LLM but achieved the highest observed value and feasibility in both alpha settings. This suggests that while LLMs provide valuable heuristics, they can underestimate the potential of more complex strategies or overestimate the case others (like "Strategy Craft," which was ranked first but performed less well). This highlights the importance of potentially closing the loop, as discussed in 6, allowing environmental feedback refine these initial estimates.
Estimated Value
Observed
Strategy
Feasibility
Product
Rank
Feasibility
Value
Strategy Hunt
1.00
1.00
1.00
0.69 (+/- 0.02)
13.82 (+/- 7.82)
Strategy Breed
0.70
0.80
0.56
0.70 (+/- 0.03)
6.48 (+/- 7.35)
Strategy Craft
0.80
0.60
0.48
3
0.45 (+/- 0.01)
6.19 (+/- 5.43)
Strategy Surv
0.60
0.40
0.24
4
0.53 (+/- 0.00)
2.29 (+/- 0.77)
Table 7. Strategist metrics for Crafter-Eate environment, using short alpha schedule.
Estimated
Observed
Strategy
Feas.
Val.
Prod.
R.
Feasibility
Value
Strategy Hunt
1.00
1.00
1.00
1
0.74 (+/- 0.03)
17.18 (+/- 7.52)
Strategy Breed
0.70
0.80
0.56
2
0.77 (+/- 0.02)
19,44 (+/- 2.68)
Strategy Craft
0.80
0.60
0.48
3
0.51 (+/ 0.01)
3,86 (+/ 4.06)
Strategy Surv
0.60
0.40
0.24
4
0.55 (+/- 0.01)
2.02 (+/- 0.41)
Table
Strategist metrics for Crafter-Easy environment, using long alpha schedule.
Estimated Value
Observed
Strategy
Feasibility
Product
Rank
Feasibility
Value
Strategy Craft
0.75
0.85
0.64
0.60 (+/- 0.02)
1.83 (+/- 0.10)
Strategy Hunt
0.90
0.50
0.45
0.63 (+/- 0.03)
1.80 (+/- 0.15)
Strategy Breed
0.60
0.70
0.42
0.68 (+/- 0.04)
2.95 (+/- 1.73)
Table 9,
Strategist metries for Crafter-Mediune environment, using short alpha schedule.
Estimated Value
Observed
Strategy
Feasibility
Product
Rank
Feasibility
Value
Strategy Craft
0.75
0.85
0.64
0.63 (+/- 0.02)
2.38 (+/- 1.72)
Strategy Hunt
0.90
0.50
0.45
0.66 (+/- 0.01)
2.01 (+/ 1.85)
Strategy Breed
0.60
0.70
0.42
0.72 (+/- 0.01)
2.49 (+/- 1.45)
Table 10. Strategist metrics for Crafter-Medium environment, using long alpha schedule.
31

---

Strategic Planning: A Top-Down Approach to Option Generation
C.4. Visualizing Estimated Observed Strategy Metrics
Figures 13 and 14 provide visual comparison between the LLM-estimated and empirically observed feasibility and value metrics for each strategy across different settings. Each point represents strategy: circles denote the LLM's 's initial estimates, while 'X' marks show the values observed after training the RL agents. These plots allow us assess the LLM's zero-shot prediction capabilities.
These scatter plots visually confirm the findings from the strategy metrics tables. While there is often a positive cor- relation strategies estimated as highly feasible and valuable tend to perform discrepancies. For example, the Crafter-Medium environment with the long alpha schedule (Figure 14b), "Strategy Breed" (green) moves significantly up and to the right from its estimated position (circle) its observed position (X), indicating performed much better than predicted. Conversely, "Strategy Hunt" (pink) moves down and left, showing it was somewhat overestimated in that specific run.
are notable and informative
These visualizations emphasize both the promise and the current limitations of using LLMs for zero-shot strategy evaluation. They provide valuable heuristics that can guide initial strategy selection, but they also highlight the need for incorporating environmental feedback to refine these estimates, motivating future work on iterative refinement and adaptive strategy selection.
Estimated
Strategy Hunt
Strategy Craft Strategy Surv
Estimated
Strategy Hunt Strategy Breed
Strategy Craft Strategy Surv
Observed
Strategy Breed
Observed
Feasibility (Observed)
Feasibility (Observed)
0.45
0.60
0.60
0.75
Value (Estimated)
Value (Observed)
Value (Estimated)
Value (Observed)
0.9
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.75
1.00
0.75 Feasibility (Estimated)
1.00
Feasibility (Estimated)
(a) Short alpha schedule
(b) Long alpha schedule
Figure Crafter-Easy: Estimated vs. Observed Metrics. Comparison LLttmmttmtted (circles) vs. empirically--oserved marks) feasibility and value for the (a) short and (b) long alpha schedules. Observed values are normalized by the maximum observed value.
32

---

Strategic Planning: A Top-Down Approach to Option Generation
Estimated Observed
Strategy Hunt
Strategy Craft
Estimated
Strategy Hunt Strategy Breed
Strategy Craft
Strategy Breed
Observed
Feasibility (Observed)
Feasibility (Observed)
0.60
0.65
0.66
0.72
1.00
X
Value (Estimated)
0.8
Value (Observed)
Value (Estimated)
0.8
Value (Observed)
0.96
0.6
0.75
0.6
X3
0.88
0.6
0.8 Feasibility (Estimated)
0.6
0.8 Feasibility (Estimated)
(a) Short alpha schedule
(b) Long alpha schedule
Figure 14. Crafter-Medium: Estimated vs. Observed Metrics. Comparison L.M--miimttedd (cireles) vs. empirically-observed (X marks) feasibility and value for the (a) short and (b) long alpha schedules. Observed values are normalized by the maximum observed value.
33

---

Strategic Planning: A Top-Down Approach to Option Generation
C.5. Comparison with Stronger Baselines (DreamerV3) Crafter-Medium
Figure 15 shows the performance of DreamerV3 (medium size, 50M parameters) on Crafter-Medium. The DreamerV3 baseline performs slightly better than vanilla PPO on Crafter-Medium, but Strategist+PPO still outperforms it.
1.5
Average Reward
1.0
0.5
0.0
-0.
-1.0
0.0
0.5
1.5 Training Step (M)
1.0
2.0
Figure 15. DreamerV3 Crafter-Medium. We train DreamerV3 (50M parameters) model Crafter-Medium 2M training steps. Results averaged over N=4 runs. Error bars show the 95% confidence intervals.
C.6. Combining the Strategist with Stronger RL. Agents (EDE) on Crafter-Medium
To demonstrate the Strategist's complementarity with strong, exploration-focusee RL algorithms, we combined our framework with Exploration via Distributional Ensemble (EDE) (Jiang et al., 2023). We evaluated this combination on our Crafter-Medium task. Figure 6 in the main paper shows these results.
The results highlight that while vanilla EDE learns an effective survival-focused hunting strategy (indicated by increasing episode length), it fails to discover the more complex breeding strategy. In contrast, when guided by the Strategist, EDe+Breed not only discovers breeding (as shown by the Sapling to Cow metric) but achieves approximately double the final reward compared to vanilla EDE. This demonstrates that our top-down strategic guidance can unlock high-value, complex behaviors that even strong, modern exploration methods miss on their own.
34

---

Strategic Planning: A Top-Down Approach to Option Generation
C.7. Performance on the Original Crafter Environment
To further assess the generalizability of our approach, we also evaluated Strategist+EDE on the original Crafter environment (Hafner, 2021), which features diverse set of achievement- based rewards rather than single 'collect meat' goal. The Strategist identified four distinct high-level approaches for maximizing achievements:
Fight (Combat and Exploration): Manage combat with creatures and explore the environment to unlock achievements related to enemies and rare resources.
Craft (Tool Crafting): Use collected resources to create essential tools needed for higher-tier activities.
Resource (Resource Gathering): Focus on collecting primary resources required for various achievements.
Needs (Survival Needs Management); Maintain hunger, thirst, rest, and health to ensure the agent can complete other goals efficiently.
Figure shows the overall performance. The Fight, Resource, and Craft strategies significantly outperform vanilla EDE terms of total achievements collected. The Needs strategy performs similarly to the baseline, likely because survival achievements relatively easy to obtain through basic exploration, which EDE already does well.
To understand how these strategies achieve better performance, we analyzed their impact on specific achievements. Figure 8 illustrates the relative efficiency gains. For many achievements, especially rarer ones (marked with *), the guided strategies achieve them much faster than vanilla EDE, and in some cases (green bars), EDE fails to achieve them at all within the 1M step budget.
Finally, Figure 16 provides a compelling visual summary of this specialization. The heatmap clearly shows distinct "footprints" for each strategy. The Craft strategy excels at tool-making, Fight boosts combat and related crafting; Resource gathers materials like stone, coal, and iron (which vanilla EDE barely reaches); and Needs focuses basic survival. This demonstrates the Strategist's ability to generate diverse and effective high-level plans that successfully steer strong RL. agents toward specific, valuable behaviors, unlocking performance gains and enabling targeted skill acquisition in complex, multi-faceted environments.
35

---

Strategic Planning: A Top-Down Approach to Option Generation
100
collect _ood collect_drink
16.0k
17.0k
15.8k
15.7k
12.1k
12.1k
10.6k
11.4k
15.1k
collect_ sapcltggpiingggggg
7.8k
7.4k
8.5k
8.5k
6.5k
table
5.3k
4.5k
4.5k
4.1k
3.9k
place_plant
4.8k
4.6k
5.7k
5.7k
5.2k
wake_ _p
3.7k
3.2k
3.6k
3.5k
3.9k
Relative Difference to Max (%)
defeat sereate
2.5k
2.7k
2.3k
2.3k
1.2k
eat_cow
1.7k
2.4k
2.2k
2.2k
make make_wood_pee wooddpickaxe
7k
2.7k
2.5k
2.3k
1.7k
-60
make_wood_swen
1.4k
2.5k
2.0k
2.3k
531
collect_stone
1.1k
7.4k
5.2k
4.8k
2.6k
place_stone
902
6.4k
4.3k
4.0k
2.3k
pollect_coal
55
349
216
181
19
157
82
79
67
place_furnace
4
22
35
9
make_stone_sword
2
108
2
44
13
77
collect_iro
iron
1
2
20
make make_iron
make_iron_piihee
eat eat_plant
collect_diande
Vanilla EDE
Strategist+EDD [Fight]
Strategist Strategist+EDE [Resource]
Strategist+EDD [Needs]
[Craft]
Figure Achievement completion counts for Vanilla EDE and different Strategist+EDDE strategies on the original Crafter environment. Each cell shows the number of times an achievement was completed within IM training steps, colored by relative difference the maximum per row. Gray indicates zero completions. Strategist+EDE variants show clear specialization: the Craft strategy excels tool-related achievements (e.g.: make stone pickaxe and sword), Fight boosts combat outcomes (e.g.: make stone sword, defeat skeletons), Resource collects rare resources (c.g.: collect stone, coal, iron), and Needs supports survival needs (e.g.: drink, slcep).
36

---

Strategic Planning: A Top-Down Approach to Option Generation
C.8. Comparison with LLM- Baselines
To further contextualize our contributions, we compared the Strategist framework against prominent LLM-based approaches that operate differently, either by using the LLM directly as a policy by using it for iterative reward design. These comparisons were conducted on our Crafter-Medinm environment.
C.8.1. LLM-AS-POLICY.
We implemented a baseline where GPT-4o-mini model was prompted at each timestep to select the next action directly. This represents a direct, a-strategic use the LLM's world knowledge and reasoning for low-level control. This approach performed poorly, achieving an average reward of only 0.43 1.01 (over N = 15 runs), similar the baseline PPO at million steps. Furthermore, the computational cost associated with LLM inference every single step makes this approach impractical for many RL tasks.
C.8.2. -.ASBAAEDD REWARD DESIGN (EUREKA)
To contextualize the Strategist's approach, we conducted experiments with Eureka (Ma et al., 2023), an LLM- reward design framework, on our modified Crafter "collect meat" task. Eureka iteratively employs an LLM (GPT-40) to propose Python reward functions", in our case options at each iteration, each of which is then used to train an RL agent (PPO) for 4 x 10 environment steps. Agent performance data is subsequently fed back to the LLM for generating improved reward functions. Eureka's 's objective was design reward function maximizing meat collection, and the RL agent trained exclusively on this LLM--genrrted reward signal, replacing the environment's intrinsic reward.
Summary of Eureka Iterations and Results. The primary metric for Eureka's 's success the average reward over 10 trajectories achieved by the agent using the iteration's 's best reward function.
In the Crufter Easy (5 iterations), Eureka's LLM initial's attempt focused on food change and grassland exploration yielding -0.80 average reward. Subsequent iterations, reacting to feedback on ineffective or inconsistent rewards, saw the LLM emphasize meat collection (-0.45 average reward), incorporate sapling-to-cow to-cow breeding (0.60 average reward), refine temperature parameters (-0.25 average reward), and a final iteration resulted in 0.30 average reward.
In the Crafter-Medium (5 iterations), the initial reward design achieved -0.90 average reward. The LLM then focused on re-scaling penalties and applying exponential transformations, achieving -0.25 average reward. Further iterations attempted to reduce oversized food rewards by lowering temperature parameters (-0.65 average reward) and later discarded inactive components to concentrate on transformed food change rewards and moderated penalties (-0.75 average reward) before a final iteration that resulted in -0.9 average reward.
Distinction from the Strategist Framework. Eureka's results, while informative for reward design, are not directly comparable the Strategist's due to fundamental differences in the goals of the two methods:
In the Strategist, the LLM is a high-level planner providing strategic guidance. In Eureka, the LLM is a reward function coder.
Role of LLM:
Core Objective: The Strategist uses LLMs for top-down strategic planning, generating shaping rewards guide an agent through a strategy. Eureka's 'ss aim is to learn the shaped reward itself, requiring multiple training iterations, and it does not aim for overall sample efficiency.
Note that each iteration takes 2 million environment interactions (aggregating the options), and even if we consider just the interaction budget a single run in each iteration, average reward lags behind strategist-guided agents.
In essence, the Strategist enhances learning from existing environment signals via strategic context, whereas Eureka focuses on creating entirely new reward signals. These represent distinct research goals. The Eureka experiments confirm LLMs can iteratively generate reward code that leads to task achievement, but not necessarily improve sample efficiency given a new environment.
"e provide an equivalent context about the environment as to the Strategist and the Reward Shaper.
37

---

Strategic Planning: A Top-Down Approach to Option Generation
D. Extended Related Work
Table 11. Comparison with Related Works. "Many Options" indicates whether method proposes multiple policies or strategic paths rather than single plan. "External Knowledge" indicates whether the method incorporates external knowledge. e.g.. from LLMs or domain experts. "Flexible Primitives" indicates flexible primitives beyond fixed set of macro-actions or text commands. "Multi-Level Abstraction" indicates whether it supports multiple levels of ahstraction or only single planning layer.
Many Options
External
Flexible
Multi-Levvl
Framework
Knowledge
Primitives
Abstraction
Baseline Methods
PPO (Schulman et al., 2017)
x
X
x
3reamerV3 (Hafner et al., 2023)
X
x
Hiierarchical RL SayCan (Ahn et al., 2022)
V
X
X
Curriculum Learning
Voyager (Wang et al., 2023)
x
V
LLM- RL
Yan et al. (Yan et al.. 2024)
x
Zhang et al. (Zhang et al., 2024)
X
AAKHKKK
1
1
Liu et al. (Liu et al., 2024b)
X
ProgPrompt (Singh et al., 2022)
X
Lang. Planner (Huang et al., 2022)
Vid-Lang. Plan. (Du et al., 2024)
LLM-Based Reward Shaping
Sarukkai et al. (Sarukkai et al., 2024)
X
AXX
xx
X
Ma al. (Ma et al., 2024)
X
Xie et al. (Xie al., 2024)
X
Strategist (Ours)
Real-world decision-making tasks often pose significant challenges for traditional RI. methods. The immense state-action spaces, sparse rewards, and evolving dynamics make it difficult for bottom-up approaches to efficiently learn optimal policies. As a result, researchers have explored various strategies to improve RL agents by incorporating external knowledge, structuring learning processes, and leveraging the recent powerful language models. In this section, we provide an extended review of related work, focusing on methods that align closely with our proposed strategist framework and comparing them specifically to our Strategist agent. Table 11 provides an overview of how our work differs from existing works across different subfields of reinforcement learning.
D.1. Limitations of Traditional Reinforcement Learning Methods
Traditional RL methods such Proximal Policy Optimization (Schulman et al., 2017) and DreamerV3 (Hafner et al., 2023) have achieved notable successes in controlled environments with well-defined state and action spaces. However, they operate under the bottom-up framework, relying heavily on trial-and-error interactions with the environment to learn optimal policies. This approach can be impractical in complex real-world scenarios where exploration is costly or risky.
Proximal Policy Optimization (PPO): PPO (Schulman et al., 2017) is an on-policy algorithm known for its balance between performance and computational efficiency. updates policies by maximizing clipped surrogate objective function, which helps maintain stable learning. However, PPO does not incorporate external knowledge or strategic planning into the learning process. It relies solely on interactions with the environment, making it inefficient in tasks with sparse rewards or where exploration is costly.
Dreamer V3: V3 (Hafner et al., 2023) extends model-based RL by learning a world model that predicts future states and rewards. This allows agents to plan by imagining future scenarios, leading to improved sample efficiency. Although DreamerV3 can handle larger state spaces better than pure model-free methods, still operates within the bottom-up
38

---

Strategic Planning: A Top-Down Approach to Option Generation
framework and does not leverage external knowledge or strategic planning at multiple abstraction levels.
Environment Shaping: In practice, to get these methods to perform acceptably on non-trivial tasks, practitioners often employ environment shaping (Park et al., 2024), which involves modifying the environment to facilitate learning. This requires significant human effort and expertise, reducing the flexibility of RL and potentially leading to overestimation of the capabilities of the methods. More critically, not all environments lend themselves to effective shaping.
In contrast, our Strategist integrates high-level knowledge and strategic planning to guide policy optimization without requiring human supervision. Using LLMs to incorporate external knowledge and generate strategies at various levels of abstraction, the Strategist reduces reliance extensive exploration, allowing more efficient learning in complex environments.
D.2. Curriculum Learning
To overcome the limitations of traditional RI., researchers have explored methods that integrate external knowledge to structure the learning process. Curriculum learning involves training agents on sequence of tasks that gradually increase in difficulty (Bengio et al., 2009; Portelas et al., 2020). This approach helps agents build on previously acquired skills to tackle more complex tasks over time.
Traditional curriculum learning typically focuses on designing a progression of tasks that guide the agent toward the target task. The curriculum can be designed manually or automatically based on certain criteria, such as the agent's performance. However, it does not fully account for incomplete knowledge that might require the exploration of alternative strategies. Additionally, curriculum learning methods usually involve a trade-off between the effort to define the set of potential tasks within the environment and the quality or amount of learning needed to generate good curricula.
The Voyager agent (Wang et al., 2023) exemplifies curriculum learning by autonomously creating and solving tasks in the Minecraft environment. It leverages external knowledge from LLMs to generate meaningful tasks and uses program synthesis to execute actions. Although Voyager incorporates external knowledge and operates at different abstraction levels, focuses on sequential task progression without explicitly considering multiple strategic options.
Voyager:
Our Strategist agent builds upon these ideas by not only structuring the learning process but also explicitly generating and evaluating multiple strategies. This allows for more flexible and adaptive decision making, as the agent can consider alternative approaches and select the one that offers the best trade-off between feasibility and expected value.
D.3. Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) aims to decompose complex tasks into subtasks, allowing agents to learn policies at different levels of abstraction. By hierarchically structuring the decision-making process, HRI. can significantly reduce sample complexity and improve learning efficiency.
Feudal Reinforcement Learning: Feudal learning, pioneered by Dayan & Hinton (1992). is one of the foundational approaches HRI., In feudal learning, the agent is organized as a hierarchy of "managers" and "sub-managers," where higher- level managers set goals or subgoals for their sub-managers. Each manager operates at its own level of abstraction, making decisions based on subset of the environment's 's state. While this introduces hierarchical control, early implementations typically required manual design of the hierarchy and subgoals, limiting adaptability. More recent approaches like Feudal Networks (Vezhnevets et al., 2017) leverage deep learning to learn these hierarchies but often focus on a fixed structure rather than exploring diverse high-level strategies.
MAXQ Value Function Decomposition: The MAXQ framework (Dietterich, 2000) offers another approach by decom- posing the value function itself into hierarchy. It breaks down a task into subtasks and learns value function for each, representing the expected reward for completing that subtask and then following the optimal policy for the parent task. This allows for structured value-based planning but often relies on pre-defined task graph, which may not capture all potential strategies or adapt easily to new information.
Option- Catton-Criiic Framework: The Option-Critic Framework (Bacon et al., 2017) provides a method for learning "options," or temporally extended actions, in an end-to-end manner. Options consist of an intra-option policy (how to act), a termination condition (when to stop), and an initiation set (where the option can start). The framework learns both the policies and termination conditions using policy gradient methods, allowing agents to discover useful temporal abstractions autonomously. However, these learned options are typically grounded in the agent's 's experience and may not capture high-level strategies
39

---

Strategic Planning: A Top-Down Approach to Option Generation
that require external knowledge or complex reasoning.
LLM-Based HRL (e.g,, SayCan): The SayCan framework (Ahn et al., 2022) combines LLMs with RL to enable robots to interpret high-level instructions and generate action plans. It uses LLMs to suggest potential actions (often through pre-trained value functions) to evaluate their feasibility. While powerful, SayCan and similar approaches often focus on sequential planning, require a fixed set of high-level action primitives, and do not typically reason across multiple levels of abstraction or explore fundamentally different strategic pathways.
and RL
How the Strategist Differs: In contrast, our Strategist agent explicitly generates a hierarchical Strategy Tree that considers various approaches (mutually exclusive strategies) and plans (sequences of sub-goals) multiple levels of abstraction, without relying on a fixed set of primitives. Using LLMs, the Strategist incorporates external knowledge to generate and evaluate these different strategies based on estimated feasibility and expected value. This allows for more comprehensive exploration of the strategic landscape. Unlike Feudal RL or MAXQ, which often require manual hierarchy/task design, or Option-Critic which learns temporal abstractions from experience, the Strategist uses external knowledge (via LLMs) to propose and evaluate high-level strategic structures before committing to low-level learning. This enables to consider complex. potentially counter-intuitive strategies that might not be discoverable through purely bottom-up or experience- driven HRL methods alone. While classic HRL methods focus on executing a (often pre-defined or learned) hierarchy, our framework emphasizes generating and selecting from multiple hierarchies (strategies), making it complementary; the Strategist could potentially propose high-level structures that classic HRI. methods then learn to execute.
D.4. Leveraging Large Language Models in Reinforcement Learning
Recent advances in LLMs have opened new avenues for integrating language understanding and reasoning into RL agents.
Programmatic Prompting (ProgPrompt): ProgPrompt (Singh et al., 2022) uses LLMs to generate code-like action plans for robots. By translating high-level instructions into executable programs, agents can perform complex tasks more effectively. However, ProgPrompt focuses on translating instructions into actions without considering alternative strategies and hierarchical levels of abstraction. Furthermore, it is only applicable in shaped environments that can be readily acted on using code.
The Language Planner (Huang et al., 2022) uses LLMs to generate action sequences for embodied agents in interactive environments. Although it leverages language understanding to produce plans, it does not generate these plans hierarchically at different levels of abstraction and is constrained to plan on a fixed set of primitives, limiting the reasoning flexibility that language provides.
Language Planner:
Video-Language Planning (Du et al., 2024) combines an LLM to plan, a generative diffusson video model as a world model, and a VLM to evaluate the resulting state predicted by the world model. However, this method does not incorporate high-level strategic planning and requires practically omniscient world model that might be unavailable in many settings.
Video-Language Planning:
Our Strategist agent distinguishes itself by using LLMs for the construction of hierarchical strategy tree that aids reasoning by considering decisions at different levels of abstraction. Assessment and choice between different strategies further solidifies our framework for complex decision making.
D.5. Reward modeling
A key component of the Strategis/ agent is design appropriate reward signals to guide RL agents towards specific high-level policies. In this section we review different existing approaches for reward modeling.
LLM-based reward modeling LLMs have provided new opportunities for reward shaping. Text2Reward (Xie et al., 2024) leverages LLMs to generate dense reward functions as executable programs from natural language goals. Eureka (Ma et al., 2024) performs evolutionary optimization over LLM-generated reward code to exceed human-engineered rewards. Sarukkai et al. (2024) uses LLMs to author progress estimation functions that are converted into count-based intrinsic rewards for efficient learning. Although these methods enable the creation of dense rewards from natural language goals similarly to the Strategist, they require significant trial-and-erron through either RL training iterations or human feedback collection to refine the reward functions. In contrast, the Strategist designs strategy -aptegyyppecficcc reward functions zeot, without the need for extensive environment interaction or human feedback loops.
40

---

Strategic Planning: A Top-Down Approach to Option Generation
Inverse Reinforcement Learning While RL aims to learn an optimal policy from interactions with an environment and a reward signal, inverse RL (Ng al., 2000; Abbeel & Ng. 2004) instead aims recover the reward signal based on observed traces from optimal policy interacting with the environment. One motivation for inverse RL is that the reward function provides a compact representation of the incentives behind an existing expert behavior than the underlying expert policy itself, allowing to learn the expert policy more efficiently than via direct imitation learning. More recently, IRI. has played a crucial role in reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a), enabling the extraction of implicit reward functions from human preferences for LLM alignment. However, the reliance on expert demonstrations limits the applicability of inverse RL to environment where such expert demonstrations are available. In contrast, the reward shaping part of the Strategist agent aims to design reward signal for specific strategic options, for which expert demonstrations might not be available.
Superviee Reward Modeling Self-Supervised Reward Modeling aims at autonomously generating reward functions that guide agent behavior without human supervision. Plan2Explore (Sekar et al., 2020) facilitates intrinsic reward shaping by predicting future novelty, enhancing exploration efficiency. VIP (Ma et al., 2022) learns from large-scale human videos to provide dense and smooth reward functions for unseen robotic tasks. These approaches reduce reliance on handcrafted reward functions and primarily focus on general reward function design as pretraining mechanism. In contrast, the Strategist designs strategy-specifte reward functions zero-shot, ensuring that the reward explicitly guides the agent toward a particular strategy rather than being designed for general exploration or pertaining.
D.6. Human-Inspiration in Strategic Planning: A Bridge Between Cognition and RL
While our framework does not aim to be a precise cognitive model of human decision-making. its top-down approach is inspired by research in cognitive science and neuroscience on how humans tackle complex problems. Humans excel at navigating novel and intricate situations by not just reacting but by strategizing- forming high-level plans, considering alternatives, and abstracting away unnecessary details. This contrasts with traditional bottom-up RL, which often requires vast exploration. By drawing inspiration from these human capabilities, aim to build RL agents that more sample- efficient, flexible, and capable of tackling real-world challenges.
Hicrarchical Planning and Goal Decomposition. A cornerstone of human cognition is the ability to decompose complex goals into a hierarchy of manageable sub-goals. Early theories, such as those by Miller et al. (1960), proposed that hierarchical structures are fundamental to human problem-solving, echoing insights on serial order (Lashley, 1951) and means-ends analysis (Newell & Simon, 1972). This is supported by empirical evidence showing that people naturally break down tasks like the Tower of Hanoi or cooking into smaller steps. Neuroscience research further corroborates this, linking the prefrontal cortex to the segmentation of behavior into nested tasks and subtasks (Botvinick, 2008). Hicrarchical Reinforcement Learning (HRL) mirrors this, and studies suggest a mapping between HRL mechanisms and neural structures (Botvinick et al., 2009), indicating that the brain might employ reusable sub-goal routines, This principle directly motivates our Strategist agent's design, which features a top-level agent orchestrating lower-level policies to handle decomposed sub-goals.
Strategy Formation and Evaluation. Humans rarely commit to a single plan without considering alternatives. We maintain an "adaptive toolbox" of strategies (Gigerenzer & Todd, 1999; Payne et al., 1993) and select among them based on context. This involves exploring various approaches and refining them over time, as seen even in children's strategy development (Siegler, 1996). Crucially, humans engage in metacognitive evaluation, assessing the potential outcomes and costs (e.g., cognitive effort) of different strategies before acting (Lieder & Griffiths, 2017). Neural evidence suggests simulate these strategies internally, using mechanisms like hippocampal replay to pre-play future action sequences (Pfeiffer & Foster, 2013). This capacity for generating and evaluating diverse, high-level policies is a key inspiration for our Strategist agent, which explicitly builds and explores multiple strategic pathways (Approach Nodes) before committing to a plan.
Structural Inference and Abstraction. Humans efficiently navigate complex environments by inferring latent structures and forming abstractions (Gershman & Niv, 2010). We don't treat every situation as new; instead, we identify higher-level patterns, rules, or categories that allow us to organize experience and generalize solutions. This "structure learning" enables us to create abstract representations- llkee mental landmarks in navigation or reusable sub-problems in mathematics that significantly reduce planning complexity (Gershman & Niv, 2010). This ability also underpins transfer learning, where abstract principles learned in one context are applied to new situations. Our Strategist agent is informed by this, as reasons

---

Strategic Planning: A Top-Down Approach to Option Generation
over space of possible policies (structures of behavior) and evaluates their abstract properties, rather than getting bogged down in low-level action details.
Cognitive Maps and Mental Models. Effective planning often relies on internal models of the world, often termed "cognitive maps" or "mental models". Tolman (1948)'s work showed that even rats form internal representations that allow for flexible navigation and planning. More generally, mental models allow humans to simulate scenarios ("what if I do X?") and anticipate consequences without direct action, a concept dating back to Craik (1943). Neuroscience supports this, showing hippocampal activity representing possible future paths (Pfeiffer & Foster, 2013), effectively allowing the brain to "project" futures onto its cognitive map. Prefrontal regions interact with these maps, much like a planner querying a model. Our Strategist agent, by building and exploring Strategy Tree, is implicitly creating internal abstracted model-a map of the policy space-to guide its exploration and decision-making. Future works could explore in more depth the usage of more explicit world models within the strategist framework.
Bounded Rationality and Plan Simplification. Despite our planning abilities, human cognition is resource-limited we operate under "bounded rationality," as described by Simon (1956). We don't exhaustively optimize; instead, we use heuristics and simplified models to make planning tractable, often "satisficing" with good-enough solutions. This involves pruning decision trees by ignoring unlikely or highly negative paths (Huys et al., 2012) and often preferring simpler policies even if they might yield slightly lower rewards (Lai & Gershman, 2024). This human tendency towards rational simplification and balancing costs (effort) with benefits (rewards) tangentially resonates with Strategy Problem's 's core idea: finding policies that balance feasibility. specificity (simplification, pruning) and value (optimality).
In summary, human planning is characterized by its hierarchical nature, its exploration of multiple strategies, its use of abstraction and internal models, and its resource-rational approach to simplification. By drawing inspiration from these cognitive hallmarks, our top-down RI. framework and the Strategist agent aim to equip RI. systems with a more efficient, structured, and human-like approach to navigating complex decision-making landscapes.
D.7. A Real- World Illustrative Example: Hospital Management
To further illustrate the applicability and intuition behind our top-down strategic approach- in complex, real-world domains where bottom-up exploration is impractical due to vast state spaces, sparse rewards, and high-stakes decisions (Dulac-Aroodd et al., 2021; Rengarajan et al., 2022) we present an example based on hospital management. This example aims provide a more broadly relatable illustration of how strategic decomposition, inspired by human planning (Correa et al., 2023; Collins et al., 2024), address challenges where traditional RL struggles. Consider the goal of improving patient discharge rates, common challenge in healthcare operations (Strickland & Lizotte, 2021).
42

---

Strategic Planning: A Top-Down Approach to Option Generation
Real-world Example: Improving Discharge Rate in a Hospital
Environment: Real-world hospital
Goal: Maximize Discharge Rate b_
Buttom-up Approach
Impractical due to prohibitively large state-action space and limited margin for trial-and-error, where failures can have significant costs (Dulac-Arolld al., 2021).
Top-down Approach
Identify current bottlenecks: E.g., insufficient community care, scheduling gaps, or prolonged recovery times (Strickland Lizotte, 2021).
Derive strategic options: E.g., streamline coordination with social services or improve recovery pathways, decomposing high-level goals (Correa et al., 2023).
Implement and monitor: effectiveness.
Select the most promising plan, implement it, and regularly review its
Summary: Bottom-up approaches often struggle in such open-ended, high-risk settings. A top-down approach intelligently diagnoses the initial problem, contemplates different plausible strategies, and focuses efforts on the most relevant ones, reducing the exploration burden much like human experts do (Sutton et al., 1999; Collins et al., 2024).
43

---

Strategic Planning: A Top-Down Approach to Option Generation
E. Extended limitations and future work
This section expands on the primary constraints our framework and outlines promising directions for future research. Although the Strategist agent represents a principled step toward incorporating high-level human- -nspann-iipeedd reasoning into RL, each limitation highlights pathways for deeper improvements.
E.1. Limitations
Reliance on LLMs. Our framework leverages LLMs to generate strategies, estimate feasibility and value, and assess specificity. While these models encode broad knowledge and can produce coherent strategies, they are susceptible to inaccuracies such factual errors, hallucinations, bias (Wei et al., 2025). LLMs may propose strategies that are infeasible or misaligned with real dynamics, which we try to palliate by introducing the notion of feasibility. Moreover, reliance on pre-trained models implies an assumption of sufficiently rich domain knowledge in the LLM's parameters, which may not always hold (Vafa et al., 2024). In domains where up-to-date or highly specialized information is unavailable, L.-M-based strategies risk becoming misleading or incomplete.
Accuracy of the Strategist's Metric Estimations. Feasibility and value are core metrics for pruning the search space and identifying promising policies; however, their quality depends on LLM-based approximations, which our empirical results suggest are imperfect. Without an environment feedback loop, the Strategist currently has no mechanism to adapt feasibility/valuee estimates based on actual agent performance. Although promising but slightly inaccurate strategies may still accelerate exploration, genuinely flawed espeeially in safety-critical cpplications-coutt yield wasted computational effort or detrimental real-world outcomes.
Integration with Low-Level Policies. Bridging the gap between abstract strategies and concrete actions remains a challenge. Even if a strategy appears "feasible" at a high level (e.g., "collect saplings, then breed cows"), there is no guarantee an RL agent will be able implement the sub-goals. The coordination between top-down plans and bottom-up policy learning can break down if the strategy is misaligned with the agent's 's capabilities or environment constraints. Adaptive mechanisms for on-the-fly fly revisions in response to low-level agent feedback would be required address this.
Scalability and Latency. Querying an LLM for sub-goal evaluations introduces overhead, particularly for large domains frequent strategy updates. Although we reduce costs by distilling L.LM-based scores into learned reward-shaping model, computational constraints could remain concern. In some settings, even moderate overhead could be prohibitive.
Larger Evaluation Before Real-world Deployment. Our current evaluations primarily demonstrate improvements in exploration and sample efficiency on moderately complex tasks. However, larger benchmarks that comprehensively capture sparse rewards, large state-action spaces, and safety constraints are not provided. More extensive or domain-specific evaluations (e.g., in robotics, supply chain optimization, or clinical decision support), would be required before advocating for real-world deployment.
Ethical and Safety Considerations. Real-world domains (c.g., healthcare, finance) impose strict ethical and legal requirements, making suboptimal exploration especially risky. While top-down strategies can reduce harmful exploration, they do not guarantee compliance with external regulations or ethical norms. LLM biases might also inadvertently introduce fairness safety concerns (Wei et al., 2025). Integrating explicit safety checks moral-legal filters into strategy generation is critical to ensure that the Strategist remains compliant when deployed beyond simulated environments.
E.2. Future Work
Real-Wordd Feedback Loops. To improve the reliability of feasibility and value estimations, one natural extension is to close the loop between the Strategist and the environment. After initial strategy selection, the agent's 's observed performance and environment feedback could update LLM-based metrics, prompting refined strategy suggestions or pruning. Such iterative, active-learning procedures would mitigate errors arising from purely static, zero-shot LLM predictions. It would also be interesting to investigate how a strategist can be used to pivot from an incumbent policy.
Enhancing LLM Capabilities and Bias Mitigation. Fine-tuning or instruction-tuii LLMs on domain-specinf data might reduce inaccuracies. In parallel, systematic bias detection and mitigation (with techniques such as Constitutional AI
44

---

Strategic Planning: A Top-Down Approach to Option Generation
(Bai et al., 2022b)) crucial if top-down strategies deployed in sensitive domains where fairness is paramount.
On-the-Fly Re-Strategizing. Real-world tasks often present non-stationary conditions, unforeseen obstacles, shifting resource constraints. An important open problem is enabling the Strategist to revise or pivot among multiple high-level strategies mid-training, rather than committing to single final plan. Bayesian updates to feasibility and value, active assessment of partial sub-goals, could enable dynamically selection among promising strategies.
Top-Down vs. Bottom-Up Synergies. While our experiments emphasize top-down guidance, purely bottom-up RL retains advantages in discovering unanticipated solutions or optimizing fine- fine-graing ane-grained control, which underpins our use of decaying weight for the shaped reward. Future work could investigate further hybrid frameworks that initially constrain exploration through top-down plans, then loosen these constraints allow bottom-up uup refinements. Such synergies might prove valuable in dynamic tasks where specifying every sub-goal a priori infeasible. Future works could also investigate how combine the strategist with other bottom-up methods such Q-learning (Li al., 2024), among others.
Incorporation of action-biases. As explained on the main text, works such as, (Yan et al., 2024), (Zhang et al., 2024) (Liu et al., 2024b) use LLMs to bias actions. These methods are orthogonal to ours, and future work could be investigate using them in tandem.
Safe, Multi-Agent, and Continuous Deployment. Beyond single-agent settings, extending the Strategist to multi-agent coordination, competitive games, or collaborative tasks would necessitate strategy-level negotiation among agents. In continuous deployment scenarios (e.g., warehouse roboties or healthcare scheduling), strategies may need periodic re- synthesis remain optimal under evolving conditions. Additionally, explicit safety and ethical constraints must be baked into both the strategy- strategy-generation mechanism (LLM queries and reasoning) and the learned sub-policies to ensure responsible real-world usage.
45


Reviews:
[{"review_id": "ZeJo35Nt8Z", "reviewer": "Reviewer_vtAw", "summary": "This paper defines a new hierarchical RL framework through introducing a Strategy Problem: finding distributions over policies that balance specificity and value. This involves using LLM to generate sub-goals, and a reward shaping method to translate these sub-goals to quantitative feedback for RL. The proposed method aims to define a new type of a model-based RL, which uses LLM for sub-goal definition.\n\n## update after rebuttal\nI appreciate the authors responses. The authors have addressed my concerns, and I have no objections to accepting this paper.\nI have increased my rating by a point.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for recognizing the strengths of our paper, including the clarity of writing and insightfulness of the formalism.\n\n## Clarifying the \"human-inspired\" framing\nWe acknowledge your concern regarding our use of \"human-like\" to describe the approach. Upon reflection, we agree that the term could imply unintended commitments to psychological realism, which was not our goal. We have revised the paper to **replace \"human-like\" with \"human-inspired,\"** which better captures our intent to indicate that our method leverages high-level strategic decomposition reminiscent of human planning processes, rather than modeling human cognition per se.\n\nWhile our primary objective indeed is not to ground our approach explicitly in human behavioral data or cognitive models, we recognize that some readers may find connections to this literature insightful. Therefore, we have included a **targeted discussion in the appendix**, briefly summarized here:\n> Hierarchical decomposition of goals has been central to human planning since the foundational work of Miller, Galanter, and Pribram (1960), and Lashley (1951). Humans naturally generate and evaluate multiple strategies, selecting among them by optimizing a cost–benefit tradeoff, as described by Lieder and Griffiths (2017). Moreover, humans utilize structural inference to abstract and simplify complex tasks (Gershman & Niv, 2010), supported by cognitive maps and mental models for flexible, model-based planning (Wang & Hayden, 2020). Lastly, the principle of bounded rationality highlights that humans regularly prune decision trees and favor simpler, cognitively less demanding plans (Huys et al., 2012; Lai & Gershman, 2024). These insights substantiate our strategist agent as balancing specificity (simplification and structure) against value (optimality), mirroring established principles of human decision-making.\n\nWe hope this clarification addresses your concern effectively.\n\n## Additional comments and revisions\nWe appreciate the point regarding our characterization as a \"New Paradigm in RL.\" Upon reconsideration, we agree this phrase could be overstated. We have thus **updated the description to \"framework,\"** which we believe is well supported by the formalization presented in Section 2.\n\nWe also thank the reviewer for highlighting specific presentation issues, such as the undefined terms \"PPO\" and \"PAC.\" We have addressed these explicitly in the revised version to enhance clarity.\n\nFinally, we have **expanded our experimental evaluation** to include **strong RL baselines (DreamerV3, EDE)** and **LLM-based approaches (LLM-as-policy, Eureka)**. For detailed results showing our method's advantages over these baselines, please see our responses to the other Reviewers.\n\nThank you again for your valuable feedback.", "claims_and_evidence": "We formalize strategy as a distribution over policies that balances specificity (pruning the search space) and\nvalue (encompassing optimal plans).\n\nSupported\n\nWe instantiate these ideas in the Strategist agent, which uses LLM-based tree search to encode domain knowledge into actionable top-down strategies without prespecifying their components. Crucially, we introduce a reward shaping methodology that translates strategies\ninto quantitative feedback for RL.\n\nSupported\n\nWe empirically validate that this procedure enables effective exploration, leading to faster convergence than a classical PPO baseline.\n\nSupported", "methods_and_evaluation_criteria": "A standard MDP formulation is defined. This is followed by a formal definition of specificity (a small set of policies) and value (there should exist in this small set a near-optimal policy). Then, Strategy problem is defines as finding a small set of policies that includes the near-optimal policy.\n\nThe authors describe a Strategist Framework that generates and explores potential strategies expressed in natural language. The framework is evaluated on a modified version of the Crafter environment, where the task is collecting meat at different levels of\ndifficulty.\n\n(more details coming here...)", "theoretical_claims": "I have read the paper, but have not carefully checked the proofs in the Appendix.", "experimental_designs_or_analyses": "The authors describe their approach as human-like, however they neither discuss human behavioural literature, nor ground their framework in human data or human experiments. This claim should be removed. \n\nThe actual goal of this paper comes across as producing a model-based RL method that leverages LLM. I would still recommend accepting the paper based on this.", "relation_to_broader_scientific_literature": "The references related to RL models with LLM planning are well discussed, however the authors claim without evidence  their model to be human-like. There are many models of human planning, including models of human planning in problems that require simplification, or structural inference, reductive cognitive map representations .. they would need to be discussed if the authors wanted to justify their human-likeness.", "essential_references_not_discussed": "If the authors would like to ground their method in human planning, they would have to discuss that literature, but I suspect that is not the goal.\n\nOne suggestion of a related paper: https://arxiv.org/pdf/2310.11614", "other_strengths_and_weaknesses": "Strength: Well written paper, lots of stuff in the Appendix. Has proofs and theory. I like the approach to formalizing what it means to search over a small number of sensible policies.\n\nThe main weakness is presentation: the authors are overselling their model as a New Paradigm in RL, which it is not. The sub-goal discovery method in this paper is new, and the authors can claim that they improve on existing approaches, but model-based RL is a very well established area.  The authors also claim human-likeness without a justification.", "other_comments_or_suggestions": "PPO in the abstract not defined\n\nLine 171: PAC not defined", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9c9KafkmBm", "reviewer": "Reviewer_c4E9", "summary": "The paper considers a top-down approach to hierarchical planning/option generation. The proposed Strategist Agent builds a tree structure that specifies alternative plans or sequential plans (approach and plan nodes), which are broken further, if necessary. The tree structure is generated by a sufficiently `strong' LLM with prompts describing the design of the Strategist Agent, and a sufficiently rich description of the domain. The LLM attaches a reward function and a feasibility evaluation to the nodes as well.\n\nThe proposed algorithm is illustrated on the Crafter environment. There is an empirical comparison showing favorable performance compared to PPO.  \n\nThe paper also include a generic result that links the specificity and value of a policy to the sample complexity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback regarding our evaluation methodology. To address these concerns, we have substantially expanded our experimental comparisons to include **two additional RL baselines specifically designed for sparse reward settings and long episodes (DreamerV3 [1], EDE [2]), as well as two LLM-based approaches (LLM-as-policy, Eureka [3]).**\n\n## Model-based and exploration-focused RL baselines\nWe now include **DreamerV3** (50M parameters) [1], a leading model-based RL approach, which slightly outperforms PPO but still falls short of our Strategist+PPO combination. Additionally, we evaluate **EDE** (Exploration via Distributional Ensemble) [2], another high-performing method on the original Crafter leaderboard. [Preliminary results](https://imgur.com/a/NuBI80H) indicate that it achieves strong performance on Crafter-Medium (~6 avg reward after 2M training steps) by following a hunting strategy while learning to maximise survival to extend episodes. Notably, our Strategist framework is complementary to the underlying RL algorithm. When paired with EDE, it introduces a breeding strategy that vanilla EDE fails to uncover. This results in a **2× improvement in reward**, highlighting how high-level strategic guidance enhances and expands the exploration capabilities of existing methods. This also demonstrates that our method is broadly applicable across different RL approaches and can draw from the strengths of each method depending on the environment of interest.\n\n## Comparison with LLM-based approaches\nWe have expanded our evaluation to include two prominent LLM-based baselines to clarify our method's advantages. First, we implemented an **LLM-as-Policy** baseline using GPT-4o-mini, which selects actions at each step directly. This baseline achieved only 0.43 ± 1.01 reward (N=15), significantly underperforming compared to our Strategist framework. Furthermore, its computational cost is high due to requiring inference at each timestep, which underscores its impracticality for complex environments.\n\nSecond, we compared our method to **Eureka** [3], an LLM-based approach designed for iterative reward discovery using GPT-4o. Eureka’s primary objective is finding high-quality reward functions, which often comes at the expense of sample efficiency. In our experiments, Eureka's evolutionary search (population size 3, depth 3) did not yield rewards meaningfully superior to the default environment reward, performing equivalently to baseline PPO despite utilizing greater exploration resources. For completeness, we will also provide results with a deeper Eureka run in the revised manuscript. These results clearly illustrate that our Strategist framework—translating high-level LLM-generated strategies into actionable rewards—is significantly more effective and sample-efficient than both direct LLM action-selection and standard iterative reward optimization methods, especially in complex, strategic domains.\n\n## Modified Crafter environment\nRegarding the modification of the Crafter environment, our intention was to explicitly showcase the benefits of strategic exploration facilitated by our approach. By adjusting the environment to feature clear alternative strategies, such as hunting versus breeding, we aimed to rigorously test the Strategist Agent's ability to identify and exploit distinct high-level strategies effectively. \n\n## Comparison with SayCan\nWe thank the reviewer for highlighting important differences between our method and SayCan. Indeed, SayCan addresses bottom-up planning in robotic environments, which typically admit defined primitive actions. In such contexts, feasibility assessment through pretrained value functions is straightforward but requires costly pre-training of low-level actors, limiting sample efficiency.\n\nConversely, our top-down Strategist Agent is tailored for complex strategy-driven environments that inherently have a wide range of possible high-level actions. Unlike SayCan, our method does not rely on a fixed set of predefined primitives, making it suitable for broader strategic exploration. Our approach fundamentally capitalizes on the domain-agnostic strategic planning capabilities of LLMs combined with language-conditioned RL, enabling iterative refinement and robust exploration. Thus, our framework is uniquely positioned to effectively address settings where bottom-up methods like SayCan would be less feasible or impractical.\n\nThank you again for your valuable feedback and suggestions.\n\n## References\n[1] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\". \n\n[2] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. \"Uncertainty-driven exploration for generalization in reinforcement learning.\".\n\n[3] Ma, Yecheng Jason, et al. \"Eureka: Human-Level Reward Design via Coding Large Language Models.\".", "claims_and_evidence": "The paper claims empirical superiority over PPO on the task considered, which is not surprising for a task with sparse rewards and long scenarios. The approach seems interesting enough, but the evaluation is limited.", "methods_and_evaluation_criteria": "The evaluation method is reasonable, but the set of baselines is very limited. Even in the absence of a larger pool of implemented/tested baselines (there should have been some that are suited to deal with sparse rewards and long episodes), the test environment could have been chosen such that the scoreboard of Crafter would have been a measure for comparison.", "theoretical_claims": "I parsed the proofs, but the theoretical result has a minor role in the paper anyway.", "experimental_designs_or_analyses": "I have looked closely to the empirical design, including the prompt details.", "relation_to_broader_scientific_literature": "It is an interesting approach, very much relevant for training hierarchical agents using LLM.", "essential_references_not_discussed": "The most relevant work is sufficiently discussed (especially in the supplementary material)", "other_strengths_and_weaknesses": "The prompts and the generated plans are quite illustrative.", "other_comments_or_suggestions": "The authors argue that SayCan can is more limited because the options are specified bottom-up, while here it is completely top-down. While this is true, I would add that the two operated in a very different environments: the environment discussed here is essential a strategy game, while SayCan deals with robots. It is much easier to evaluate the feasibility and shape reward function in the former. LLMs would probably would have more difficulty in defining options in the latter. The prompts detailed in this paper are also quite rich in domain knowledge that helps defining the rewards.", "questions_for_authors": "Would it be possible to compare Strategist Agent with the leader board for Reacher?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tAn6PH7uOJ", "reviewer": "Reviewer_rDHf", "summary": "The paper proposed Strategic Planning, a top-down approach for decomposing complicated reinforcement learning (RL) task into natural language-described sub-tasks. The paper also designs a reward shaping methodology that translates these strategies expressed in natural language into quantitative feedback for RL methods. Detailed implementation is introduced with theoretical justifications. The overall pipeline has been tested on Crafter environment to show its effectiveness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your detailed and insightful feedback. \n\n## Additional RL baselines\nFollowing your suggestion, we have added **two competitive RL baselines**: DreamerV3 [1] and Exploration via Distributional Ensemble (EDE) [2], both high ranking on the original Crafter leaderboard. DreamerV3 achieves slightly better performance than PPO (0.15 ± 1.39 (std, N=4)) but remains below our Strategist + PPO framework.\n\n[Preliminary results](https://imgur.com/a/NuBI80H) indicate that EDE reaches high reward on Crafter-Medium (~6 avg) by initially discovering a hunting strategy and subsequently learning survival skills to lengthen episodes further. However, our Strategist approach is orthogonal to the underlying RL agent: combined with EDE, the Breeding strategy—which vanilla EDE does not exploit—results in a **2× increase in rewards compared to EDE’s final performance**. This highlights the value of our top-down approach in uncovering effective, alternative strategies that traditional bottom-up methods may not discover.\n\n## LLM baselines \n\nWe agree it is important to compare our method to other LLM-based baselines. We have added **two additional LLM baselines**: LLM-as-Policy and Eureka [3]. \n\nFirst, we implemented an **LLM-as-Policy** baseline with GPT-4o-mini, which selects actions at each step directly. This baseline achieved only 0.43 ± 1.01 reward (N=15), significantly underperforming compared to our Strategist framework. Furthermore, its computational cost is high due to requiring inference at each timestep.\n\nSecond, we compared our method to **Eureka**, an LLM-based approach designed for iterative reward discovery using GPT-4o. Eureka’s primary objective is finding high-quality reward functions, without aiming for sample efficiency. In our experiments, Eureka's evolutionary search (population size 3, depth 3) did not yield rewards meaningfully superior to the default environment reward, performing equivalently to baseline PPO despite utilizing greater exploration resources. For completeness, we will provide results with a deeper Eureka run in the revised manuscript. \n\nSayCan is designed for robotic tasks using predefined action primitives and a pretrained value function for feasibility evaluation. This setup does not transfer well to environments where the space of meaningful behaviors is broader and the choice of primitives is not natural, plus the existence of a pre-trained low-level actor is a strong requirement, rarely available. Instead, our method leverages LLMs for high-level strategy generation and uses lang. cond. RL to ground those strategies through interaction. In practice, approaches like SayCan and action-proposing LLMs are actually orthogonal to ours—they focus on low-level decision-making given a set of primitives, which could be combined with our high-level strategic planning and exploration.\n\n## Additional comments\n- **Generalizability**: we clarify that one of the strategist's strengths is precisely its flexibility. Our method is environment-agnostic conditioned on the LLM being able to generate meaningful strategies given domain descriptions, which is reasonable for a broad range of tasks. Unlike other methods, the Strategist does not require pre-training, expert demonstrations or predefined sub-goals.\n- **Hierarchical RL**: Thank you for pointing out key prior work. We expanded Appendix D.3 to better situate our method within the hierarchical RL literature. In addition to citing Feudal RL (Dayan & Hinton, 1992), we now include discussions of MAXQ (Dietterich, 2000) and Feudal Networks (Vezhnevets et al., 2017), highlighting both differences and potential complementarities with our framework.\n- **Confidence intervals in Fig. 4**: We acknowledge the nonstandard use of the 80% interval. We now include 68% intervals in the appendix, noting that this change does not affect the qualitative conclusions.\n- **Crafter environment modifications**: We modified Crafter to better expose strategic trade-offs (e.g., hunting vs. breeding), making the evaluation of strategy discovery more explicit. Our method remains applicable to the original environment, but we found it less effective for highlighting the core contributions given its achievement-based reward.\n- **Minecraft-meat collection example**: To address accessibility, we added a real-world motivating example (hospital management) in the appendix, illustrating how our framework supports discovering diverse high-level strategies in complex decision-making settings.\n\nThank you once again for your constructive critique and valuable recommendations. \n\n## References\n[1] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\". \n\n[2] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. \"Uncertainty-driven exploration for generalization in reinforcement learning.\".\n\n[3] Ma, Yecheng Jason, et al. \"Eureka: Human-level reward design via coding large language models.\"", "claims_and_evidence": "See below.", "methods_and_evaluation_criteria": "Yes, it makes sense.", "theoretical_claims": "No problem with me.", "experimental_designs_or_analyses": "Yes, I have checked. See below.", "relation_to_broader_scientific_literature": "The current paper discussed using LLM and human prompts to make more effective hierarchy of planning and design auxillary reward functions to help solving RL problems. The high-level idea of using LLM to make plans and design options are not new (e.g., SayCan), the using LLM to design reward functions is not new (e.g., Eureka). However, the current paper is novel in terms its detailed implementation and methodology.", "essential_references_not_discussed": "Many classic hierarchical RL studies should be discussed. E.g., Feudal RL actually shares similar high-level idea of top-down hierarchy.\n\n\n- Dayan P, Hinton G E. Feudal reinforcement learning[J]. Advances in neural information processing systems, 1992, 5.\n\n- Dietterich T G. Hierarchical reinforcement learning with the MAXQ value function decomposition[J]. Journal of artificial intelligence research, 2000, 13: 227-303.\n\n- Vezhnevets A S, Osindero S, Schaul T, et al. Feudal networks for hierarchical reinforcement learning[C]//International conference on machine learning. PMLR, 2017: 3540-3549.", "other_strengths_and_weaknesses": "### [Strengths]\n- Ideas are theoretically justified.\n- Figures are nicely made.\n- The writting is easy to follow.\n\n\n### [Weakness]\n- The largest concern I've got is the to what degree the proposed method can be effective in general. If I understood correcly, Strategic Planning requires humans to input a textual strategy, e.g., \"collect much meat as much as possible\". I am not sure how much this can generalize to other tasks. How do you know a strategy is good in general?\n- The comprehensiveness of empirical results is yet to be improved. That is to say, it is expected to have more types of tasks and environments to be evaluated, and more strong baselines (rather than vanilla PPO, but those also used LLMs, i.e., SayCan and Eureka) to be compared with, to meet the quality bar of ICML.\n- Using Minecraft-Collecting Meat as an illustrative example may not be a proper choice since not everyone is familier with it.\n- The introduction is broken into several parts with respective subtitles. This may prevent the reading flow and make readers feel discountinual. \n- Many classic works on hierarchical RL are ignored, see \"Essential References Not Discussed\".", "other_comments_or_suggestions": "- Figrue 4 contains an 80% percentile confidence interval, which is not a common choice (95% or 68% is more common), is there any reason to do so?\n- The Crafter environment was modified by the authors in this study? Could the authors explain why? Can Strategic Planning also work in the non-modified version?", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BLTyLBu2xk", "reviewer": "Reviewer_JCuE", "summary": "This paper proposes a top-down learning method for Reinforcement Learning (RL). The method leverages a Large Language Model (LLM) to decompose a complex task from high-level goals into fine-grained plans, considering specificity, value and feasibility. Following decomposition, these plans are transformed into reward signals through reward shaping for RL training. The paper demonstrates the effectiveness of using rewards derived from LLM decomposition. In experiments, the proposed method is evaluated on the Crafter environment, and the results show that it outperforms conventional RL methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review and for recognizing the novel perspective. \n\nWe appreciate the reviewer's concern about lacking comparisons between our top-down approach and bottom-up approaches that also leverage LLMs. This is a valid point, and we have now expanded our evaluation to include **two additional strong RL baselines (DreamerV3 [1] and EDE [2])** as well as **two LLM-based bottom-up approaches (LLM-as-policy and Eureka [3])** to provide a more comprehensive assessment of our method's effectiveness.\n\n## Extension to other RL baselines\nWe've added **DreamerV3** (50M parameters) [1] and **Exploration via Distributional Ensemble (EDE)** [2], both highly ranked on the original Crafter leaderboard. **DreamerV3** achieves slightly better performance than PPO (0.15 ± 1.39, N=4) but remains below our Strategist + PPO framework. [Preliminary results](https://imgur.com/a/NuBI80H) indicate that **EDE** reaches high reward (~6 avg after 2M training steps) by initially discovering a hunting strategy and subsequently learning survival skills to lengthen episodes. Critically, our Strategist framework can be combined with EDE, enabling it to be steered toward distinct strategies such as hunting and breeding. While the hunting strategy performs comparably to early EDE stages, the breeding strategy—which was overlooked by vanilla EDE—enables a **2× reward improvement over EDE's final performance**. This demonstrates that our approach is broadly applicable across different RL algorithms and can enhance the exploration capabilities of existing methods.\n\n## New LLM-based baseline experiments\nWe implemented an **LLM-as-policy** approach (shown to be effective in prior works such as Voyager and Reflexion), using GPT-4o-mini for direct action selection at each timestep. Notably, this is the same model used to annotate states in the Strategist’s reward shaping procedure to ensure a consistent LLM backbone across both baselines and our main method. This approach achieved 0.43 ± 1.01 reward (N=15), substantially underperforming our Strategist framework combined with PPO or EDE agents while being computationally expensive due to per-step inference requirements. Additionally, we evaluated **Eureka** [3] with GPT-4o, a state-of-the-art bottom-up LLM-based iterative reward discovery method. In our preliminary experiments with an evolutionary search (population=3, depth=3), Eureka's designed rewards didn't meaningfully outperform the environment reward, yielding results equivalent to baseline PPO while requiring a higher exploration budget. These comparisons directly demonstrate the advantages of our top-down strategic approach over bottom-up LLM-augmented alternatives in terms of both effectiveness and computational efficiency.\n \n## Distinction between bottom-up and top-down methods\nWe appreciate the clarification requested regarding the distinction between top-down and bottom-up methods. Indeed, we agree that bottom-up approaches augmented with LLMs can significantly improve sample complexity, and we emphasize that our top-down method is not inherently mutually exclusive but rather orthogonal to these bottom-up methods. Consequently, direct empirical comparisons were initially omitted as our goal was not to position these approaches in competition, but to present a complementary framework. For instance, methods using an LLM as a policy or hierarchical RL can be readily integrated alongside our Strategist framework to potentially further improve sample efficiency, as discussed in our future work section. \n\n## Additional comments and revisions\nWe acknowledge the importance of the detailed questions raised about Section 4.3, and will address these clearly in a newly added appendix section, briefly summarized here due to character constraints:\n\n- Regarding feasibility, this probability represents how much of the probability mass of the current strategy aligns with policies achievable by the agent’s learning hypothesis class.\n- Initially, attributes like specificity, feasibility, and value are estimated by the LLM based on its prior knowledge. As demonstrated in Section 5.3, these estimates can be dynamically updated with actual environment interactions, opening possibilities for future multi-armed bandit-inspired strategy adjustments during learning, which we encourage in our future work discussion.\n\nThank you again for your valuable insights and constructive suggestions.\n\n## References\n[1] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\". \n\n[2] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. \"Uncertainty-driven exploration for generalization in reinforcement learning.\".\n\n[3] Ma, Yecheng Jason, et al. \"Eureka: Human-Level Reward Design via Coding Large Language Models.\".", "claims_and_evidence": "This paper claims that bottom-up approaches learn low-level actions and improve through extensive trial and error, and argues that LLM-augmented RL methods (e.g., Hierarchical RL, LLM-as-policy, Reward shaping) also inherit the drawbacks associated with the bottom-up paradigm. However, these methods can also accelerate the convergence of RL agents to optimal solutions by leveraging language priors, which reduce the need for large-scale trial and error. Consequently, the primary advantage of top-down approaches remains unclear, and the paper lacks experimental comparisons between its proposed method and other LLM-augmented RL techniques.", "methods_and_evaluation_criteria": "The evaluation benchmark is limited, consisting of only two tasks, \"Hunt\" and \"Breed,\" which are not sufficiently representative.", "theoretical_claims": "I have checked theorem 2.1.", "experimental_designs_or_analyses": "The baselines are lacking. In Sections 1-3, this paper discusses the advantages of top-down approaches compared to bottom-up ones that are also augmented by LLMs (e.g., LLM as policy, hierarchical RL). However, in the experiments, the authors only compared their method with traditional RL approaches, such as PPO, and did not conduct comparisons with bottom-up approaches that also use LLMs. This omission makes it difficult to demonstrate the superiority of the top-down approach in scenarios where LLMs are introduced in both paradigms.", "relation_to_broader_scientific_literature": "This paper enhances RL learning using an LLM following a top-down paradigm. While previous research has also explored LLM-augmented RL, these studies typically adopt a bottom-up approach, which often requires extensive trial-and-error.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "** Strengths**\n\n1. This paper introduces a top-down paradigm for RL with LLMs, offering a novel perspective on task decomposition and RL learning.\n\n2. The paper proposes a flexible tree structure for task decomposition, featuring two node types – approach nodes and plan nodes – and three selection metrics.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. In Section 4.3, how is the entropy H calculated using LLM?\n\n2. In Section 4.3, how is the value v calculated? Is J(·) obtained through rollout or from the environment’s reward process?\n\n3. In Section 4.3, what does the probability in feasibility refer to?\n\n4. In lines 152-153, it is stated that the entropy H needs to be lower than $H_{max}$ to satisfy being sufficiently specific. In Section 4.3, how does the calculation of specificity ensure this requirement is met?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a top-down RL method called Strategist, which uses LLMs to generate high-level strategies and transform them into reward signals to learn policy. Experiments on a modified Crafter environment show that Strategist allows faster training convergence compared to standard RL algorithms.

The paper is well-written and clearly presents its motivation. The proposed integration of LLMs with a top-down RL approach is novel. The main weakness is its limited experiments, which are only verified in a single environment. While the results are promising, it remains unclear whether the method can generalize to other tasks.
Decision: Accept (poster)