ICML 2025

Strategic Planning: A Top-Down Approach to Option Generation
Max Ruiz Luyten * 1 Antonin Berthon * 1 Mihaela van der Schaar 1
Abstract
Real-world human decision making often relies
on strategic planning, where high-level goals
guide the formulation of sub-goals and subsequent
actions, as evidenced by domains such as health-
care, business, and urban policy. Despite notable
successes in controlled settings, conventional re-
inforcement learning (RL) follows a bottom-up
framework, which can struggle to adapt to real-
world complexities such as sparse rewards and
limited exploration budgets. While methods like
hierarchical RL and environment shaping provide
partial solutions, they frequently rely on either
ad hoc designs (e.g. choosing the set of high-
level actions) or purely data-driven discovery of
high-level actions that still requires significant ex-
ploration. In this paper, we introduce a top-down
RL framework that explicitly leverages human-
inspired strategy to reduce sample complexity,
guide exploration, and enable high-level decision
making. We first formalize the Strategy Problem,
which frames policy generation as finding distri-
butions over policies that balance specificity and
value. Building on this definition, we propose
the Strategist agent—an iterative framework that
leverages large language models to synthesize do-
main knowledge into a structured representation
of actionable strategies and sub-goals. We fur-
ther develop a reward shaping methodology that
translates these strategies expressed in natural lan-
guage into quantitative feedback for RL methods.
Empirically, we demonstrate that our framework
significantly enhances the performance of differ-
ent underlying RL algorithms, leading to faster
convergence and the discovery of more complex
behaviors. Taken together, our findings highlight
that top-down strategic exploration opens new
avenues to improve RL in real-world decision
problems.
*Equal contribution 1University of Cambridge. Corresponden...