Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Authors: Zican Hu
Wei Liu
Xiaoye Qu
Xiangyu Yue
Chunlin Chen
Zhi Wang
Yu Cheng
Venue: ICML 2025 poster (2025)
Keywords: Language agents
Hierarchical reinforcement learning
Offline reinforcement learning
Primary area: deep_learning->large_language_models
TL;DR: We leverage offline RL with a parameter-efficient and generally applicable hierarchy to ground LLMs as efficient decision-making agents.

Abstract: While showing sophisticated reasoning abilities, large language models (LLMs) still struggle with long-horizon decision-making tasks due to deficient exploration and long-term credit assignment, especially in sparse-reward scenarios. Inspired by the divide-and-conquer principle, we propose an innovative framework **GLIDER** (**G**rounding **L**anguage Models as Eff**I**cient **D**ecision-Making Agents via Offline Hi**E**rarchical **R**einforcement Learning) that introduces a parameter-efficient and generally applicable hierarchy to LLM policies. We develop a scheme where the low-level controller is supervised with abstract, step-by-step plans that are learned and instructed by the high-level policy. This design decomposes complicated problems into a series of coherent chain-of-thought reasoning sub-tasks, providing flexible temporal abstraction to significantly enhance exploration and learning for long-horizon tasks. Furthermore, GLIDER facilitates fast online adaptation to non-stationary environments owing to the strong transferability of its task-agnostic low-level skills. Experiments on ScienceWorld and ALFWorld benchmarks show that GLIDER achieves consistent performance gains, along with enhanced generalization capabilities.

Divide and Conquer: via Offline Hierarchical Reinforcement Learning
Grounding LLMs as Efficient Decision-Making Agents
Zican Hu"
12
Wei Liu Xiaoye Qu
Xiangyu Yue
Chunlin Chen
Zhi Wang
12
Yu Cheng
Abstract
high planer
Hier w/o Hier
R8.34
While showing sophisticated reasoning abilities, large language models (LLMs) still struggle with long-horizon decision-makinn tasks due to defi- cient exploration and long-term credit assignment, especially in sparse-reward scenarios. Inspired by the hwvide-and-conuerr principle, we propose innovative framework GLIDER (Grounding Language Models as EffIcient Decision-Making Agents via Offline HiErarchical Reinforcement Learning) that introduces a parameter-efficient and generally applicable hierarchy to LLM poli- cies. We develop scheme where the low-level controller is supervised with abstract, step-by- step plans that are learned and instructed by the high-level policy. This design decomposes complicated problems into a series of coherent thain-of-thought reasoning sub-tasks, providing flexible temporal abstraction to significantly en- hance exploration and learning for long-horizer tasks. Furthermore, GLIDER facilitates fast on- line adaptation to non-stationary environments owing to the strong transferability of its task- agnostic low-level skills. Experiments on Sci- ence World and ALFWorld benchmarks show that GLIDER achieves consistent performance gains, along with enhanced generalization capabilities.
Task: boiling water
53,94
go to kitchen low executor
1. open kitchen door go to kitchen
in kitchen
prepare tools
6
1. pick up pot take thermometer
prepare done
filling water
1. move pot to sink activate sink sink is broken
Figure GLIDER's hierarchical framework, showing sig- nificant performance gain over non-hierarchical approaches.
2024). Recent attempts to exploit large language models (LLMs) as agents have shown commendable results tack- ling interactive decision- decision-making tasks (Li et al., 2022; Yao et al., 2023b; Song et al., 2024). Prompt- based methods, like ReAct (Yao al., 2023b) and Reflexion (Shinn al., 2023), recursively augment the prompt to a frozen LLM with bal feedback. They are prone to exceed the input length limit of in-context learning, especially for Long-horizon tasks. Scaling with supervised fine-tuning techniques can further unlock the potential LLMs for downstream ap- plications (Zeng al., 2023; Chen et al., 2023; Lin et al., 2023). However, their performance is highly dependent expensive expert demonstrations and be limited due to deficient exploration of target environments.
1. Introduction
A longstanding goal of artificial general intelligence is to build agents capable of reasoning, decision-making, and communication (Wooldridge & Jennings, 1995; Xu et al.,
Intelligent agents must excel at both imitating demonstra- tions and adapting behaviors through trial-and-error (Silver et al., 2021; Rafailov et al., 2023). Modern approaches adopt reinforcement learning (RL) algorithms to steer LLMs to- ward user-specified tasks (Ouyang al., 2022), such as offline Q-learning (Snell al., 2023), PPO (Zhai al., 2024; Szot et al., 2024), or DPO (Song et al., 2024). This paradigm enables SOTA performance in advanced LLMs like OpenAl ol and DeepSeek-Ri (Guo et al., 2025). How- ever, RL intrinsically requires tedious and vast environment interactions, leading to brittle performance and poor sam-
'Nanjing University "Shanghai AI Laboratory 'The Hong Kong University of Science and Technology "The Chinese University of Hong Kong. Interns at Shanghai AI Laboratory. [R] Cor- responding Authors: Zhi Wang <zhiwang nju.edu.en>>. Yu Cheng <chengyueeeecocccc ese, cuhk.edu.hkk The code available at https///ithub.com/NJU http:/hhhb..com////UUAAALLLLEEE
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
ple efficiency (Burda et al., 2019; Mahankali al., 2024). Building efficient LLM agents with open-ended textual com- mands poses several challenges, such as tackling huge ac- tion spaces, executing long-horizon planning, and learn- ing from sparse-reward feedback (Rocamonde et al., 2024; Dwaracherla et al., 2024). Existing works still struggle with complex tasks that demand broad spectrum of vital capabil- ities, including long-term credit assignment, understanding the real physical world, and sophisticated exploration with structured reasoning (Qiao et al., 2024; Zhou et al., 2024).
2. Related Work
LLMs as Decision-Making Agents. With a wealth of semantic knowledge about the world, LLMs have shown remarkable potential in building competent agents across diverse domains (Xi et al., 2023), including reasoning (Wei et al., 2022; Zhou et al., 2023; Luo et al., 2024), robotics (Ahn et al., 2022; Shah et al., 2023) and multi- agent (Chen et al., 2024; Ma et al., 2024), Early stud- ies use a prompt-based based framework, such as classical CoT methods (Wei et al., 2022; Yao et al., 2023a; Wang et al., 2023a) that prompt LLMs with intermediate reasoning steps. Follow-up up research employs synergy between reasoning and acting (Yao et al., 2023b), incorporates self-reflectiv ver- bal feedback (Shinn et al., 2023), and uses strategic reason- ing (Gandhi et al., 2023). They use recursive feedback traces to augment prompts, which helps address long sequence and complex, long-horizon task challenges.
Humans naturally tackle complex problems through hierar- chical decomposition (Sutton et al., 1999). This divide-and- conquer principle is evident across scales in natural systems, from how corporations divide into specialized departments to how biological systems organize cells form tissues and organs. The hierarchy design plays crucial role in advanc- ing frontier research on language agents (Li et al., 2024; Zhou et al., 2024) and embodied intelligence (Ahn et al., 2022; Black et al., 2024), showcasing remarkable efficiency for solving intricate tasks in a more human-likk -lkee manner.
Scaling with supervised fine-tuning techniques can unlock the potential for downstream tasks (Hu et al., 2022), such as fine-tuning LLM agents on oracle action trajectories (Lin dataset
Inspired by this, we propose an innovative framework GLIDER (Grounding Language Models as Efflcient Decision-Makee Making Agents via Offline HiErarchical RL) that introduces a parameter-effiiient and generally applicable hierarchy to train competent LLM policies for complex inter- active tasks. Our scheme contains two L.M policies, where the low-level controller is supervised to achieve abstract, step-by-step plans learned and proposed by the high-level instructor. By harnessing the strong reasoning and plan- ning capabilities of LLMs, we can decompose complicated problems into a series of coherent chain-of-thour thought (CoT) reasoning sub-tasks and perform efficient exploration in a semantically structured space. To enhance learning stability. we first build a base agent via behavior cloning, followed by reinforcement fine-tuning of the hierarchical token-level actors and sentence-level critics. This two-level agent is trained in offline mode using hierarchical datasets achieve prominent sample efficiency, and also can be seamlessly de- ployed for offline-to-online fine-tuning scenarios.
et al., 2023), on lightweight instruction-tunde containing high-quality interaction trajectories (Zeng et al., 2023), and on agent trajectories generated from multiple tasks and prompting methods (Chen et al., 2023). Due to inherent limitations, the performance relies heavily on expensive high-quality data and could easily be restricted by deficient exploration of target environments.
RL offers a natural paradigm to unleash the LLM agents' Recision-making capabilities (Ouyang et al., 2022). Snell et al. (2023) guides language generation towards maximiz- ing user-specified utility functions using implicit language Q-learning. ETO (Song et al., 2024) collects contrastive trajectory pairs from interactions to update the LLM policy using DPO (Rafailov et al., 2023). Other works employ a similar pipeline where an LLM policy interacts with the en- vironment to receive goal-directed task rewards, which are then used to fine-tune the policy with classical algorithms like PPO (Zhai et al., 2024; Szot et al., 2024; Tan et al., 2024). Building LLM agents with open-ended textual com- mands can involve a huge action space, and mng-horizon planning or sparse-reward scenarios (Mahankali et al., 2024; Zhou et al., 2024). This motivates us to introduce hierarchy to ground LLMs as efficient decision-making agents.
In summary, our main contributions are as follows:
Inspired by the divide-and-conquer principle, we propose an offline hierarchical framework GLIDER, empowering LLM agents to tackle complex decision-making tasks via sophisticated exploration and structured reasoning.
Hierarchical RL emerges as a powerful framework for managing complexity in decision-making tasks. Classical approaches of Options (Sutton et al., 1999; Bacon al., 2017) and MAX-Q (Dietterich, 2000) formalize temporal abstractions in RL. Recent advances have significantly ex- panded these foundations, such as HiRO (Nachum et al., 2018) with data-efficient off-policy training for hierarchical policies, HAC (Levy et al., 2019) with parallel training of 3- level hierarchies, and HiPPO (Li et al., 2020) with efficient
Our method enables fast offline-to-online adaptation to non-stationary environments by developing highly gen- eralizable skills through hierarchical LLM agents.
Comprehensive studies on ScienceWorld and ALFWorld benchmarks show that our method consistently improves performance and generalization capacity, surpassing a range of baselines by a significant margin.
2

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
select gate
hh
VI à
Actor-Critic
task
Hierachy Structure
concatanate
8
π
value head
9t
ac t
9t
9t
high prompt
8t
π
8 s
π
LLM
Fine-Tunite
low prompt
at
attc
RS
Environment
Hierarchy Control
Og 80
(a)
(b)
high
low
r2t
t+c
sampling
data collection
Q V
S
High
d
oz
00
go
Ro
0c
0z
gt
R
Ottc
R
S
A
S x (g, A)
Low
(c)
SFT
ORL
020
(d)
00
a0
00
01
Oc
O2
at
f
ot+1
Ot+c
Figure 2: Overview of the GLIDER framework. (a) Hierarchical Actor-Critic architecture with prompt-controlled high- and low-level training on sampled trajectories from offline datasets. (b) Hierarchical policy structure where the high-level v² generates sub-task only when the low-level n' executes primitive actions for e steps. The high-level policy provides the low-level with an intrinsic reward that indicates the sub-task completion, and collects environment rewards across c timesteps as one-time reward as R,= -r (c) The training pipeline comprises SFT, ORL (offline RL), and 020 (offline-to-online RL) stages. (d) Structured hierarchical trajectories composed of high-level transitions (d: ot St. Rr, Octc) and low-level transitions (g: Ot, ae, ft. 0r+1).
hierarchical policy gradient approximation for robust skill training. In general, persistent challenge is the dependence on domain expertise to specify meaningful hierarchies. This limitation also motivates us to harness the strong semantic understanding and reasoning abilities of I.LMs for natural task decomposition with an autonomous hierarchy.
ground LLMs as efficient decision-making agents for tack- ling complex interactive tasks. Figure 2 illustrates the frame- work, containing a three-stage pipeline of base agent con- struction via supervised fine-tuning, policy refinement via offline RI., and seamless adaptation to online deployment. The algorithm pseudocode is given in Appendix A, and detailed implementations are presented follows.
Offfine-to-Ohi Online RL. Offline RL (Levine et al., 2020) har- nesses offline data without environment interactions, yield- ing effective algorithms such as BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), and Fisher-BRC (Kostrikov et al., 2021). Also, it remains beneficial to fine-tune the pretrained offline policy with further online interactions, that is, offline- to-online RL (Nair al., 2020). However, such benefit is often diminished due remarkable distribution shifts between pretraining and deployment, leading to accumu- lated bootstrap errors (Lee et al., 2022; Wang et al., 2023b). Moreover, existing works often require extensive retraining for new tasks (Yu & Zhang, 2023). In contrast, we show that our method achieves fast online adaptation to non-stationary environments via training highly generalizable low-level skills that are robust distribution shifts.
3.1. Problem Setup of the LLM Agent
We formulate the agent task as a standard Markov decision process (MDP) with a tuple (S, T, R, 7). where S/A the state/action space, T/R is the state transition/reward function, and 7 E (0, 1] is the discount factor. We introduce an LLM-based policy, To S A + [0, 1], probability dis- tribution that maps states to actions, where both S and A are drawn from text spaces constrained by user-specified tasks, and policy parameters 9 are initialized from pretrained LLM. The objective to optimize the policy receive maximal expected return as J(T) E, E, /rr..
3.2. Hierarchical Architecture of the LLM Agent
We extend the LLM agent setup to a hierarchical two-layer strueture, with high-level policy Th and low-level con- troller <//- The high-level policy operates at coarser layer of task planning and sets sub-task goals for the low-level to accomplish. At each timestep I, the environment provides
3. Method
In this section, we present GLIDER, which introduces a parameter-efffcient and generally applicable hierarchy to
3

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
an observation Og- The high-level planner = receives the observation og together with textual task description d, and produces high-level goal (or sub-task) as gr ~ (- d, ot) when & = 0 (mod c). This provides temporal abstraction, since high-level decisions are made only every e stpps.. I For the next с timesteps, the low-level controller receives the environment observation or and goal gt. and produces low- level primitive action as ar ~ m (- (t). The action at is applied to the environment, which yields a reward re and transitions to new observation 1.+11 The high-level pol- icy collects environment rewards through these e timesteps, storing the high-level transition (d: ot, Dr, Eraa+e-1 oc+e) for offline training. Correspondingly, the high-level dataset for offline learning stages is constructed as
model (Brown et al., 2020). In contrast, traditional hierarchi- cal methods usually train independent models each level, resulting in a multiplication of model parameters (Nachum et al., 2018; Levy et al., 2019; Li et al., 2020).
Our hierarchy setup achieves broad applicability. The hierarchical structure provides temporal abstraction with ef- ficient exploration since high-level decisions are made only when the low-Ievel controller executes for several steps. The high-level policy unlocks the chain-of-though reasoning of LLMs to decompose a complicated task into a series of coherent sub-task plans, while the low-level model trans- lates abstract plans into precise, executable atomic actions. Our setup achieves generality by training the low-level pol- iey to accomplish sub-task goals learned and instructed by the high-level planner. The high-level planner is guided by environment provided perooiideet-- rewards, while the low-level policy is instructed by the sub-task completion signal derived from environment observations. The whole learning process elim- inates the necessity for any manual or task-specific design, making it broadly applicable.
Dd = -iN [d] (oo. 9o. Dro:c-1. 0c),
9t. It. Eri:te-1,, octo) ],.], ...].
(1)
which captures strategic task planning over N trajectories.
The high-level policy provides the low-level with an intrin- sic reward f that indicates the sub-task completion (1 when the sub-task is completed and 0 otherwise). The sub-task completion can be easily accessible from the environment observation or, without requiring any manual design or do- main knowledge.
3.3. Base Agent Construction via Behavior Cloning
Directly deploying LLMs as decision-making agents in downstream tasks may generate hallucinatory or inconsis- tent actions and perform brainless trial-and-error due to the semantic discrepancy between natural language (trained with next-token prediction) and user-specified environments. To improve learning stability and sample efficiency, we first construct a base LLM agent through supervised fine-tuning of the hierarchical actors using pre-collected demonstration trajectories. This behavior cloning process aligns the initial policy with valid action sequences and serves as a solid starting point for building a powerful agent.
Each generated goal It corresponds to a sequence of e atomic transitions in the low-level dataset as
D'= = Ex [ ge: (oe, Q1, Pr. 04+1), .....
(0t+o-1: 0t400 Peto-1, (2)
We design a parameter-effiiien hierarchical model ar- chitecture. As shown in Figure 2-(a), our model offers superior parameter efficiency from two perspectives. First, the actor and critic share the same frozen LLM backbone, each introducing a minimal number of parameters for effi- cient fine-tuning at a lightweight computing cost. The actor is formed by augmenting the backbone with LoRA (Hu et al., 2022), which adds a trainable low-rank bypass to each transformer block. The critic is constructed by adding additional MLP layers to the last transformer block of the backbone. Second, the high- and low-level policies share the same actor-critic models, differing only in a hierarchy prompt that specifies the level of current inputs. This design benefits from harnessing the powerful capability of LLMMs to perform in-context learning, i.e., tackling series of com- plex tasks by feeding short prompts to a single foundation
Specifically, we imitate the behavior patterns, i.e., the state- to-action mapping function, within both levels of datasets in Eqs. (1) and (2). The objective function of behavior cloning via supervised fine-tuning is to maximize the log- likelihood of the observed data. Pre-trained LLMs tend to generate lengthy, verbose sentences that could be redundant and challenging to understand in the user-specified envi- ronment. Hence, we incorporate a length regularization term to encourage the LLM policy to generate concise task plans and atomic actions for effective interaction with the environment. The final loss function is formulated as
(SFT(0) = -B(d.og)~Di [logT] (g (d, o)] + A RA ( [log ( g. o)] + > n4,
'c could differ across sub-tasks, as harder sub-tasks naturally require more primitive actions to accomplish.
where À is the length regularization ratio, and ma/ma is the output length of the high-/lowllveel policy.
"In boiling water task, the high-level policy decomposes it into atomic subtasks (e.g., navigation, tools preparation). For navigation subtask, the low-level policy receives reward of upon reaching kitchen (verified through observation), O otherwise.
By employing behavior cloning for a base LLM agent con- struction, we establish a robust foundation for generating
4

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
valid actions with significantly improved sample efficiency. However, the base agent is easily limited by the data quality of demonstration trajectories and lacks the ability to ex- plore the environment. It could easily result in sub-optimal policies, especially when tackling complex long-horizon decision-making challenges.
The actor is trained to maximize the expected return of the policy, i.e., the estimated Q-function, which is also equivalent to maximizing the advantage function. Following the practice in AWAC (Nair et al., 2020), we formulate the policy optimization of the token actor as a weighted maximum likelihood estimation problem. The resulting loss function is derived as
3.4. Offline Hierarchical Policy Refinement
(0) = (s.))=
exp A ( -log To(u $)
To unlock the capacity of LLM agents in long-horizon decision- decision-making, we continue to train hierarchical actor- critic models in an offline mode using the reward-annotated datasets D and D'. The actor outputs a sequence of to- kens autoregressively for fine-grained action generation and control at the token level, while the critic aims to evalu- ate the output policy at the sentence level. In the follow- ing, we use and u to denote the state and action uni- formly, i.e., 8 = (d, o), u = for the high-level policy and 8 - (g.0), 0), u = a for the low-level.
= Be
exp
-Vo(s (7)
[ -
log (w; 8, 8,
This "supervised" formulation implicitly enforces a con- straint to mitigate distribution shift and avoids overly con- servative updates with advantage weighting, thus facilitat- ing efficient hierarchical policy learning from offline data, Further, by eliminating over-conservatism and explicit mod- eling of the behavior policy, it is well suited to perform fast adaptation to new tasks in online deployment, as studied in Sec. 3.5.
Sentence-Level Critic. Following the practice in advanced offiine RL algorithms (Snell et al., 2023; Zhou et al., 2024), the critic component consists of a Q-function Qa(8, u) and a value function (s) that are optimized using temporal difference learning. The Q-function is trained to minimize the Bellman bootstrapping error as
3.5. Ofllin-to-Online Adaptation
(Ø =E ( (rr (V Qs(s, Qu(l))))) (4)
In the offline stage, decompose the LLM agent into series of low-level sub-tasks (or skills) and high-level policy with strong reasoning abilities. With this flexible hierarchical structure, GLIDER can be efficiently adapted to new environments with further online interactions in offline- to-online scenarios. The low-level skills are pre-trained using intrinsic reward functions rather than task-specific ones, allowing for high generalization capacity across tasks and good robustness to the distribution shift between offline pre-traine pre-training and online deployment. Naturally, we freeze the task-agnostic low-level skills that interact with the new environment, and only fine-tune the high-level policy with the cetiroommet--roviidee reward signals.
where the Bellman target is computed from delayed copy of the value model V.. The value function is trained using
an asymmetric loss function to maintain a conservative value estimation as
Lv
E (u-n [ (Qp(s, u) (o))-VVll (+))] (5)
where L5(x) = |r 1(x < 0)(2² is an asymmetric loss function with a expectile parameter r E [0.5, 1), introduced in implicit Q-learning (Kostrikov et al., 2022). This loss function assigns more importance to Q V predictions (weighted by r) while reducing the influence of Q < V predictions (weighted by 7) The asymmetric design helps prevent the learned value function from being overly optimistic, as overestimation could casily lead to poor policy updates in offline RL settings due to distribution shift. The delayed target networks e and á are periodically updated using Polyak averaging (Haarnoja et al., 2018 () to improve training stability.
Formally, at each timestep 1, the high-level policy receives the environment observation o, together with the task de- scription d, and selects low-level skill as 9: < o,).
Then, the fixed skill 9r interacts with the environment for c primitive actions, resulting in a c-step trajectory as [gri (or, ar, Tt, 0t+1), ..., (Ot+o-1, 0t+c-1 1, Ttto- 1, ot+.)], We construct the transition sample for the high-level policy as (d; Ot: Its Er Ot+c). Finally, we collect these transition samples fine-tune the high-level critic with Eqs. (4)-(5) and the actor with Eq. (7). By harnessing the temporal abstraction knowledge embodied in the pretrained low-level skills, GLIDER can quickly adapt to non-stationary environments with significantly improved exploration efficiency.
Toven-Level Actor The LLM-based actor outputs a se- quence of tokens wi... autoregressivelly where n is the out- put sentence length. Each token is selected according to the token probability distribution generated by token actor as
[u s) = To (w ss) == =I[ To(w; W1. (6)
5

---

Divide and Conquer:
Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
4. Experiments
1) ReAct (Yao al., 2023b), pioneering approach that incorporates CoT prompting in decision-making tasks through structured Thought-Action-Obseee loop. 2) Reflexion (Shinn et al., 2023), an ad- vanced prompt-based framework that enhances agent decision-making through self-reflective verbal feedback. 3) Swi Sage (Lin et al., 2023), a dual- dual-process cognitive framework that integrates the strengths of behavior cloning and prompting for complex interactive reasoning and action- planning tasks. 4) NAT (Wang et al., 2024), fine-tuning approach that enables LLMs to learn from failure trajec- tories through data quality control. 5) ETO (Song et al., 2024), an iterative optimization framework between explor- ing the environment to collect contrastive trajectory pairs and fine-tuning the policy using DPO (Rafailov al., 2023).
We evaluate GLIDER in offline settings from Sec. 4.2 to Sec. 4.5, and then test its adaptability in an online fine- tuning manner in Sec. 4.4. Through comprehensive experi- ments, we aim to answer the following research questions:
How effective and robust is GLIDER across diverse set- tings? We examine its performance against prompt-based and fine-tuning baselines, assess consistency across dif- ferent backbones, and evaluate agent capacity in both sparse and dense reward environments. (See Sec. 4.2).
What is the contribution of each component GLIDER's s performance? Through systematic ablation studies, we analyze the impact of the hierarchical structure, training stages (SFT and ORL), and variations in model architec- ture and scale. (See Sec. 4.3).
4.2. Primary Performance
How well does GLIDER generalize to out-of-domain tasks through online fine-tuning? We evaluate the model's adaptation capabilities on previously unseen task distributions. (See Sec. 4.4).
Table illustrates the comprehensive evaluation results of GLIDER across three backbone models (Mistral-7B, Gemma-7B, and Llama-3-8B) on both ScienceWorld and ALFWorld benchmarks, compared to competent prompt- based methods (ReAct, Reflexion, and SwitchSage) and fine-tuning approaches (NAT and ETO). Generally, fine- tuning approaches yield better results than prompt-based methods, and GLIDER further exceeds these strong base- lines by a significant margin both seen and unseen tasks across diverse settings. Taking World as an example, GLIDER obtains the best performance with Llama-3-88 as the backbone, achieving high scores of 77.43 (+33.73%) on seen tasks and 68.34 (+30.59%) on unseen tasks. Simi- lar improvements are observed with Mistral-7B (+15.71 % on seen tasks, +25.63% on unseen tasks) and Gemma-77 7B (+26.23 on seen tasks, +22.28% on unseen tasks). Notably, the substantial performance gains on unseen tasks (ranging from +22.28% to +30.59%) highlight GLIDER's impressive generalization capability, which is essential for modern Al agents. In summary. the consistent performance improve- ment across diverse model architectures and benchmarks validates the effectiveness and robustness of our method.
How do varying ratios of expert demonstrations to medium-quality data affect model performance? We evaluate different mixture strategies for the composition of training data. (See Sec. 4.5).
4.1. Experimental Settings
Benchmarks and Offline Dataset. We evaluate GLIDER on two popular language-basedd interactive decision-maker tasks: 1) enceWorr r 1 (Wang et al., 2022) is textual environment for elementary science experiments, featuring 30 tasks. 2) ALFWorld (Shridhar et al., 2021) contains 6 types of household manipulation tasks, requiring agents to navigate and interact with objects following language instructions a binary reward setting.
For offline training data, we construct dataset that com- bines expert demonstrations (optimal trajectories provided by benchmarks) and medium-quality trajectories with mix- ture ratio of 2. The medium-quality trajectories are collected through two strategies: in-distribution and cross- task generalization sampling. Appendix B presents more details of benchmarks and offline dataset construction.
4.3. Ablation Studies
We conduct extensive studies to analyze the respective con- tributions of the hierarchical structure and learning stages in GLIDER, yielding five ablations: 1) w/ Hier (SFT), it ablates the offline RL stage and trains hierarchical agent with SFT only; 2) w/ Hier (ORL), it ablates the SFT stage and trains a hierarchical agent with offline RL only; 3) w/o Hier (SFT+ORL), it ablates the hierarchy and trains single-layer agent with SFT and offline RL; 4) w w/o Hier (SFT), it ablates the hierarchy and the offline RL. stage, training a single-layer agent with SFT only; 5) w/o Hier (ORL), it ablates the hierarchy and the SFT stage, training single-layer agent with offline RL only.
Models and Baselines. We build our method on three open-source language models: 1) Mistral-7B (Jiang et al., 2023), 2) Gemma-7B (Team et al., 2024) and 3) 1llam--3-88 (Meta, 2024). We employ LoRA for serameter-efficient fine-tuning of all models. Appendix C presents detailed model architectures, hyperparameters, and training and evaluation setups.
We compare GLIDER against various strong baselines:
6

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Table I: Main Results. Performance comparison across three backbone models on Science World and AlfWorld benchmarks. Q
indicates prompt-based methods without model parameter update, while O represents fine-tuning approaches using LoRA. 1 denotes the performance improvement of GLIDER compared to the best results among the baselines.
Science World
AlfWorld
Backbone
Method
Seen
Unseen
Seen
Unseen
! ReAct
20.72
17.65
7.86
5.22
! Rellexion
21.07
18.
11.56
6.00
① SwitchSage
48.40
45,25
30.29
26.52
Mistral-7B
NAT
57.12
50.79
64.43
68.90 .6
ETO
58.17
51.85
66.84
71.43
GLIDER
67.31 (1 15.71%)
65.14 (1 25 25.63%)
70.02 (1 4.76%)
74.83 (1 4.76%)
!D
ReAct
3.58
3.51
6.43
2.24
O
Reflexion
4.94
3.93
7.14
2.99
O SwitchSage
33.43
30.90
8,23
5.72
Gemma-7B
NAT
47.63
44.98
67.86
65.88
ETO
50.44
47.84
66.43
68.66
GLIDER
63.67 (1 23.23%
58.50 (1 22 28%)
|
72.12 (1 6.28%)
70.88 (t 3.23%)
O ReAct
24.76
22, 66
2.86
3.73
! Reflexion
27.23
25.41
4.29
4,48
O SwitchSage
42.22
40.58
20.39
10.78
Llama-3-8B
NAT
55.24
48.76
60.71
59.70 70
ETO
57.90
52.33
64.29
64.18
GLIDER
77.43 (1 33.73%)
68.34 (1 30.59%)
| 71.56 (1 11.31%)
75.38 (1 17.45%)
wi Hiee (5PT) Wit Hier ISFT!
wi Herr (OB))) wio Hier (DRLL)
wi Hie (SFT
lights the higher potential of reinforcement fine-tuning over supervised fine-tuning, akin to the observation in DeepSeek- RI (Guo et al., 2024). Initializing ORL from SFT parame- ters (green) proves to be also consistent to the common practice in literature (Silver et al., 2016; Song et al., 2024). Moreover, using different backbones exhibits similar patterns in these ablations, while Mistral-7B and Llama-3-8B induce better performance com- pared to Gemma-71 In summary, these results validate the effectiveness of both the hierarchical structure and the multi- stage training in GLIDER, with their combination yielding the most significant results across all implementations.
5SC
FSEE wio Hier (SIT+ORLI
70
00.34
60
53
effective strategy, which is
Performance
50
45 30
Mistral-7B
Gemma-78
Llama-3-88
Figure Ablation performance on unseen tasks in Sci- enceWorld across model architectures. Solid pillars denote hierarchical models and shaded pillars indicate ablating the hierarchy. The purple/yello/grrenn pillars correspond to SFT/ORL/SFT+-OLL training stages, respectively.
Table 2:
Ablation performance on unseen tasks in Science- World across model scales. 3
w/o Hier
w/ Hier
Model
SFT
ORI. SFT+ORI.
SFT
ORI.
SFT+ORI.
Ablation across Model Architectures. Figure 3 presents the performance on unseen tasks in ScienceWorld. We conduct these ablations across different language models to ensure robust findings. First, the hierarchical structure plays a crucial part all training stages, as models incor- porating hierarchy outperform their hon-hiirarl coun- terparts by significant margins. The improvement is most pronounced in the full stage of SFT+ORL (green), followed by the ORL stage (yellow), and finally the SFT setting (pur- plc). This interesting phenomenon highlights the superiority of our method as a whole. Another interesting observa- tion is that training offline RL agents from scratch (yellow) performs better than training SFT agents (purple). It high-
Llama-1B IB
37.24
45.31
48.48
44,50
50.43
53.62
Llama-3B
38.19
52.47
56.93
48.11
55.98
61.29
Llama-8B
41.88
50.16
53.94
50.17
57.12
68.34
Ablation across Model Scales. Further, we investigate the impact of model seales on ablation performance. Ta- ble 2 presents the ablation results on World's unseen tasks with Llama models ranging from 1B to 8B parame- ters, consistently demonstrating the advantages of the hi- erarchical structure and multi-stage training pipeline. No-
Llama-1B and Llama-3B models refer the Meta-Llama-3.2- IB-Instruct Meta--llama-3.2--1
version, respectively.
7

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
tably, our hierarchical approach demonstrates remarkable efficiency even with small parameter counts. Taking the w/ Hier (SFT+ORL)
falls between 2:1 1:5, performing the best 1:2 with a score of 68.3. An interesting phenomenon is that training with only expert demonstrations results in limited perfor- mance of score 29.7, and training solely on medium data can obtain a slightly higher score of 36.0. It suggests that increasing the rril--anderrror experience and coverage of the state-action space (expert data is somewhat homogeneous) might facilitate the generalization performance on unseen tasks. Compared to supervised learning. RL naturally learns from sub-optimal data and continually reinforcing its capa- bilities through self-evolving. This finding also supports our motivation to boost the LLM agent's 's competence via RL. In summary, both the data quality and diversity are crucial for building capable and generalizable LLM agents.
example, Llama-3B achieves score of 61.29, surpassing even larger models like Mistral- 7B with a score of 58.50. This suggests that our hierarchical structure effectively enhances agent capacity without ne- cessitating particularly large parameter counts, making our method more practical for resource-constraing scenarios.
4.4. Generalization Analysis via Online Fine-tuning
To evaluate GLIDER's generalization capacity to non- stationary environments, we test it in offline-to-onling fine- tuning scenarios with comparison to the traditional AC algo- rithm (Konda& & Tsitsiklis, 1999) and the classical offline-to- online algorithm AWAC (Nair et al., 2020). In the Science- World benchmark, we categorize the science experiment
80
w/ Hier
68. 34
tasks into three distinct domains: electrical, biology, and
Performance
60
w/o Hier
57
60.31
thermodynamics. From each group, we exclude one rep-
resentative task conductondtt,, find-animal, and boil)
40
29
66
28
37
02
41.59
38
42
6400
during offiine training, and observe the trained agent's 's adap-
25
56
19
41
20
tation performance with online fine-tuning that task. As shown in Figure 4, GLIDER exhibits strong generalization ability to new tasks in at least two aspects, First, GLIDER achieves a higher initial test score, highlighting its superior zero-shot generalization capacity and enhanced knowledge transfer to new online tasks. Second, during the online fine- tuning process, GLIDER shows significantly faster adapta- tion and achieves substantially better final performance all tasks. In summary, these results comprehensively vali- date GLIDER's s superior generalization capability to new tasks in both zero-shot knowledge transfer and fast online adaptation, establishing GLIDER as a more competent lan- guage agent with impressive autonomous adaptability.
expert
10:1
5:1
2:1
1:1
1:2
1:5
1:30
medium
Mixture rate
Figure 5: Performance on unseen tasks in ScienceWorld
with different seperttto-meduum data mixture ratios in the offlin RI. stage with Llama-3-8B as the LLM backbone.
5. Conclusions, Limitations, and Future Work
We propose GLIDER, an innovative framework that empow- ers LLM agents with high-capacity decision-making abil- ities through offline hierarchical RL. We design a concise hierarchical model architecture that achieves superior param- eter efficiency and broad upplicability, efficiently grounding L.MM agents to tackle complex, long-horizon tasks via so- phisticated exploration and structured reasoning. Extensive experiments validate GLIDER's 's consistent improvement on learning performance and generalization capability.
GLIDER(Durs)
AWAC
test:canductivity
find-animal mal
boil
1.0
1.0
0.0
0.8
0.8
0.66
0.6
60.4
0.4
0.4
0.2
0.2-
0.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.0
0.2
0.4
0.8 Stops
0.6
1.0
0.0
0.0
0/2 0.4 0.8 1. Steps
Steps
Though, our method employs a multi-stage pipeline that involves a somewhat complex training procedure. A promis- ing future work is to streamline the training pipeline while maintaining high efficiency, inspired by DeepSeek-R1 re- cent advances in reinforcement fine-tuning. Further, our framework's 's potential can extend beyond strict agent tasks, since many LLM tasks can also be reformulated as the se- quential decision -making paradigm through process reward model (PRM). A crucial future step is to extend our method to broader domains such as mathematical reasoning and code generation tasks, unleashing the inherent capability of hicrarchical agents in addressing complicated problems.
Figure 4: Online fine-tuning performance (score/100) of GLIDER against AC and AWAC baselines in ScienceWorld
4.5. Impact of Data Mixture Ratios
We investigate how different mixture ratios between expert and medium data affect agent performance during offline RL training. Figure 5 presents the GLIDER's 's performance (with and without hierarchy) on unseen tasks in Science- World across data mixture ratios. The agent achieves satis- factory capabilities when the expert-to-mediug data ratio
8

---

Divide and Conquer:
Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Acknowledgements
Dwaracherla, V., Asghari, S. M., Hao, B., and Van Roy, B. Efficient exploration for llms. In Proceedings of International Conference on Machine Learning, pp. 12215-12227, 2024.
We thank Xuyang Hu and Zhenhong Sun for their helpful discussions. This work was supported part by the National Natural Science Foundation of China (Nos. 62376122 and 72394363), in part by the Al & AI for Science Project of Nanjing University, and in part by the Nanjing University Integrated Research Platform of the Ministry of Education- Top Talents Program.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Proceed- ings of International Conference on Machine Learning, pp. 2052-2066 2019.
Gandhi, K., Sadigh, D., and Goodman, N. D. Strate- gic reasoning with language models. arXiv preprint WrXiv:2305199163 2023.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guo, D., Zhu, Q. Yang. D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y. Li, Y., et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401.1411 2024.
References
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. Do as I can, not as say: Grounding language in robotic affordances. In Proceedings of Conference on Robot Learning, 2022.
Guo, D., Yang. D., Zhang, H., Song, J., Zhang, R., et al. DeepSeek-Rl: Incentivizing reasoning capabil- ity in Ilms via reinforcement learning, arXiv preprint arXiv:2501.12945 2025.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In Proceedings of International Conference on Machine Learning, pp. 1861-1870, 2018
Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. In Proceedings of AAAI Conference Artificial Intelligence, volume 2017.
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K. Ichter, B., et al. To: A assionllagguage-attoon flow model for general robot control. arXiv preprint urXiv:2410.24164, 2024.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., et al. LoRA: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Explo- ration by random network distillation. In Proceedings of International Conference on Learning Representations, 2019.
Konda, V. and Tsitsiklis, J. Actor-critic algorithms. In Advances in Neural Information Processing Systems, vol- ume 12, pp. 75993-76005, 1999.
Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pp. 5774-5783, 2021.
Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. FireAct: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05913 2023.
Chen, W., Su, Y., Zuo, J., Yang. C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-H., Qian, C., et al. AgentVerse: Facilitating multi-agent collaboration and exploring emer- gent behaviors. In Proceedings of International Confer- ence on Learning Representations, 2024.
Kostrikov, I,, Nair, A., and Levine, S. Offline reinforcement learning with implicit Q-learning. In Proceedings of International Conference on Learning Representations, 2022.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179 1199, 1911, 2020.
Dietterich, T. G. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of Arti- ficial Intelligence Research, 13:227-303, 2000.

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Proceedings of Conference on Robot Learning, pp. 1702-1712, 2022.
Nachum, O., Gu, S. S., Lee, H., and Levine, S. Data- efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, pp. 3307-3317, 2018.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
Nair, A., Gupta, A., Dalal, M., and Levine, S. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0955 09359, 2020.
Levy, A., Konidaris, G., Platt, R., and Saenko, K. Learning multi-level hierarchies with hindsight. In Proceedings of International Conference on Learning Representations, 2019.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 17730-27744 2022.
Li, A., Florensa, C., Clavera, I., and Abbeel, P. Sub-policy adaptation for hierarchical reinforcement learning. In Proceedings of International Conference on Learning Representations, 2020.
Qiao, S., Fang, R., Zhang, N., Zhu, Y., Chen, X., Deng, S., Jiang, Y., Xie, P., Huang, F,, and Chen, H. Agent planning with world knowledge model. In Advances in Neural Information Processing Systems, volume 37, 2024.
Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L.., Chen, T., Huang, D.-A., Akyürek, E., Anandkumar, A., et al. Pre-trained language models for interactive decision- making. In Advances Neural Information Processing Systems, volume 35, pp. 31199-31211 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, volume 36, pp. 53728-53741, 2023.
Li, Z., Xie, Y., Shao, R., Chen, G., Jiang, D., and Nie, L. Optimus-1: 1: Hybrid multimodal memory empowered agents excel in long- morizooizon tasks. In Advances Neural Information Processing Systems, volume 37, 2024.
Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot reward models for reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024.
Lin, B. Y., Fu, Y., Yang, K., Brahman, F., Huang, S., Bha- gavatula, C., Ammanabrolu, P., Choi, Y., and Ren, X. SwiftSage: a generative agent with fast and slow think- ing for complex interactive tasks. In Advances in Neurul Information Processing Systems, pp. 23813-23822 2023.
Shah, D., Osiński, B., Levine, S., et al. LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action. In Proceedings of Conference on Robot Learning, pp. 492-504, 2023.
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. WizardCoder: Em- powering code large language models with evol-instruct. In Proceedings of International Conference on Learning Representations, 2024.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal rein- forcement learning. In Advances in Neural Information Processing Systems, volume 36, pp. 8634-8652, 2023.
Ma, W., Mi, Q. Zeng, Y., Yan, X., Wu, Y., Lin, R., Zhang, H., and Wang, J. Large language models play StarCraft II: Benchmarks and chain of summarization approach. In Advances in Neural Information Processing Systems, volume 37, 2024.
Shridhar, M., Yuan, X., Cote, M.-.,, Bisk, Y., Trischler, A., and Hausknecht, M. ALFWorld: Aligning text and embodied environments for interactive learning. In Pro- ceedings of International Conference on Learning Repre- sentations, 2021.
Mahankali, S., Hong, Z.. Sekhari, A., Rakhlin, A., and Agrawal, P. Random latent exploration for deep reinforce- ment learning. In Proceedings of International Confer- ence on Machine Learning, pp. 24219-34252, 2024.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, I.,, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484448) 2016.
Meta. Introducing Meta Llama 3: The most capable openly available LLM to date, 2024. https: / /ai... meta. com/blog/men /asloo//mea---ae -1amtt333
Silver, D.,. Singh, S., Precup, D., and Sutton, R. S. Reward is enough. Artificial Intelligence, 299:103535, 2021.
10

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Snell, C. V., Kostrikov, I., Su, Y., Yang, S., and Levine, S. Offline RL for natural language generation with implicit language Q-learning. In Proceedings of International Conference on Learning Representations, 2023.
Wooldridge, M. and Jennings, N. R. Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2):115-12, 1995.
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., et et The rise and potential of large language model based agents: A survey. arXiv preprini arXiv:2309.07880 07.044744 2023.
Song, Y., Yin, D., Yue, X., Huang, J., Li, S., and Lin, B. Y. Trial and error: Exploration- based trajectory optimization for Ilm agents. In Proceedings of Annual Meeting of the Association for Computational Linguistics, 2024.
Xu, Z., Yu, C., Fang, F., Wang. Y., and Wu, Y. Language agents with reinforcement learning for strategic play in the werewolf game. In Proceedings of International Con- ference on Machine Learning, pp. 55434-55464, 2024.
Sutton, S., Precup. D., and Singh, S. Between mdps
and semi-mdps: A framework for temporal abstraction in reinforcement learning, Artificial intelligence, 112(1-2); 181-211, 1999.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pp. 11809 110--18822,, 2023a.
Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, A. T. Large language models as generalizable policies for embodied tasks. In Proceedings of International Confer- ence on Learning Representations, 2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In Proceedings of International Conference on Learning Representations, 2023b.
Tan, W., Zhang, W., Liu, S., Zheng, I.., Wang, X., and An, B. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In Proceedings of International Conference on Learning Representations, 2024.
Yu, Z. and Zhang, X. Actor-critic alignment for offline- to-online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 40452 40474, 2023.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, I.,, Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., and Tang. J. AgentTuning: Enabling generalized agent abilities for LLMs. arXiv preprint arXiv:2310.12223, 2023.
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.W.,, and Lim, E.-P Plan-and-solve prompting: Improving zero-shot chain-of-hee of-thought reasoning by large language models. In Proceedings of Annual Meeting of the Associ- ation for Computational Linguistics, pp. 9-14, 2023a.
Zhai, Y., Bai, H., Lin, Z., Pan, J.. Tong. S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al. Fine-tuning large vision-language models as decision-making agents via re- inforcement learning. In Advances in Neural Information Processing Systems, volume 37, 2024.
Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P. GedenceWWrlde Is your agent smarter than a 5th grader? In Proceedings of Empirical Methods in Natural Language Processing, pp. 11279-11299 1298, 2022.
Zhou, D., Schärli, N., Hou, L.., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of International Conference on Learning Representations, 2023.
Wang, R., Li, H., Han, X., Zhang, Y., and Baldwin, T. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint ar Xiv:22402. 1651, 2024.
Wang, S., Yang. Q.. Gao, J., Lin, M., Chen, H., Wu, L., Jia, N., Song, S., and Huang, G. Train once, get a family: State-adaptive balances for offlin-to-online reinforce- ment learning. In Advances in Neural Information Pro- cessing Systems, volume 36, pp. 47081 47081-47111 104, 2023b.
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. ArCHer: Training language model agents via hierarchical multi-turn rl. In Proceedings of International Conference on Machine Learning, pp. 62178-62209, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thougle prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 24824 24837, 2022.

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Appendix A. Algorithm Pseudocodes
Based on the implementations in Section we summarize the brief procedure of GLIDER. Algorithm presents the complete training pipeline of GLIDER, which consists of three stages. In SFT stage, we perform behavioral cloning train both high-level and low-level policies using demonstration data. Notably, both policies share the same LLM parameters but differentiated through distinet prompts, which significantly reduces the parameter count while maintaining the hierarchical structure. The high-level policy prompt focuses task decomposition, while the low-level policy prompt emphasizes primitive action generation. In ORL Stage, where both policies and critics are updated using data from high-level (B+) and low-level (Bc) replay buffers, where contains a balanced mixture of expert demonstrations and medium-quality trajectories. Critics are updated through bootstrapping, while policies are optimized via a policy gradient. O20 stage describes the optional online adaptation process. Due to parameter sharing between policies, we cannot strictly fix the low- level policy parameters. Instead, we maintain low-level policy performance by continually training on offline demonstration data while simultaneously fine-tuning the high-level policy using newly collected transition data.
Algorithm I: GLIDER: A Hierarchical Framework for LLM-based Decision Making
Input:
High-level replay buffer: Bx (d, ou 90, Ru, Oc... OU, 9t, R2, O(te) where R2 E l+c-l
1=t T3b
Low-level replay buffer: Bc = [9t, Ot, Oz, ft,, 0++1, ... 0t 01 Cttc- Otte) (60}
Environment: env
Hierarchical policy: Tg.
Hierarchical critic: Q V Q' V
Hyperparameters: discount y, update rate т, regularization weight À
Output: Optimized hicrarchical policy = {TT. m
Stage 1: SFT
for iteration = 1, 2...... do
Update hierarchical policy via BC loss (Eq.3,Eq.6)
end
// Stage 2: ORL for iteration i = 1, do
Sample batches from By and B&
Update critics via bootstrapping (Eq.44 Eq.5)
Update policies via policy gradient (Eq.7)
Soft update target networks: +- (1 7 njj + TM
10
11 end
12 // Stage 3: 020 (Optional) Fix low-level policy Tf for episode = 1,2,... do
13
14
Og + env.rese((task), +- env.eestee enw.tuskk.escrrptiion)
15
Initialize trajectory for ( 1,.., do
16
Sample subtask: Ir (. d.) for step h = l....,co do
Sample action: a, Ty(. gr,0,) 0r)
29
Tt, 01,1 done + env.step(a.)
end
22
Store transition: & + & (d, oz, St. o cc)
23
/-/+c
24
end
Store trajectory: Bx t- BH U{}
25
26
Update high-level policy and critic using By (Eq.4-7)
27
end
12

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Appendix B. Dataset Information
Benchmarks.
We evaluate GLIDER on two popular language- interactive decision decision-making tasks:
1. ScienceWorld (Wang et al., 2022) is a textual environment for elementary science experiments, featuring 30 tasks across 10 categories. Agents must demonstrate scientific understanding through interactive experimentation, with progress measured by dense reward (0 1) at each step.
2. ALFWorld (Shridhar al., 2021) simulates household environments that require navigation and object manipulation in sparse, binary reward setting. The reward is only upon successful task completion, and 0 otherwise.
Beyond standard evaluation on seen tasks, it includes unseen scenarios assess generalization ability, Table 3 presents the statistical information of our datasets. Both Science and ALFWorld contain Text-Seen and Text-Unseen test sets, where Text-Unseen comprises out-of-distribution variations to evaluate the generalization capabilities of different agents.
Table 3: Dataset statistics.
Dataset
Train
Text-Seen
Text-Unseen
ScienceWorf 1,483
194
211
ALFWorld
3,119
140
134
Expert Demonstration
To support imitation learning, the ScienceWorld and ALFWorld provide suman-annotated trajectories. For hierarchical data structuring, we utilize GPT-4 to decompose these trajectories into subtasks, creating a clear hierarchical representation of the demonstration data. We show an example expert demonstration trajectory for w/o and w/ hierarchical ScienceWorld in Figure.
Medium Data Collection
For medium-quality data collection, we employ two distinct sampling strategies:
1. In-distribution Sampling: During the SFT training process, we utilize the intermediate policy to sample sub-optimal trajectories on the training tasks. This sampling strategy helps better cover the task's world model, as the intermediate policy explores diverse solution paths and state transitions, leading to a more comprehensive understanding of the environment dynamics and task structure.
2. Cross-task Generalization Sampling: We partition the training tasks into (we set 3) subsets. For each iteration, train an SFT model on one subset and use it to collect data on the unseen training tasks from other subsets. This sampling strategy is specifically designed to capture trajectories that exhibit better generalization capabilities, as the policy is forced to adapt to novel but related tasks during data collection.
Both of strategies approach ensures diversity in our medium -ucluutttuatttyya dataset, combining both task-specific expertise and cross-task generalization abilities. We provide examples of both strategies with and without hicrarchical structure in Figure 7 and Figure& 8, respectively.
In Figure 7 (in-distribution sampling), we can see the the exploration without hierarchy terminates early after attempting to "pick up apple juice". In contrast, the hierarchical approach extends further through clear subtask planning ("Prepare tools", "Find apple juice", "Heat apple juice"), continuing until it discovers the stove malfunction.
In Figure 8 (cross-task generalization sampling) demonstrates enhanced exploration capabilities. Under the same initial conditions, the non-hierarchical approach attempts to "fix the stove", while the hierarchical approach not only identifies the stove malfunction but actively seeks alternative solutions ("Find other stove to heat apple juice "),, showing more sophisticated problem-solving strategies.
13

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Expert Demonstration
w/o Hierarchy
w/ Hierarchy
Task Description: Your task to find a(n) animal. First, focus on the thing. Then, move it to the red box in the kitchen.
Task Description:
Your task is to find a(n) animal. First, focus on the thing. Then, move it to the red box in the kitchen.
Observation: This room is called the hallway. In it, see: the agent, substance called air, picture. You also see: A door to the art studio (that is closed), door to the bedroom (that is closed), A door to the greenhouse (that is closed), A door to the kitchen (that closed), door to the living room (that is closed), A door the workshop (that is closed).
Observation: This room is called the hallway. In it, you see: agent, a substance called air, picture. You also see: A door to the art studio (that is closed), A door to the bedroom (that closed), door the greenhouse (that is closed), A door to the kitchen (that is closed), A door to the living room (that is closed), A door to the workshop (that is closed).
Action: open door to outside
Subtask: Find an animal and focus it
Reward: 0.09
Action: open door to outside
Observation: The door is now open.
Observation: The door now open.
Action: go to outside
Action: go to outside
Reward: 0.08
Observation: You move the outside.
Observation: You move the outside.
Action: look around
Action: look around
Observation: This outside location is called the outside. Here you see: the agent, a substance called air, an axe, a crocodile egg, a fire pit (containing nothing), a foun- tain (containing a substance called water), the ground, turtle egg, baby wolf, a substance called wood. You also see: A door the foundry (that closed), door to the greenhouse (that closed), A door to the kitchen (that is open)
Reward: 0.0
Observation: This outside location is called the outside. Here see: the agent, substance called air, an axe, a crocodile egg, a fire pit (containing nothing), a fountain (containing a substance called water), the ground, a turtle egg, a baby wolf, substance called wood. You also see: door the foundry (that is closed), A door the greenhouse (that is closed), A door to the kitchen (that is open)
Action: focus on egg turtle
High Reward: 0.75; Low Reward: 1.0
Action: focus on egg turtle
Observation: You focus on the turtle egg.
Reward: 0.5
Subtask: Navigation to kitchen with the animal
Observation: You focus on the turtle egg.
Action: pick up egg turtle
Action: pick up egg turtle
Observation: You move the turtle to the inventory.
Reward: 0.08
Action: open door to kitchen
Observation: You move the turtle the inventory.
Observation: The door is already open.
Action: open door to kitchen
Action: go to kitchen
Reward: 0.0
High Reward: 0.08; Low Reward: 1.0
Observation: The door is already open. Action: go to kitchen
Observation: You move to the kitchen.
Subtask: Move the animal to the red box in the kitchen
Reward: 0.0
Action: move egg turtle egg in inventory red box
Observation: You move the kitchen.
High Reward: 0.17: Low Reward: 1.0
Action: move egg turtle egg in inventory red box
Observation: You move the turtle to the red box. done.
Reward: 0.17
Observation: You move the turtle to the red box. done.
Figure 6: Expert demonstration for w/o and w/ hicrarchical structure in Science World.
14

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Medium Trajectory Collected by In-distribution Sampling Strategy
w/o Hierarchy
w/ Hierarchy
Task Description: Your task to boil apple juice. For compounds without boiling point, combusting the sub- stance is also acceptable. First, focus the substance. Then, take actions that will cause it to change its state of matter.
Task Description: Your task is to boil apple juice. For compounds without a boiling point, combusting the sub- stance is also acceptable. First, focus the substance. Then, take actions that will cause it to change its state of matter.
Observation: This room is called the bathroom. In it, you see: the agent, substance called air, a bathtub, which is turned off. In the bathtub nothing. glass cup (containing nothing), picture, a sink, which is turned off. In the sink is: nothing. toilet. In the toilet is: A drain, which is open, substance called water. You also see: A door to the kitchen (that is closed). Action: open door to kitchen
Observation:
This room called the bathroom. In it, you see: the agent, substance called air, a bathtub, which is turned off. In the bathtub is: nothing. a glass cup (containing nothing), picture, sink, which is turned off. In the sink is: nothing. toilet. In the toilet is: A drain, which is open, substance called water. You also see: A door to the kitchen (that is closed).
Subtask: Navigate to kitchen
Reward: 0.0
Action: open door to kitchen
Observation:
Observation: The door is now open.
Action: go kitchen
Action: go to kitchen
Reward: 0.03
High Reward: 0.03; Low Reward: 1.0
Observation: You move to the kitchen.
Observation: You move to the kitchen.
Action: look around
Subtask: Prepare tools for measuring temperature and boiling
Reward: 0.0
Observation: This room is called the kitchen. In it, you see: counter. On the counter is: a bowl (containing a red apple, banana, an orange, potato), a drawer. a cupboard. The cupboard door is closed. freezer. The freezer door is closed. fridge. The fridge door is closed. glass jar (containing substance called sodium chloride) lighter a oven, which is turned off. The oven door is closed. a painting sink, which is turned off. In the sink is: nothing. a substance called soap a stopwatch, which is deactivated. a stove, which is turned off. On the stove is: nothing. a table. On the table is: glass cup (containing nothing). ther- mometer, currently reading temperature of 10 degrees celsius You also see: A door to the bathroom (that is open) A door to the hallway (that is closed) A door to the outside (that is closed).
Action: pick up thermometer
Observation: You move the thermometer to the inven-
tory.
Action: pick up metal pot
High Reward: 0.0; Low Reward: 1.0
Observation: You move the metal pot to the inventory.
Subtask: Find apple juice and focus on it
Action: open cupboard
Observation: The cupboard now open. Action: focus on apple juice
High Reward: 0.0; Low Reward: 0.0
Observation: No known action matches that input.
Subtask: Find apple juice and focus on it Action: open fridge
Observation: The fridge is now open.
Action: focus on apple juice
Action: pick up thermometer
High Reward: 0.67; Low Reward: 1.0
Reward: 0.0
Observation: You focus on the apple juice.
Observation: You move the thermometer the inven- tory.
Subtask: Heat apple juice with stove
Action: move cup containing apple juice stove
Action: pick up metal pot
Observation: You move the glass cup to the stove.
Reward: 0.0
Action: activate stove
Observation: You move the metal pot to the inventory.
High Reward: 0.02; Low Reward: 0.0
Action: pick up apple juice
Observation: The stove is broken.
Reward: 0.0
Subtask: Can't heat apple juice because stove is broken
Observation: No known action matches that input. not done.
Action: action generated
High Reward: 0.0; Low Reward: 0.0
Observation: No known action matches that input. not done.
Figure 7: Comparison of w/o and w/ hierarchical trajectories collected through in-distribution sampling in Science World.
15

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Medium Trajectory Collected by Cross-task Generalization Sampling Strategy
w/o Hierarchy
w/ Hierarchy
Task Description: Your task to boil apple juice. For compounds without boiling point, combusting the sub- stance is also acceptable. First, focus the substance. Then, take actions that will cause it to change its state of matter.
Task Description: Your task is to boil apple juice. For compounds without a boiling point, combusting the sub- stance is also acceptable. First, focus the substance. Then, take actions that will cause it to change its state of matter.
Observation: This room is called the bathroom. In it, you see: the agent, substance called air, a bathtub, which is turned off. In the bathtub nothing. glass cup (containing nothing), picture, a sink, which is turned off. In the sink is: nothing. toilet. In the toilet is: A drain, which is open, substance called water. You also see: A door to the kitchen (that is closed). Action: open door to kitchen
Observation:
This room called the bathroom. In it, you see: the agent, substance called air, a bathtub, which is turned off. In the bathtub is: nothing. a glass cup (containing nothing), picture, sink, which is turned off. In the sink is: nothing. toilet. In the toilet is: A drain, which is open, substance called water. You also see: A door to the kitchen (that is closed).
Subtask: Navigate to kitchen
Reward: 0.0
Action: open door to kitchen
Observation:
Observation: The door is now open.
Action: go kitchen
Action: go to kitchen
Reward: 0.03
High Reward: 0.03; Low Reward: 1.0
Observation: You move to the kitchen.
Observation: You move to the kitchen.
Action: pick up thermometer
Subtask: Prepare tools for measuring temperature and boiling
Reward: 0.0
Observation: You move the thermometer to the inven-
Action: pick up thermometer
tory.
Observation: You move the thermometer to the inven-
Action: pick up metal pot
tory.
Reward: 0.0
Action: pick up metal pot
Observation: You move the metal pot the inventory.
High Reward: 0.0; Low Reward: 1.0
Action: open fridge
Observation: You move the metal pot to the inventory.
Reward: 0.0
Subtask: Find apple juice and focus on it
Observation: The fridge now open.
Action: open cupboard
Action: focus on juice
Observation: The cupboard now open.
Reward: 0.67
Action: focus on apple juice
Observation: You focus on the apple juice.
High Reward: 0.0; Low Reward: 0.0
Action: move cup containing apple juice to stove
Observation: No known action matches that input.
Reward: 0.02
Subtask: Find apple juice and focus on it
Observation: You move the glass cup to the stove.
Action: open fridge
Action: activate stove
Observation: The fridge is now open.
Reward: 0.0
Action: focus on apple juice
Observation: The stove broken.
High Reward: 0.67; Low Reward: 1.0
Action: fix the stove
Observation: You focus on the apple juice.
Reward: 0.0
Subtask: Heat apple juice with stove
Observation: No known action matches that input not done.
Action: move cup containing apple juice stove
Observation: You move the glass cup to the stove. Action: activate stove
High Reward: 0.0; Low Reward: 0.0
Observation: The stove is broken.
Subtask: Find other stove heat apple juice
Action: focus on stove
High Reward: 0.0; Low Reward: 0.0
Observation: No known action matches that input. not done.
Figure 8: Comparison of w/o and w/ hierarchical trajectories collected by cross-task generalization sampling in Science World.
16

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Data Structure and Setups
The high-level and low-level training dataset structure follows sequential format that captures the complete interaction trajectory:
Training Dataset Structure
High-Level Trajectory:
{high prompt, task description, obs 0, subtask high reward obs T-1, subtask T-1, high reward T-1, obs T)
Low-Level Trajectory:
[Low prompt, subtask O, obs 0, action 0, low reward obs c-1, action c-1, low reward c-1, obs c}
020 task Setups
To evaluate the generalization capabilities of our method, we construct an 020 (Online-to-Offine) dataset covering three distinct domains: electrical, biology, and thermodynamics. Each domain contains one test task and multiple train tasks, as shown in Table 4. For the electrical domain, we have "tesstccoduuctii ivity" as the test task, along with three train tasks related to conductivity testing and power components. The biology domain features " "find-annine task, accompanied by ten train tasks involving various biological concepts such as living/non-living identification, plant growth, and lifespan studies. In the thermodynamics domain, "boil" " serves as the test task, supported by seven train tasks covering different aspects of state changes and chemical mixing processes. This setup provides rigorous test of the agent's ability to transfer knowledge from trained tasks novel but related domains in the Science environment.
as the test
Table 4: Distribution of and train tasks across electrical, biology, and thermodynamics domains
Domain
Task Name
Тype
test-conductivity
Test
Train
Electrical
power-component
Train
Train
find-animal
Test
find-living-thing
Train
fnn-non- hid-noo-livinge -ivinn-ttiing
Train
find-plant
Train
grow-fruit
Train
Biology
grow-plant
Train
Seentify-life-staeees
Train
identify-lifee-staess22
Train
lifespan-loonges-lived
Train
Train
lifespan-shortest-lived
Train
boil
Test
freeze
Train
melt
Train
Train
Thermodynamics
chemistry-mix
Train
chemistry-mix wwwtiiipanittecoodarrcooooomo iitiiiiiaaattttrr--collllll
Train
Train
use-thermometer
Train
17

---

Divide and Conquer:
Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Reward Design
Science The agent receives dense reward ranging from 0 to 1 at each step, reflecting the continuous progress in scientific experimentation tasks across 30 scenarios in 10 categories.
ALFWorld:
The agent receives sparse binary reward (0 or 1), where is given only upon successful completion of household navigation and manipulation tasks, and 0 otherwise.
The high-level policy accumulates environmental rewards upon the completion of subtasks by the low-level policy, reflecting the agent's 's progress in achieving the overall objective.
High-level Reward:
Low-level Reward: The low-level policy receives binary rewards from the high-level policy, indicating whether subtask is successfully completed not.
18

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Appendix C. Training and Evaluation setups
Models
We build our method on three open-source language models: 1) Mistral1778 (Jiang et al., 2023), the Histral-7B-Instrucee v0.2 version. 2) Gemma-7B (Team al., 2024), the Gemma- 1ermaa--.11770 1.-tt version. 3) L1ama-3-88 (Meta, 2024), the La----aam------nstrree version. We employ LoRA for parameter-efficient fine-tuning of all language models.
Baselines
We compare GLIDER against various strong baselines: 1) ReAct (Yao al., 2023b), a pioneering approach that incorporates CoT prompting decision-makinn tasks through a structured Soaugght-Attion-bbervaaa loop. Reflexion (Shinn al., 2023), advanced prompt-based framework that enhances agent decision-making through self-reflective verbal feedback. 3) Swi Sttaaaaaeeeeee (Lin et al., 2023), dual-process theory-based cognitive framework that integrates the strengths of behavior cloning and prompting for complex interactive reasoning and action-plas eetionnplanning tasks. 4) NAT (Wang et al., 2024), a fine-tuning approach that enables LLMs to learn from failure trajectories through quality control and fine-tuning strategies. 5) ETO (Song et al., 2024), an iterative optimization framework between exploring the environment to collect contrastive trajectory pairs and fine-tuning the LLM policy using DPO (Rafailov et al., 2023).
Hyperparameter
we employ LoRA for parameter-efficien fine-tuning for all language models. During the SFT phase, we train for 5 epochs with batch size of 32, using the AdamW optimizer with learning rate of 4. The detailed hyperparameters are shown the following table Policy are evaluated on both seen and unseen tasks across two benchmarks. For the ORL phase, we train for epochs with different learning rates for the actor (le 5) and critic (le 4) networks. The target critic network is updated using soft updates with 0.2, and we set the advantage weighted factor À to 0.99. The training data consists of expert and medium data in a 1:2 ratio.
Table 5: Hyperparameters used for GLIDER.
Hyperparameter
Value
Hyperparameter
Value
batch size
64
temperature
0.7
batch size per device
2
advantage weighted factor À
0.99
gradient accumulation steps
8
soft update
0.2
actor learning rate
1 10
discount factor
0.99
critic learning rate
10-4
sft epochs
5
sft learning rate
10-4
orl epochs
4
lora
16
data mixture radio
1:2
lora alpha
32
warmup ratio
0.03
lora dropout
0.05
tokens
32
19

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Appendix D. Prompts and Case Study
illustrate the prompts used in our paper, where High-Level and Low-Level prompts establish hierarchical control between different agents, while Check Subtask Complete Prompt enables the high-level agent evaluate subtask completion by the low-Ievel agent.
Prompts
High-Levvel Prompt:
You are a high-level planner. Based on the state (task description, group action and current observation), please generate a clear and simple subtask.
Low-Level Prompt:
You are a low-level action executor. Based on the current subtask and observation, please generate a executable action and determine if the subtask is completed (true/false).
Check Subtask Complete Prompt:
Determine if the low-level actions successfully completed the given subtask by high-level:
Subtask: [subtask]
Initial observation: [initial obs]
Actions: (action sequence]
Final observation: [final obs]
Output only single digit:
True if the actions successfully completed the subtask False if the actions failed to complete the subtask Give the "True" or "False":
To demonstrate the advantages of hierarchical decomposition, we present a case study show in Figure. comparing two different tasks: freezing apple juice and boiling water. The highlighted portions reveal two types structural similarities: identical subtasks (highlighted in yellow) that can directly reused, and analogous subtasks (highlighted in red) that share similar underlying patterns despite different objectives. For example, both tasks contain the identi- cal subtask "Prepare temperature and metal pot." while "Monitor apple juice until frozen " and "Monitor water until boiling" represent analogous patterns of state monitoring and waiting. This hierar- chical approach not only enables direct subtask reuse, but also allows the planner to recognize and adapt similar strategic patterns across different tasks, thereby reducing the planning complexity and improving efficiency.
20

---

Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
Case Study
freeze apple juice
boil water
Task Description: Your task is melt ice. First, focus on the substance. Then, take actions that will cause change its state of matter.
Task Description: Your task is to boil water. For com- pounds without boiling point, combusting the substance is acceptable. First, focus on the substance. Then, take actions that will cause it to change its state of matter. Observation: This room is called the kitchen. In it, you see: a counter. On the counter a bowl (containing red apple, banana, orange. potato), drawer. cupboard. The cupboard door is closed. freezer. The freezer door is closed. fridge. The fridge door is closed.
Observation: This room is called the greenhouse. In it, you see: the agent, substance called air, bee hive. The bee hive door is closed, jug (containing nothing), a shovel, sink, which is turned off.
Subtask: Navigate to kitchen Action: open door hallway Ohservation: The door is open. Action: go to hallway
Subtask: Prepare temperature and metal pot
Observation: You move to the hallway.
Action: pick up thermometer
Action: open door to kitchen
Observation: You move thermometer to inventory.
Observation: You move to the kitchen. Subtask: Prepare temperature and metal pot
Action: open cupboard
Observation: The cupboard is now open.
Action: pick up thermometer
Action: pick up metal pot
Observation: You move thermometer inventory.
Observation: You move metal pot to inventory.
Action: open cupboard
Subtask: Fill metal pot with water and focus on
Observation: The cupboard is now open.
Action: move metal pot to sink
Action: pick metal pot
Observation: You move the metal pot to the sink.
Observation: You move metal pot to inventory.
Action: activate sink
Subtask: Find apple juice then focus it
Observation: The sink is activated.
Action: open fridge
Action: deactivate sink
Observation: The fridge now open.
Observation: The sink is now deactivated.
Action: focus apple juice
Action: focus on substance in metal pot
Observation: You focus on the juice.
Observation: You focus on the water.
Subtask: Move apple juice to metal pot then freeze it
Subtask: Heat water with stove
Action: pour cup containing apple juice into metal pot Observation: You pour the apple juice into metal pot.
Action: move metal pot to stove
Observation: You move the metal pot the stove.
Action: pick up metal pot
Action: activate stove
Observation: You move the metal pot the inventory.
Observation: The stove is now activated.
Action: open freezer
Subtask: Monitor water until boiling
Observation: The freezer is open.
Action: examine substance in metal pot
Action: move metal pot to freezer
Observation: a substance called water
Observation: You move the metal pot to the freezer.
Action: use thermometer in inventory on substance in metal pot
Subtask: Monitor apple juice until frozen
Action: examine apple juice
Observation: The temperature is 68 degrees celsius
Observation: orange juice
Action: wait2
Action: thermometer in inventory apple juice
Observation: You decide to wait for 20 iterations.
Observation: The temperature is 2 degrees celsius
Action: examine substance in metal pot Observation: substance called steam.
Action: wait
Observation: You decide to wait for 10 iterations.
done.
Action: examine orange juice Observation: solid orange juice done.
Figure 9: Hierarchical decomposition reveals shared subtask patterns across two example tasks Science World.
21


Reviews:
[{"review_id": "5lAmgHA0tC", "reviewer": "Reviewer_DNb1", "summary": "This paper addresses the challenges of using Large Language Models (LLMs) for long-horizon decision-making tasks, specifically their difficulties with exploration and long-term credit assignment, especially in sparse-reward settings. To mitigate these challenges, the authors propose a framework called GLIDER (Grounding Language Models as Efficient Decision-Making Agents via Offline HiErarchical Reinforcement Learning). GLIDER introduces a parameter-efficient hierarchical structure to LLM policies. The framework employs a scheme where a low-level controller is supervised by abstract, step-by-step plans that are learned and instructed by a high-level policy. This hierarchical design aims to decompose complex problems into a sequence of chain-of-thought reasoning sub-tasks, thereby providing temporal abstraction to enhance exploration and learning in long-horizon tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your constructive comments for improving our paper. We would be incredibly grateful for your continued support in reviewing our response and offering further encouragement.\n\n***\n\n`Q1. What are the advantages of GLIDER compared to direct LLM planning approaches?`\n\nA1. We have conducted thorough ablation studies on hierarchical structure. As shown the following Table, hierarchical models consistently outperform their non-hierarchical counterparts across all LLM backbones, demonstrating the necessity of our approach.\n\n| Backbone | w/o Hier | **w/ Hier** |\n| --- | --- | --- |\n| Mistral-7B | 47.30 | 65.14  |\n| Gemma-7B | 50.16 | 58.50 |\n| Llama-3-8B | 53.94 | 68.34 |\n\n***\n\n`Q2. What is the parameter efficiency of GLIDER and how does it compare with baseline methods in terms of parameter count?`\n\nA2. Based on our additional experimental evaluation on ScienceWorld using LLaMA3-8B as the base model, GLIDER demonstrates parameter efficiency in two key aspects: (1) With LoRA, it requires only 0.18GB trainable parameters while achieving superior performance over baselines ETO and NAT; (2) GLIDER with LoRA (0.58\\% parameters) achieves comparable performance to full fine-tuning (99.40\\% parameters), showcasing effective parameter utilization through prompting-based hierarchical control.\n\n| Method | Trainable parameters | Total parameters | Percentage | Performance |\n| --- | --- | --- | --- | --- |\n| **GLIDER(LoRA)** | 0.18 GB | 30.22 GB | 0.58% | 68.34 |\n| **GLIDER(Full)** | 30.04 GB | 30.22GB | 99.40% | 70.50 |\n| ETO | 0.05 GB | 29.97GB | 0.17% | 52.33 |\n| NAT | 0.05 GB | 29.97GB | 0.17% | 48.76 |\n\n***\n\n`Q3. Why the benchmarks are appropirate?`\n\nA3. ScienceWorld and ALFWorld are ideal benchmarks for evaluating GLIDER,  featuring dense and sparse reward structures respectively. The complex, non-deterministic nature of both environments requires the agent to continuously adjust its strategy rather than pre-computing a complete solution path, validating GLIDER's real-time planning abilities.\n\n***\n\n`Q4. How does GLIDER handle sparse-reward scenarios?`\n\nA4. GLIDER addresses sparse-reward challenges through its hierarchical structure. High-level planning breaks down complex goals into manageable subgoals, while low-level execution ensures efficient exploration and clearer credit assignment for each subtask. This is validated by our strong performance compare to baseline in AlfWorld,  a sparse-reward benchmark.\n\n| Method | AlfWorld(Seen) | AlfWorld(Unseen) |\n| --- | --- | --- |\n| NAT | 60.71 | 59.70 |\n| ETO | 64.29 | 64.18 |\n| **GLIDER** | **71.56**($\\uparrow$ 11.31%) | **75.38**($\\uparrow$  17.45%) |\n\n***\n\n`Q5. Grammar and Clarification of Terminology`\n\nA5. We appreciate the reviewer's suggestions and will thoroughly proofread the manuscript. Regarding the \"long-horizon\" task, it refers to tasks that cannot be fully planned at the beginning but require continuous planning throughout a long sequence of steps. For example, when \"boiling water\", if the sink breaks, the agent must replan to find alternative water sources.\n\n***\n\nBest regards,\n\nAuthors", "claims_and_evidence": "Claim: LLMs struggle with long-horizon decision-making tasks due to deficient exploration and long-term credit assignment, especially in sparse-reward scenarios.\nEvidence: The paper cites the challenges of exploration and credit assignment as motivation for their approach. The introduction sets up this problem as a key issue in applying LLMs to decision-making. The experiments likely demonstrate improved performance in long-horizon, sparse-reward tasks compared to baseline methods, further supporting this claim.   \nClaim: The GLIDER framework, which introduces a parameter-efficient hierarchical structure to LLM policies, mitigates these challenges.\nEvidence: The paper details the GLIDER framework, including the hierarchical structure with high-level and low-level policies. The experimental results, particularly comparisons with baseline methods, demonstrate the effectiveness of GLIDER in improving performance on long-horizon tasks. Ablation studies, if present, would further strengthen this claim by showing the contribution of the hierarchical structure.   \nClaim: The hierarchical design decomposes complex problems into a series of coherent chain-of-thought reasoning sub-tasks, providing flexible temporal abstraction to enhance exploration and learning.\nEvidence: The paper explains how GLIDER learns and uses abstract, step-by-step plans to supervise the low-level controller. The results should show improved performance and exploration, suggesting that the decomposition into sub-tasks is beneficial. Visualizations or analyses of the learned sub-tasks and how they contribute to the overall solution would provide further support. Figure 9, for example, illustrates the hierarchical decomposition.", "methods_and_evaluation_criteria": "Methods:\n\nHierarchical Reinforcement Learning: The decomposition of the problem into high-level planning and low-level control is a logical approach for long-horizon tasks. It allows the high-level policy to focus on strategic planning while the low-level policy executes the plan, which aligns with the \"divide-and-conquer\" principle mentioned in the paper.   \nLanguage Models for Decision-Making: Utilizing LLMs to generate and interpret plans is novel and leverages the reasoning capabilities of LLMs. The integration of LLMs within the reinforcement learning framework seems appropriate for tasks that require complex reasoning and planning.\nParameter-Efficient Hierarchy: The emphasis on parameter efficiency is important for scaling the method and making it practical.\n\nEvaluation Criteria:\n\nThe evaluation criteria and benchmark datasets seem reasonable for assessing the effectiveness of GLIDER.\n\nLong-Horizon Decision-Making Tasks: Evaluating the method on tasks that require long-term planning and execution is essential for demonstrating its ability to handle the challenges it aims to address. The paper should define these tasks clearly and explain why they are appropriate benchmarks.\nSparse-Reward Scenarios: Evaluating the method in sparse-reward settings is critical, as this is where exploration and credit assignment are most challenging. The evaluation should demonstrate that GLIDER outperforms other methods in these scenarios.\nComparison with Baseline Methods: Comparing GLIDER with appropriate baseline methods, including other reinforcement learning algorithms and potentially LLM-based approaches, is crucial for demonstrating its superiority. The choice of baseline methods should be justified.\nMetrics: The paper should employ appropriate metrics to evaluate performance, such as success rate, return, or other task-specific metrics. These metrics should be clearly defined and relevant to the evaluation goals.\nAblation Studies: If included, ablation studies can provide valuable insights into the contribution of different components of the GLIDER framework. This helps to understand the importance of the hierarchical structure, the planning mechanism, and other design choices.", "theoretical_claims": "The paper focuses primarily on an empirical approach, presenting a novel framework (GLIDER) and demonstrating its effectiveness through experiments. While the core contribution is algorithmic and empirical, there are some implicit theoretical claims.\n\nThe paper does not contain formal mathematical proofs for theoretical claims. The emphasis is on the empirical validation of the GLIDER framework. This is not necessarily a weakness, as many papers in machine learning focus on empirical contributions. However, it's important to recognize that the theoretical underpinnings are largely based on established principles and demonstrated through experiments rather than formal proofs.\n\nAreas for Potential Improvement:\n\nWhile not strictly required, the paper could benefit from a more detailed discussion of the theoretical implications of using LLMs in hierarchical RL. This could involve relating the approach to existing theoretical results in hierarchical RL or discussing the limitations and potential failure modes of the approach from a theoretical perspective.", "experimental_designs_or_analyses": "The experimental design appears to be sound and generally supports the claims made in the paper. The authors have attempted to evaluate GLIDER in a comprehensive manner, considering various aspects of its performance.", "relation_to_broader_scientific_literature": "the paper innovatively combines the strengths of LLMs (reasoning, language understanding) with the structural advantages of hierarchical reinforcement learning to improve long-horizon decision-making. It builds upon existing research in LLMs, hierarchical RL, and planning, but it proposes a novel integration of these areas.", "essential_references_not_discussed": "By discussing these related works, the paper can provide a more comprehensive context for its contributions and better position GLIDER within the broader scientific literature.", "other_strengths_and_weaknesses": "Strengths:\n\nOriginality: The paper presents a novel approach by effectively combining LLMs and hierarchical reinforcement learning for long-horizon decision-making. The way it leverages LLMs to generate and follow abstract plans within a hierarchical framework is a creative combination of existing ideas.\nSignificance: The paper addresses a significant challenge in applying LLMs to complex, sequential tasks. Overcoming the limitations of LLMs in exploration and credit assignment has the potential to broaden the applicability of LLMs in various domains, including robotics, game playing, and automation.\nClarity: The paper is generally well-written and easy to follow. The authors clearly explain the GLIDER framework and its components. The figures and examples, such as Figure 9, aid in understanding the proposed approach.\nWeaknesses:\n\nLimited Theoretical Analysis: As mentioned earlier, the paper lacks strong theoretical underpinnings. While the empirical results are promising, a more in-depth theoretical analysis of the proposed approach would strengthen the paper.\nScope of Evaluation: While the paper evaluates GLIDER on relevant tasks, expanding the evaluation to a broader range of complex, long-horizon tasks would further demonstrate its generalizability and robustness.\nAnalysis of Limitations: The paper could benefit from a more thorough analysis of the limitations of GLIDER. Discussing potential failure cases, scenarios where GLIDER might struggle, and the computational costs associated with the approach would provide a more balanced perspective.", "other_comments_or_suggestions": "Typos and Grammar: A careful pass for typos and grammatical errors is recommended to improve the overall polish of the paper.\nClarification of Terminology: While generally clear, some terminology could be further clarified. For example, explicitly defining what constitutes a \"long-horizon\" task in the context of the paper would be helpful.", "questions_for_authors": "Could you provide a more detailed comparison of GLIDER with other LLM-based approaches to decision-making, if any exist? Specifically, what are the key differences and advantages of GLIDER compared to methods that directly use LLMs for planning without the hierarchical RL structure?\n\nIn the experimental evaluation, could you provide a more detailed analysis of the parameter efficiency of GLIDER? Please include a comparison of the number of parameters used by GLIDER and the baseline methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "TnkdScrvgI", "reviewer": "Reviewer_5Xjh", "summary": "This paper introduces GLIDER, a hierarchical reinforcement learning framework designed to enhance the decision-making capabilities of large language models (LLMs) in long-horizon tasks. The authors propose a two-layer structure where a high-level policy decomposes complex tasks into sub-goals, which a low-level controller executes using reinforcement learning. \n\nThis hierarchical decomposition allows for better exploration and improved long-term credit assignment, addressing the challenges faced by LLMs in sparse-reward scenarios. The method is designed to be parameter-efficient, leveraging shared LLM parameters between the high and low levels to reduce computational overhead.\n\nThe framework is evaluated on ScienceWorld and ALFWorld benchmarks, demonstrating significant performance improvements over prompt-based methods (ReAct, Reflexion) and fine-tuning baselines (NAT, ETO).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your positive and constructive comments for improving our paper. We would be incredibly grateful for your continued support in reviewing our response and offering further encouragement.\n\n***\n\n`Q1. The training pipeline is a bit complicated (three stages), which might be tough for others to reproduce.`\n\nA1. We have provided code with the submission to ensure reproducibility of our main paper. Detailed tutorials and datasets will be released later.\n\nWhile GLIDER involves a three-stage pipeline, each stage serves a distinct purpose. As shown in the following Table, where we compare the performance of baseline, ORL-only, and complete GLIDER implementations on unseen ScienceWorld tasks, the offline RL phase alone can still outperform baseline approaches. Adding SFT improves the results further by efficiently learning valid interactions, though at additional training cost. The optional online phase, enabled by AWAC's smooth offline-to-online transition, makes the overall process more robust and adaptable.\n\n| Method | Mistral-7B | Gemma-7B | Llama-3-8B |\n| --- | --- | --- | --- |\n| NAT | 50.79 | 44.98 | 48.76 |\n| ETO | 51.85 | 47.84 | 52.33 |\n| **GLIDER (ORL)** | 60.23 | 49.48 | 57.12 |\n| GLIDER (SFT+ORL) | 65.14 | 58.50 | 68.34 |\n\n***\n\n`Q2. The paper doesn’t get deeply into real-world deployments or more physically grounded tasks, so scope could be potentially limited.`\n\nA2. This work demonstrates significant algorithmic value through a hierarchical approach that decomposes complex tasks into high-level planning and low-level execution skills. The effectiveness has been validated in sophisticated benchmarks featuring dynamic, sparse reward environments and diverse task scenarios. Meanwhile, these applications in environments with varied tasks and changing states demonstrate the robustness of this design, showing potential for embodied domains and enabling more sophisticated skill composition in robotic manipulation, game strategy learning, and autonomous navigation where multi-level control and temporal abstraction are essential for handling real-world complexity.\n\n***\n\nBest regards,\n\nAuthors", "claims_and_evidence": "In this paper, the authors provide thorough experimental comparisons and ablation studies on both ScienceWorld and ALFWorld. The results show that \n(1) hierarchical learning improves performance over strong baselines, \n(2) offline-to-online adaptation is faster and more stable, and \n(3) the approach is parameter-efficient across different model scales. \n\nEach of these claims has corresponding quantitative evidence (e.g., success rates, adaptation curves, ablation metrics). No obvious discrepancies are found.", "methods_and_evaluation_criteria": "The authors picked well-known benchmarks (ScienceWorld, ALFWorld) that require text-based reasoning and long-horizon planning, which makes sense for testing a hierarchical RL approach. They also used standard offline RL and supervised fine-tuning methods, which makes it easier to judge how well their model handles efficiency, generalization, and adapting to new tasks.", "theoretical_claims": "No formal proofs or heavy theoretical claims are made.", "experimental_designs_or_analyses": "The training pipeline is well-documented and seems logically consistent. The authors show clear comparisons with multiple baselines and ablation studies.", "relation_to_broader_scientific_literature": "Not obvious.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. It nicely combines hierarchical RL with LLMs in an original way, showing strong performance on complex text-based benchmarks.  \n2. The paper is pretty thorough, with lots of experiments and ablation studies that support their claims.  \n3. The idea of reusing learned low-level skills for fast online adaptation is practical and smart.\n\nWeaknesses  \n1. The training pipeline is a bit complicated (three stages), which might be tough for others to reproduce. However, I think this could be alleviated by the codes shared by the authors.\n2. The paper doesn’t get deeply into real-world deployments or more physically grounded tasks, so scope could be potentially limited.", "other_comments_or_suggestions": "No", "questions_for_authors": "Would like to see authors' thoughts on the scope of environments and complexity of the training pipeline.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qn5LhuJmBC", "reviewer": "Reviewer_8JbL", "summary": "The work focuses on long-term decision-making problems with LLM's. They propose GLIDER using concepts of hierarchical reinforcement learning, namely decomposing a complicated task into pieces of small tasks. The low-level controller can be goal agnostic and finish its low level task efficiently as directed by high-level policy. \nThe key selling points for the algorithm are:\n- enhanced exploration in long context tasks\n- fast online adaptation to non-stationary environments\n- strong transferability due to task-agnostic (low-level controller is task agnostic)\n- parameter efficient (this comes by sharing of weights in value and actor-network, and also across high and low-level policies)\nThe authors demonstrate performance on ScienceWorld and ALFWorld datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***\n\n**`Q1. About online fine-tuning, generalization, empirical evidence on fast online adaptation.`**\n\nA1. Online fine-tuning refers to adapt a pretrained policy to unseen tasks using several fine-tuning steps,  i.e., few-shot generalization to new tasks. We train a policy using offline datasets on all tasks except test-conductivity, find-animal, and boil. Then, we fine-tune the trained policy by interacting with these tasks online. The evidence is in Sec. 4.4. Generalization Analysis via Online Fine-tuning (L394-L429), also shown in the following table. GLIDER obtains **a 65%-226% performance improvement** over baselines.\n\n| Method | test-conductivity | find-animal | boil |\n| --- | --- | --- | --- |\n| AC | 0.30 ± 0.10 | 0.45 ± 0.05 | 0.40 ± 0.15 |\n| AWAC | 0.35 ± 0.15 | 0.60 ± 0.10 | 0.45 ± 0.10 |\n| GLIDER | **0.98 ± 0.02** | **0.99 ± 0.01** | **0.95 ± 0.05** |\n\n***\n\n**`Q2. Performance comparison between sharing weights and using different weights.`**\n\nA2. We compare the full GLIDER to two more ablations: using different weights for the actor and critic, and using different for the high-level and low-level policies. The following table shows the performance on unseen tasks in ScienceWorld, with the LIama-3-8B backbone. **We save 50% of model parameters with a <3% performance loss.**\n\n|  Method | Total parameters  | **Score** |\n| --- | --- | --- |\n| GLIDER | 30.22 GB | 68.34 |\n| Decouple actor and critic | 60.19 GB | 70.47 |\n| Decouple high and low policies | 60.44 GB | 69.05 |\n\n***\n\n**`Q3. The novelty of combining existing ideas and demonstrates its use case in solving long context tasks for LLMs.`**\n\nA3. We innovatively combine the strengths of LLMs with the structural advantages of hierarchical RL to **address the significant challenge** of tackling long-horizon interactive tasks. We propose a novel integration of these areas, design a parameter-efficient and generally applicable hierarchy, and enable fast offline-to-online adaptation.\n\n***\n\n**`Q4. It's not clear to me if they really do need a value function here or if it can be surpassed.`**\n\nA4. Classical RL algorithms learn a value function to backpropagate expected optimal returns using dynamic programming updates. GLIDER learns a value function to estimate the action advantage, and regress the advantage-weighted policy. The existence of a value function is a crucial property of RL, which enables the agent to adapt behaviors through trial-and-error rather than only imitating demonstrations. The ablation study (L354-L369) shows **a 13.9%~33.5% performance improvement** of pure ORL over SFT.\n\n| Backbone | SFT | ORL |\n| --- | --- | --- |\n| Mistral-7B | 45.11 | **60.23** |\n| Gemma-7B | 40.54 | **49.48** |\n| LIama-3-8B | 50.17 | **57.12** |\n\n***\n\n**`Q5. L241: What is explicit modeling of the behavior policy?`**\n\nA5. GLIDER maximizes an advantage-weighted likelihood function as $\\max E[\\exp(A(s,a))\\cdot\\log\\pi(a|s)]$. We learn a value function to estimate the action advantage, and regress the advantage-weighted policy. The maximization of the likelihood $E[\\log\\pi(a|s)]$ corresponds to an explicit modeling of the behavior policy. \n\n***\n\n**`Q6. L213: How $n_h/n_l$ affects the gradient of loss.`**\n\nA6. $n$ is the length of the sentence generated by policy $\\pi_{\\theta}$. We divide the original SFT loss by the sentence length as $-E[\\log\\frac{\\pi_{\\theta}(a|s)}{n}]=-E[\\log\\pi_{\\theta}(a|s)] + \\log n$. A smaller length $n$ will induce a larger gradient of loss, thus regularizing the model to output relatively shorter sentences.\n\n***\n\n**`Q7. L247-259: You first train low level and then fix it and later train only high level?`**\n\nA7. **At SFT and ORL, we simultaneously train the high- and low-level policies.** Then, at the offline-to-online stage, we fix the low-level skills and only fine-tune the high-level policy. \n\n***\n\n**`Q8. L270: what is temporal abstraction knowledge?`**\n\nA8. Temporal abstraction refers to **abstracting a sequence of temporal actions into a skill**, gaining the potential to greatly speed planning and learning on large problems. \n\n***\n\n**`Q9. L89: hierarchical token level not clear.`**\n\nA9. “The hierarchical token-level actors and sentence-level critics” refers to that we have **a token-level actor and a sentence-level critic at both levels**.\n\n***\n**`Q10. It would be good to apply it on multiple datasets.`**\n\nA10. We will add a new benchmark WebShop. Due to very limited time, we have only completed the offline dataset collection, and we are rushing to update them before the rebuttal deadline. \n\n***\n## Summary Response\n\nWe appreciate your time in reviewing our manuscript and your valuable comments. We have made a number of changes and justifications to address your concerns, and we are more than delighted to have further discussions to improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "Yeah, most of the claims are supported by empirical evidence and ablation studies.\nI do not find empirical evidence on \"fast online adaptation to non-stationary environments\".\nAlso, they claim parameter efficient which they are by sharing of weights but i think it needs to be also empirically validated by loss in performance due to sharing of weights as compared to using different weights.", "methods_and_evaluation_criteria": "yeah, the method is tried on long context tasks. However would be good to apply it on multiple datasets, currently on a limited number of datasets.", "theoretical_claims": "The paper is experimental (no theoretical claims).", "experimental_designs_or_analyses": "Yeah, the presented empirical evidence is strong for hierarchical structure.", "relation_to_broader_scientific_literature": "I am not aware of similar work in the context of LLM's. The paper connects existing ideas from offline RL, Hierarchical RL, SFT, etc.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n- Proposed an overall framework with offline hierarchical RL for LLM's which is parameter-efficient, generalizable to an extent and scalable.\n- Presented strong evidence of hierarchical breakup for long context task\n\nWeakness:\nThe work combines existing ideas and demonstrates its use case in solving long context tasks for LLMs. However, there is no starking novelty. Also, it's not clear to me if they really do need a value function here or if it can be surpassed. Also see claims and evidences above for other weaknesses.", "other_comments_or_suggestions": "line 80: two LLM policies: which two? low and high level?\n\n89: hierarchical token level not clear\n\nremove 2nd bullet point of contribution or merge it, it's a feature of your algorithm (not a contribution)\n\n3.1: add details about your state and action space\n\n180: what is \\sigma_N\n\n213: around eqn3: Does n_h, n_l belongs to expectation? i don't understand why it would affect gradient of loss?\n\n241: What is explicit modelling of the behaviour policy?\n\nline 247-259: You first train low level and then fix it and later train only high level?\n\n270: what is temporal abstraction knowledge? \n\n305:311: Add more details about both the tasks.. elementary science experiments doesn't say much, some e.g., would be nice\n\n406: what is online fine-tuning? I am wondering why is it saying about generalization. Ideally, for generalization you could fine tune on all tasks except test-conductivity, find-animal, boil and then check trained model performance on these tasks?", "questions_for_authors": "See above\n\n\n### Update after rebuttal\n\nThanks to authors for addressing my questions, especially to run the experiment of quantifying performance loss. Please include the promised changes in the final version. I will maintain my inclination towards acceptance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gdo7oEiJlF", "reviewer": "Reviewer_876J", "summary": "This paper integrates hierarchical reinforcement learning with LLMs, leveraging a high-level planner to decompose tasks and a low-level executor to perform actions. Experimental results on ScienceWorld and ALFWorld demonstrate significant performance gains over existing baselines, with strong adaptability to unseen tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***\n\n**`Q1. The authors could introduce GPT-based prompting baselines (Reflextion or ReAct) to compare whether small-scale LLMs with hierarchical training can match or even outperform larger models.`**\n\nA1. Following your advice, we compare GLIDER using small-scale LLMs (LIama-3-8B) to ReAct using large-scale LLMs (GPT4). Results in the following table demonstrating that our small-scale LLMs with hierarchical training can match prompting baselines with much larger LLMs, with **a 5%~98% performance improvement**. \n\n| Method | ScienceWorld (seen/unseen) | Alfworld (seen/unseen) |\n| --- | --- | --- |\n| GLIDER (LIama-3-8B) | **77.43/68.34** | **71.56/75.38** |\n| ReAct (GPT-4) | 67.32/65.09 | 44.29/38.05 |\n\n***\n\n**`Q2. The novelty of the hierarchical approach in LLM to generate high-level plans.`**\n\nA2. We innovatively combine the strengths of LLMs with the structural advantages of hierarchical RL to **address the significant challenge of tackling long-horizon interactive tasks**.  While building upon existing research in LLMs and hierarchical RL, we propose a novel integration of these areas, design a parameter-efficient and generally applicable hierarchy, and enable fast offline-to-online adaptation.\n\n***\n\n**`Q3. The paper relies on environmental observations to indicate sub-task completion. Is this design applicable when extended to other, possibly more complex, environments?`**\n\nA3. **This design alleviates the necessity for any manual or task-specific design, and is naturally applicable in more complex environments.** For example, in Minecraft, a broad task like \"build a house\" could be decomposed into simpler sub-tasks like \"gather wood materials\". The completion of that sub-task can be easily monitored by an attribute “inventory changes” of the environment observation. Our design remains robust at any complexity level, as long as the environment provides clear completion signals.\n\n***\n\n**`Q4. Did you observe any effects of the quality of high-level sub-tasks generated on the execution performance of the low-level actions?`**\n\nA4. The quality of generated sub-tasks can affect the execution performance of the low-level policy. Intuitively, if the generated sub-task is too hard (e.g., close to the original task), the low-level policy can fail to accomplish it. That is why we need the hierarchical decomposition to break down the original task into simpler sub-tasks. **By trial-and-error, the high-level policy adapts its behavior using environment-provided rewards, and is reinforced to generate increasingly fine-grained sub-tasks.** \n\nThis “reinforcement” mechanism can be illustrated by a task evolution example: when the high-level policy generates a sub-task “heat substance” in the kitchen, the low-level policy attempts to use the stove but receives feedback that it's broken. After this failed attempt, the high-level policy adapts by generating a more specific sub-task “navigate to foundry to find blast furnace”.\n\n***\n\n**`Q5. Testing on more complex environments, such as WebShop.`**\n\nA5. We will add a new benchmark, WebShop, to further validate GLIDER’s performance. Due to very limited time, we have only completed the offline dataset collection, and we are rushing to update them before the rebuttal deadline. \n\n***\n\n**`Q6. More extensive literature like RL-GPT and Thought Cloning.`**\n\nA6. Thanks for your advice, and we will cite more relevant references.\n\n***\n## Summary Response\n\nWe appreciate your time in reviewing our manuscript and your valuable comments. We have made a number of changes and justifications to address your concerns, and we are more than delighted to have further discussions to improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The benchmarks used—ScienceWorld and ALFWorld—have already demonstrated strong performance from advanced LLMs, such as the GPT-series models, leveraging prompt-based methods. These tasks are highly structured. The authors employ small-scale LLMs and apply a multi-stage training approach, yielding improvements over the base models; however, this outcome is somewhat expected. To more effectively evaluate the method’s impact, the authors could introduce GPT-based prompting baselines to compare whether small-scale LLMs with hierarchical training can match or even outperform larger models. Furthermore, testing on more complex environments, such as WebShop, would provide stronger evidence of the proposed approach’s scalability.", "theoretical_claims": "The paper does not appear to provide formal mathematical proofs for theoretical claims, as it primarily focuses on empirical validation.", "experimental_designs_or_analyses": "While the benchmark selection may be somewhat limited in scope, the experimental design and baseline comparisons are thorough and well-executed.", "relation_to_broader_scientific_literature": "The hierarchical approach of first leveraging an LLM to generate high-level plans before executing them is not particularly novel. However, the paper’s parameter-efficient design is well-executed. The use of a shared LLM backbone for both high- and low-level policies, combined with lightweight fine-tuning techniques, enhances efficiency without significantly increasing computational costs.", "essential_references_not_discussed": "- RL-GPT: Integrating Reinforcement Learning and Code-as-policy\n- Thought Cloning: Learning to Think while Acting by Imitating Human Thinking", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How does the proposed method compare with GPT models using reflection or ReAct-based methods in terms of performance?  \n\n2. Regarding the sub-task decomposition and intrinsic reward design, the paper primarily relies on environmental observations to determine whether a sub-task is completed. Is this design applicable when extended to other, possibly more complex, environments?  \n\n3. During the prompt design process, did you observe any effects of the quality of high-level sub-tasks generated on the execution performance of the low-level actions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HzDRwXGaep", "reviewer": "Reviewer_MQX2", "summary": "The paper proposes GLIDER, an extension of hierarchical RL to LLM agents where two separate LLM control the high and low level policy. The proposed training framework relies on a combination of supervised fine tuning and implicit Q learning. The authors test on two domains, ScienceWorld and AlfWorld, obtaining promising results in both cases. The authors' experiments use relatively small LLMs (<10B), which is understandable given the need for in-device learning and speed.\n\nOverall, this research is promising, but some aspects require more work or clarification, and I recommend rejection in its current state", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Methods\n\n**`Q1. The optimal policy guarantee and the correctness of Eq. (7), given that actions are at the token level while rewards are formulated at the transition level.`**\n\nA1. First, our policy derivation of inherits Advantage-Weighted Regression (AWR) [1-2], a popular algorithm using convergent maximum likelihood loss to convert RL into supervised learning subroutines. **Our method follows their theoretical properties,** and we will establish the theoretical counterpart in the appendix.\n\nSecond, GLIDER correctly attributes the full transition reward to each token through Eq. (7). **Assigning the transition-level advantage $A$ to each token $w_i$ is equivalent to assigning the correct advantage to the whole action as $E[\\exp(A(s,u))\\cdot\\sum_{i=1}^n\\log\\pi_{\\theta}(w_i|s,w_{1:i-1})]=E[\\exp(A(s,w_{1:n}))\\cdot\\log\\pi_{\\theta}(w_{i:n}|s)]$,** using the relationship between total probability and conditional probabilities in autoregressive models.\n\n[1] Advantage-weighted regression: Simple and scalable off-policy RL, arXiv:1910.00177, 2019.\n\n[2] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, arXiv:2006.09359, 2020.\n***\n**`Q2. Superior parameter efficiency to what?`**\n\nA2. **The superior parameter efficiency is mainly reflected in the design of the RL model and the hierarchical structure**. \n\nFirst, we share the same frozen LLM backbone for actor and critic to greatly reduce model parameters, in contrast to many RL algorithms using independent actor and critic. Second, we let the high and low levels share the same model and differentiate them with a lightweight hierarchy prompt, as opposed to typical hierarchical methods with independent models at each level. Also, we adopt PEFT to further improve parameter efficiency following the common practice in LLMs.\n***\n**`Q3. Clarification about generalization, fine-tuning, and non-stationarity.`**\n\nA3. In the offline-to-online stage, we focus on GLIDER’s fast adaptation ability to unseen tasks using a small number of fine-tuning steps, i.e., few-shot generalization. We will make more precise clarification.\n***\n**`Q4. Hierarchical RL needs domain knowledge.`**\n\nA4. **We design a generally applicable hierarchy to minimize reliance on domain knowledge.** The low-level policy is instructed by an intrinsic reward indicating the sub-task completion, which is easily accessible fro environment observations, alleviating the necessity for any manual or task-specific design.\n***\n## Experiments\n\n**`Q5. A weakness is the lack of code.`**\n\nA5. We have uploaded the code as **supplementary materials** for reproducibility when submitting the main paper. Further, we will release detailed tutorials and datasets to ensure full reproducibility.\n***\n**`Q6. Why training for only 5 epochs in SFT? Did you attempt to optimize its hyperparameters?`**\n\nA6. We trained SFT for 5 epochs to avoid overfitting. Also, we optimized the hyperparameters and added dropout in LoRA layers, which can be found in our submitted code.\n|SFT epochs|2|3|4|5|6|7|\n|---|---|---|---|---|---|---|\n|Score|34.14|42.02|45.11|**50.17**|40.16|37.43|\n***\n**`Q7. The dataset for SFT, the role of SFT, the comparison between SFT and GLIDER on expert datasets.`**\n\nA7. We use **expert datasets for SFT**, matching the nature of supervised learning. **The role of  SFT is to construct a base agent to improve learning stability and sample efficiency for subsequent stages.** GLIDER adopts RL to steer the base agent towards use-specified tasks, excelling both imitating demonstrations and adapting behaviors through trial-and-error. Experiments show **a 11.6%~19.6% performance improvement** over SFT.\n||SFT|GLIDER|\n|---|---|---|\n|ScienceWorld (seen/unseen)|69.40/57.12|**77.43/68.34**|\n|AlfWorld (seen/unseen)|62.24/65.21|**71.56/75.38**|\n***\n**`Q8. Did the authors experiment with multiple seeds?`**\n\nA8. Yes. Our method obtains close results across different seeds. The following table shows the example result of GLIDER with LIama-3-8B, score on unseen tasks in ScienceWorld.\n|Seed|0|1|2|3|\n|---|---|---|---|---|\n|Score|68.34|69.72|68.23|67.11|\n***\n**`Q9. Regularizer in Eq. (3), and using the end-of-text token in SFT.`**\n\nA9. $n$ is the length of the sentence generated by policy $\\pi_{\\theta}$. We divide the original SFT loss by the sentence length as $-E[\\log\\frac{\\pi_{\\theta}(a|s)}{n}]=-E[\\log\\pi_{\\theta}(a|s)] + \\log n$. A smaller length $n$ will induce a larger gradient of loss, thus regularizing the model to output relatively shorter sentences. Also, we did use the end-of-text token in SFT, and our length regularizer is complementary to it.\n***\n## Summary Response\n\nWe appreciate your time in reviewing our manuscript and your valuable comments. We have made a number of changes and clarifications to address your concerns, and we are more than delighted to have further discussions to improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.", "claims_and_evidence": "The experiments and methodology partially support the claims, but not some elements need additional work. For example:\n\n“we propose an offline hierarchical framework GLIDER with superior efficiency and broad applicability.”\n\nSuperior parameter efficiency to what? Any sensible baseline involving LLMs would use PEFT.\n\n“Our method enables fast offline-to-online adaptation to -stationary environments”\n\nGeneralization, fine-tuning, and non-stationarity are related but different concepts. The authors could be more precise. I think the authors focus on generalization to new tasks. It does not appear as if their online setting changes has a continuosly changing non-stationarity environment.\n\n“Comprehensive studies on ScienceWorld and ALFWorld  show that our method consistently improves  and generalization capacity, surpassing a  of baselines by a significant margin.”\n\nSee my comments in the experiment section.", "methods_and_evaluation_criteria": "Experiments are run on standard benchmarks: ScienceWorld and ALFWorld. The experiments are limited, but this is common in this literature, given the computational cost. Overall, the proposed evaluation for the methods is sensible.", "theoretical_claims": "No theoretical analysis provided.", "experimental_designs_or_analyses": "The experiment design seems sound. I have some questions:\n\n1. Did the authors experiment with multiple seeds?\n2. What dataset did you use for the SFT: the purely expert one or the mix with medium expertise too? If the latter is the case, could SFT eventually be better than your method if enough expert data is available?\n3. Why did you train for only five epochs during SFT? Since you’re also using SFT as a baseline, did you attempt to optimize its hyperparameters?", "relation_to_broader_scientific_literature": "The paper is timely and relevant to the RL community, increasingly interested in integrating LLMs. At the same time, its contribution might be limited since there are no theoretical results; the extension to hierarchical agents seems straightforward. Further, it maintains a common limitation with hierarchical RL: that there needs to be domain knowledge (in this case, pre-labelled trajectories) providing options/high-level actions.", "essential_references_not_discussed": "The paper could benefit from a more extensive literature discussion around LLMs in RL. Examples:\n\n\n* Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding large language models in interactive environments with online reinforcement learning. In ICML, 2023.\n\n* Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. In ICML, pp. 8657–8677, 2023.\n\n* Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, A. T. Large language models as generalizable policies for embodied tasks. In ICLR, 2023.\n\n* Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., Liang, Y., and CraftJarvis, T. Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents. In NeurIPS, 2023.\n\n* Wen, M., Wan, Z., Wang, J., Zhang, W., and Wen, Y. Reinforcing LLM agents via policy optimization with action decomposition. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.", "other_strengths_and_weaknesses": "A weakness is the lack of code despite being a mostly experimental paper.", "other_comments_or_suggestions": "I don’t understand the notation in (1) and (2). Why is this a sum, or what is Sigma? How can you sum “brackets” and what does the semi-colon inside the bracket mean. What is d? Please explain your notation as I believe it is non-standard in the literature.", "questions_for_authors": "1. Can you demonstrate that the given method obtains the optimal policy? Can you show that Equation (7) is correct? The LLM policy actions are at the token level, whereas the expression is formulated at the MDP transition level. In a way, the LLM has a different MDP at the token level. There should be a way to attribute the full transition reward to each token so that Equation (7) holds. For instance, what would be the equivalent formulation for assigning a reward to each token? Discussion around this topic could be useful, or even lead to more in-depth mathematical analysis.\n2. Unless I am confused, the definition of response length as a regularizer in Equation (3) seems problematic, given that SFT is applied at the token level. Did the authors intend to imply that they down-/up-weight certain dataset elements? Could this be reformulated as a weighted loss to emphasize certain dataset elements? Additionally, did you include an end-of-text token during SFT? In my experience, this kind of regularization hasn’t been necessary to produce shorter responses after SFT, but I understand each data can be different.\n3. Why did you train for only five epochs during SFT? Since you’re also using SFT as a baseline, did you attempt to optimize its hyperparameters (e.g., train longer, add dropout)?\n4. One of my concerns is that without SFT, the performance is comparable with the baselines  NAT and  ETO. Could you elaborate further on the role of SFT and why this is still a fair comparison?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "v95gC9BvBs", "reviewer": "Reviewer_yxB6", "summary": "This paper introduces GLIDER, an approach for fine-tuning LLMs to act as agents in interactive environments. GLIDER relies on a hierarchical architecture in which the LLM is used to both propose a plan (high-level policy) and execute it (low-level policy). Using Behavioral Cloning and offline RL, the authors demonstrate that GLIDER can learn to solve various tasks in ScienceWorld and AlfWorld, achieving strong performance.\n\nOne important aspect of GLIDER is that the low-level policy is trained solely on subgoals proposed by the high-level policy: the latter not only generates these subgoals but also rewards the low-level policy based on its trajectory. The authors argue that this approach preserves the low-level policy’s strong generalization abilities, as the subgoals it is trained to solve are not directly tied to the environment’s tasks.\n\nThe paper provides in-depth ablation studies on different components of GLIDER, highlighting the importance of offline RL and the hierarchical architecture. It also demonstrates that offline RL can successfully exploit suboptimal demonstrations. Finally, the authors show that GLIDER can efficiently learn new tasks using online RL to fine-tune the high-level policy. Transitioning from offline RL to online RL is achieved seamlessly using the chosen RL method: AWAC.\n\nThe authors conclude by discussing GLIDER's broader potential, suggesting that its applications extend beyond interactive textual environments.\n\n## update after rebuttal\nMost of my concerns below were due to missing explanations in the manuscript as well as misunderstandings about the generalization tests. The authors have addressed all of these concerns. I therefore maintain my recommendation to accept this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your constructive comments for improving our paper. We would be incredibly grateful for your continued support in reviewing our response and offering further encouragement.\n***\n`Q1. How is the temporal abstraction parameter c determined, given that subgoals are dynamically generated?` \n\nA1. The low-level policy dynamically determines when to end a subtask by analyzing environment observations, rather than strictly waiting for c steps, shown in L.214~215. This adaptive mechanism ensures efficient execution while maintaining the hierarchical control structure.\n***\n`Q2. How does the framework evaluate subtask completion and handle failures when subgoals are abstract or when the low-level policy cannot execute the high-level plan effectively?`\n\nA2. The low-level policy executes subtasks, while the high-level policy learns finer-grained subtasks through feedback. In ScienceWorld, during early epochs, the high-level policy generates unachievable subtasks like \"heat substance\" but finds broken stove, later adapting to executable ones like \"Navigate to foundry\". Similarly in Minecraft, it evolves from \"build a house\" to specific tasks like \"gather wood\" and \"create foundations\".\n***\n`Q3. How does ETO collect trajectories in the online regime, and what dataset does it use?`\n\nA3. For ETO baseline, we first perform SFT on the base model, then let it interact with the environment online to collect both failure trajectories and expert demonstrations for DPO training. We will provide more detailed baseline implementation details in the appendix.\n***\n`Q4. What data did AWAC use in Section 4.4? Was the dataset emptied or did it retain the demonstrations from Section 4.2?`\n\nA4. Following AWAC, we kept the offline demonstration data instead of emptying the dataset, as it helps maintain base distribution performance. Unlike Section 4.2, Section 4.4 focused on evaluating adaptation to novel scenarios by selecting specific out-of-domain tasks and collecting additional data through online interaction\n***\n`Q5. What exactly is the AC baseline in Section 4.4? A clearer explanation of these details would improve the study's transparency.`\n\nA5. The AC (Actor-Critic) baseline differs from AWAC(Advantage Weighted Actor-Critic) by using standard policy gradient instead of advantage weighting. We will provide more detailed implementation specifications in the appendix.\n***\n`Q6. Section 2's literature review could be enriched with additional works (GLAM, POAD, and ELLM) on online RL approaches for LLM agents.`\n\nA6. Thanks for your helpful advice. We will adjust the related work section based on these suggestions.\n***\n`Q7. Where is the empirical evidence showing how the generated subgoals differ between demonstrations and new tasks, and how do they support the claimed generalization ability?`\n\nA7. Our approach demonstrates cross-task generalization through qualitative and quantitative evidence. Appendix Figure 9 shows subgoal reuse between tasks, with both directly transferable components and analogous patterns. The effectiveness is validated on three out-of-domain tasks unseen during training, where GLIDER significantly outperforms baselines.\n|Method|test-conductivity|find-animal|boil|\n|---|---|---|---|\n|AC|0.30 ± 0.10|0.45 ± 0.05|0.40 ± 0.15|\n|AWAC|0.35 ± 0.15|0.60 ± 0.10|0.45 ± 0.10|\n|GLIDER|**0.98 ± 0.02**|**0.99 ± 0.01**|**0.95 ± 0.05**|\n***\n`Q8. Why wasn't the low-level policy finetuned in offline-to-online experiments, and what would be the performance impact if it was?`\n\nA8. L.252~258 explain this design choice. Low-level skills use intrinsic reward functions instead of task-specific ones, allowing cross-task generalization and robustness to distribution shifts between offline training and online deployment. While joint finetuning could improve performance, adjusting only the high-level policy is sufficient given the task-agnostic nature of low-level skills.\n***\n`Q9. Figure 2's method diagram is overcomplicated and hard to understand, with unclear connections and training flows.`\n\nA9. We will simplify the Figure 2. During value head finetuning, the LLM backbone remains fixed, while policy training updates the LLM to learn behaviors.\n***\n`Q10. What is the rationale behind including a length regularization term in Equation 3?`\n\nA10. The length regularization term constrains action sequences to match the inherently short nature of valid environment interactions. By dividing the SFT loss by sentence length n as $-E[\\log\\frac{\\pi_{\\theta}(a|s)}{n}]=-E[\\log\\pi_{\\theta}(a|s)] + \\log n$, it effectively prevents unnecessarily long action generation.\n***\n`Q11. The Appendix B mentions \"Cross-task Generalization sampling\". Were these data necessary?`\n\nA11. Distribution shift in offline RL leads to policy failures on out-of-distribution states. Cross-task sampling simulates potential shifts by exposing the model to varied scenarios, helping handle unseen tasks.\n***\nBest regards,\n\nAuthors", "claims_and_evidence": "The proposed method and empirical evidence clearly support the paper's claims by introducing an approach that leverages hierarchical training, seems pretty general, adapts to unseen tasks, and can be easily trained on new tasks in an online regime.", "methods_and_evaluation_criteria": "GLIDER is a relatively straightforward method. The choice of AWAC appears to be well-suited, as it enables both offline and online RL.\n\nOne concern I have relates to the $c$ parameter, which controls the length of low-level trajectories. I could not find detailed information on how this parameter is set, apart from a footnote stating that it depends on the subgoal. Since the subgoals are generated dynamically by the high-level policy, it is unclear how one can determine in advance the possible subgoals and the number of steps required to solve them. The paper suggests that $c$ is fixed, meaning that the high-level policy awaits $c$ steps before evaluating the low-level policy's trajectory.  However, what happens if the subgoal is completed in fewer than $c$ steps?\n\nAdditionally, the paper states that \"the subtask completion can be easily accessible from the environment observation\" to compute the intrinsic reward for the low-level policy. This assumption seems overly optimistic when moving to the online regime. First, since subgoals are generated by the high-level policy, they could be highly abstract and difficult to evaluate. Second, the feasibility of this approach heavily depends on the complexity of the environment and the LLM's capabilities. For instance, what would happen in Minecraft if the subgoal generated is \"build a house\"? Furthermore, what occurs if the low-level policy is unable to solve the proposed subgoals? How does the framework handle situations where the high-level policy produces an effective plan, but the low-level policy fails to execute it?", "theoretical_claims": "This paper contains no theoretical claims.", "experimental_designs_or_analyses": "ScienceWorld and AlfWorld are two well-established benchmarks for evaluating LLM-based embodied agents.\nGLIDER demonstrates strong performance, and the analyses using three different LLMs highlight its robustness.\n\nHowever, I believe the experiments would benefit from additional baselines, particularly those involving online RL methods.\nI also could not find sufficient details regarding the baselines. For example, what dataset is used by ETO, which is supposed to collect trajectories in an online regime?\n\nAdditionally, in Section 4.4, what data did AWAC use? Was the dataset emptied before the experiment, or did it still include the demonstrations used in Section 4.2? Also, what exactly is the AC baseline in Section 4.4? A clearer explanation of these details would improve the study's transparency.\n\nFinally, I would like to mention that I really enjoyed reading the ablations. The results provide valuable insights into the impact of different components, particularly emphasizing the importance of the hierarchical framework and offline RL.", "relation_to_broader_scientific_literature": "The links with prior work are covered overall. The paragraph discussing online RL approaches for LLM agents in Section 2 could mention other prior work, such as GLAM (Carta et al., 2023) or POAD (Wen et al., 2024). The latter also fits methods using an utterance-level critic, similar to ArCHer (Zhou et al., 2024), which is mentioned in Section 3.4. Prior work using intrinsic rewards generated by an LLM could also be mentioned, e.g. ELLM (Du et al., 2023).   \nAll these references are not essential to understanding the key contributions, they are just suggestions.", "essential_references_not_discussed": "I do not see any essential missing reference.", "other_strengths_and_weaknesses": "Concerning weaknesses, I already mentioned all my concerns in previous parts.\nOne additional strength of this paper is that it is well-written and easy to follow.", "other_comments_or_suggestions": "The authors argue that the high-level policy produces general subgoals, allowing the low-level policy to remain useful even for unseen tasks. However, I would have liked to see empirical evidence supporting this claim. For instance, did the authors analyze the subgoals in the demonstration dataset? What subgoals were generated in the offline-to-online experiments, and how different were they from those in the demonstrations?\n\nAdditionally, the authors chose not to finetune the low-level policy in offline-to-online experiments, based on the assumption that subgoals from training tasks would be reused in new tasks. Have they tested finetuning the low-level policy? If so, how does it affect the results?\n\nI also found Figure 2 difficult to interpret due to the overwhelming amount of information and arrows. Even with the caption, it was unclear to me. For instance, in (a), it is unclear what is frozen and what is finetuned—the finetuning block seems to be associated with the low-level trajectory.", "questions_for_authors": "1) Eq. 3 indicates that a regularization on the outputs' length is also applied to the low-level policy. I do not understand this regularization term as the low-level policy is supposed to output the environment's actions. Could the authors provide further explanations on this?\n2) The Appendix B mentions \"Cross-task Generalization sampling\". Were these data necessary?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the agent planning problem with a novel hierarchical RL algorithm using LLM policies. After rebuttal, all 6 reviewers agree with the writing quality and are satisfied with the thorough experiments. A solid contribution. Hence accept.
Decision: Accept (poster)