Leveraging Imitation Learning and LLMs for Efficient Hierarchical Reinforcement Learning

Authors: Runhan Yang, Jieao Shi, Mengqi SU, Dongruo Zhou
Primary area: reinforcement learning
Keywords: LLM, HRL

Abstract:
In this paper, we introduce an innovative framework that combines Hierarchical Reinforcement Learning (HRL) with Large Language Models (LLMs) to tackle the challenges of complex, sparse-reward environments. A key contribution of our approach is the emphasis on imitation learning during the early training stages, where the LLM plays a crucial role in guiding the agent by providing high-level decision-making strategies. This early-stage imitation learning significantly accelerates the agent's understanding of task structure, reducing the time needed to adapt to new environments. By leveraging the LLM’s ability to generate abstract representations of the environment, the agent can efficiently explore potential strategies, even in tasks with high-dimensional state spaces and delayed rewards. Our method introduces a dynamic annealing strategy in action sampling, balancing the agent's reliance on the LLM’s guidance with its own learned policy as training progresses. Additionally, we implement a novel value function which incorporates the LLM’s predictions to guide decision-making while optimizing token efficiency. This approach reduces computational costs and enhances the agent’s learning process. Experimental results across three environments—MiniGrid, NetHack, and Crafter—demonstrate that our method significantly outperforms baseline HRL algorithms in terms of training speed and success rates. The imitation learning phase proves critical in enabling the agent to adapt quickly and perform efficiently, highlighting the potential of integrating LLMs into HRL for complex tasks.

Decision: reject
Reviewer scores: 5, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1729467486958

thread_type:
official_review

<thread_object>
creator: reviewer_lwqs

<content>
summary:
The IHAC framework models decision-making as a hierarchical RL problem, utilizing a two-phase approach:
- In the first phase, it uses an external LLM for imitation learning, guiding the selection of high-level options to accelerate early learning when the agent's experience is limited  (in both exploration and exploitation side)
- In the second phase, a standard RL algorithm like PPO further refines the policy

Tested on benchmarks like MiniGrid, IHAC outperforms existing methods in efficiency and performance, especially in optimizing LLM token usage.

strengths:
- The paper is well-motivated, since the LLM policy can be potentially informative, it can be used to faciliate both exploration and exploitation procedure via policy mixing and KL regularization
- Comprehensive experiment is conducted, provide a clear picture of practical performance

weaknesses:
- My main concern about this paper is it's novelty:
1. To ground LLM as a policy planner in real environment, modelling the decison problem as hierarchical RL is well-discussed in literature (e.g., https://arxiv.org/pdf/2310.10645)
2. Though this work also uses LLM to help learning a RL policy beyond simple imitation, but the key regularized based method, as pointed in the paper, is discussed in https://arxiv.org/pdf/2402.16181

Hence, it seems that the main contribution is to leverage LLM in the exploration stage. I would suggest authors to better discuss and highlight the contributions

questions:
Despite the weakness, I also have the questions:
1. Can authors provide a more detailed discussion on the importance and effectiveness of using LLM policy in exploration and exploitation respectively? How such design sampling / regularization contributes to the better sample efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729871296765

thread_type:
official_review

<thread_object>
creator: reviewer_ql21

<content>
summary:
This paper introduces a framework IHAC, which combines hierarchical reinforcement learning with large language models to solve complex and sparse-reward environments, where high-level actions, e.g., macro actions or skills are provided. In phrase I, IHAC first leverages LLMs to sample heuristic actions. It applies an annealing strategy to decrease the reliance on LLM progressively. while training, RL agents learned the policy and value in an imitation style. In phase II, it directly uses the standard RL algorithm to train with the learned policy and value function. Empirical studies show that IHAC outperforms baseline methods on MiniGrid, NetHack and Crafter, in terms of sample efficiency and success rate.

strengths:
Empirical studies show that IHAC outperforms baseline methods on MiniGrid, NetHack and Crafter, in terms of sample efficiency and success rate.

weaknesses:
This paper suffers from several critical weaknesses:

1. This work has nothing to do with hierarchical RL, however, this concept seems to be the key point and contribution of the paper. Hierarchical RL usually learns both high-level planning and low-level control. However, in this work, high-level actions are already pre-defined and provided and the agent does not learn the low-level control. The setting degenerates into the most common single-layer RL, just like the common robotics setting where high-level skills are provided. Lines 172-180 also do not show the mapping from high-level action to low-level control.

2. The proposed algorithm is trivial and theoretically incorrect. In phase I, the learned value can only be applied to offline policy, since the agents also use LLM to sample actions. However, in Line 201, the authors claimed to run a standard RL algorithm like PPO. Moreover, PPO learns V(s) instead of Q(s, a), which is also incompatible with the proposed method. 

3. The paper is poorly written. Despite the incorrectly used concept of hierarchical RL, which is extremely confusing, the authors have a very limited study of works that leverage LLM to facilitate RL training. HRL and Imitation learning are not necessary to be mentioned in the related work and the LLM Agent section is quite unrelated to the topic of this paper. All the equations lack detailed explanation and analysis.

4. Due to the limited related work study, the baseline selection is also quite limited. Related work, which is not limited to GLAM[1], TWOSOME[2], SayCan[3], DigiRL[4] and ArCHer[5], are not discussed and compared. 

[1] Carta, Thomas, et al. "Grounding large language models in interactive environments with online reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[2] Tan, Weihao, et al. "True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning." The Twelfth International Conference on Learning Representations.

[3] Brohan, Anthony, et al. "Do as i can, not as i say: Grounding language in robotic affordances." Conference on robot learning. PMLR, 2023.

[4] Bai, Hao, et al. "Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning." arXiv preprint arXiv:2406.11896 (2024).

[5] Zhou, Yifei, et al. "Archer: Training language model agents via hierarchical multi-turn rl." arXiv preprint arXiv:2402.19446 (2024).


Other issues:
1. Algorithm 1 should have a caption.   

2. Line 239 see XXX is not replaced.    

3. Line 274 typo: importantbt  
4. Equation in Line 248 does not have a label.

questions:
1. How does LLM calculate the probability of high-level action in the equation shown in Line 248?

2. What is the second KL divergence used for in equation 1? And why the first KL divergence is inversed compared to the second one?  

3. What does it mean in Lines 318-319? "For all baselines, we did not train them until they converged".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730418677357

thread_type:
official_review

<thread_object>
creator: reviewer_hv47

<content>
summary:
This paper proposes a new training scheme to utilize LLMs to guide RL in tasks with sparse rewards. The evaluation results show consistent improvement over recent related works both in performance and token efficiency.

strengths:
It's an interesting and novel idea to convert the common knowledge of LLMs into options to guide the policy learning process in tasks with sparse rewards, which is certainty an important research question for RL. The authors also provide a wide range of evaluation results.

weaknesses:
(a) The technical contribution is a little bit of limited, that is, introducing a KL-regularized pre-training phase to typical RL training processes.

(b) Extra hyperparameters are introduced, such as \alpha, \lambda_t, and the number of training iterations of Phase 1.

(c) From the ablation study results, it seems that PPO only already performed well enough. It's important to show that the new algorithm can perform significantly better in scenarios where PPO only would fail.

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687177821

thread_type:
official_review

<thread_object>
creator: reviewer_3ukv

<content>
summary:
The paper proposes a two-stage algorithm aimed at enhancing exploration and reducing token usage. In the first stage, the RL agent imitates the policy generated by the LLM to improve its exploration capabilities. This phase leverages the high-level guidance from the LLM, helping the agent navigate the environment more efficiently. In the second stage, a vanilla PPO approach is applied to fine-tune the policy. The paper compares this two-stage algorithm with existing methods like LLMxHRL and LLM4Teach across several environments, including Minigrid, NetHack, and Crafter. Results indicate that this approach not only achieves higher performance but also significantly lowers token consumption, demonstrating both efficacy and efficiency in LLM-guided RL tasks.

strengths:
1. This research addresses a key problem in LLM agents, using LLMs as intrinsic reward generators to enhance RL algorithm's sample efficiency.

2. The proposed algorithm is straightforward, with clear writing and experiments supporting the method's claims.

3. Experimental results on Minigrid, NetHack, and Crafter show that the method outperforms LLMxHRL and LLM4Teach in performance and token efficiency.

weaknesses:
1. The reduction in token use may be due to early stopping: Table 3 shows a 10% pre-training percentage, and Figure 8’s last row indicates that success rates for 3 of 4 tasks saturate after 10% of training, suggesting the two-stage algorithm may reduce to a one stage algorithm. Additionally, the paper’s claim of a 90-95% reduction in token use likely stems from the fact that LLMs only generate guidance in the first phase. If the algorithm doesn't improve in the second phase, the improvement in token efficiency is less compelling. I recommend plotting the training curve and marking the transition to the second phase on the curve to highlight the effectiveness of the two-stage method.

2. ActionNet may introduce an unfair comparison: The paper uses ActionNet to translate options into low-level actions via a pre-defined mapping. It’s unclear if baseline algorithms also use ActionNet; if they don’t, this could create an unfair advantage, which should be addressed in the experimental section.

questions:
I have outlined all my concerns in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553474419

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summary

comment:
We appreciate the valuable feedback provided by the reviewers, which has greatly helped us improve the quality of our work. Below, we summarize the major revisions made to the manuscript. All the major changes have been highlighted in blue. 

- Updated Figures for MiniGrid Experiments: In response to Reviewer 3Ukv’s suggestion, we have updated Figure 8 to clearly mark the transition between the imitation learning phase and the reinforcement learning phase. The updated figure now includes dashed lines to indicate the boundary between the two phases, illustrating the effectiveness of both stages. This provides a clearer depiction of how the two-phase framework operates.

- Refined Discussion of Main Contributions: We have revised the Introduction and Conclusion sections to better highlight our key contributions. In particular, we emphasize the strategic use of LLMs during the imitation learning phase, the integration of hierarchical policy-value updates, and the lightweight, token-efficient design of our approach. These revisions address comments from Reviewer hv47 and Reviewer LwQs, clarifying the novel aspects of our framework.

- Added Sensitivity Analysis: A detailed sensitivity analysis has been added to the Appendix C to address concerns about the additional hyperparameters introduced in our method (e.g., \p, \lambda_t, and \alpha). The analysis demonstrates that these hyperparameters have minimal impact on performance, highlighting the robustness of our framework and reducing the burden of fine-tuning.

We believe these revisions address the key concerns raised by the reviewers and significantly strengthen the manuscript. Thank you for the opportunity to improve our work further.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732871098057

thread_type:
official_comment

<thread_object>
creator: area_chair_c8ac

<content>
comment:
Dear Reviewers,

This is a friendly reminder that the last day that reviewers can post a message to the authors is Dec. 2nd (anywhere on Earth). If you have not already, please take a close look at all reviews and author responses, and comment on whether your original rating stands.

Thanks,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a two-stage algorithm to tackle the challenges of complex, sparse-reward environments. In the first stage, the RL agent imitates the policy generated by the LLM to improve its exploration capabilities. In the second stage, a vanilla PPO approach is applied to finetune the policy. 

The idea of converting common knowledge of LLMs into options to guide policy learning in sparse-reward tasks is novel. However, the paper has the following weaknesses. 

- Current experiments are insufficient to support the proposed method's advantage. The authors are encouraged to test on more complex environments.

- The addition of a KL-regularized pre-training phase is an incremental change to typical RL training processes, and the technical contribution might be seen as somewhat limited.

- The ablation study results suggest that PPO already performs sufficiently well, and it would be helpful to demonstrate that the new algorithm performs significantly better in scenarios where PPO would fail.