Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

Authors: Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, Brandon Amos
Primary area: reinforcement learning
Keywords: reinforcement learning, decision making, large language model, exploration, intrinsic rewards

Abstract:
Automatically synthesizing dense rewards from natural language descriptions is a promising paradigm in reinforcement learning (RL), with applications to sparse reward problems, open-ended exploration, and hierarchical skill design. Recent works have made promising steps by exploiting the prior knowledge of large language models (LLMs). However, these approaches suffer from important limitations: they are either not scalable to problems requiring billions of environment samples, due to requiring LLM annotations for each observation, or they require a diverse offline dataset, which may not exist or be impossible to collect. In this work, we address these limitations through a combination of algorithmic and systems-level contributions. We propose ONI, a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function using LLM feedback. Our approach annotates the agent's collected experience via an asynchronous LLM server,  which is then distilled into an intrinsic reward model. We explore a range of algorithmic choices for reward modeling with varying complexity, including hashing, classification, and ranking models. By studying their relative tradeoffs, we shed light on questions regarding intrinsic reward design for sparse reward problems. Our approach achieves state-of-the-art performance across a range of challenging, sparse reward tasks from the NetHack Learning Environment in a simple unified process, solely using the agent's gathered experience, without requiring external datasets.
We make our code available at \url{URL} (coming soon).

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730418423636

thread_type:
official_review

<thread_object>
creator: reviewer_c2kc

<content>
summary:
ONI introduces a novel distributed framework for developing intrinsic rewards in reinforcement learning by leveraging feedback from large language models (LLMs). This approach supports policy optimization in sparse reward scenarios without needing external datasets. By exploring several methods for reward modeling, ONI demonstrates impressive performance in the NetHack environment, closely matching the results of previous approaches like Motif while eliminating the requirement for pre-collected data

strengths:
- ONI provides a practical and innovative way to use feedback from large language models as intrinsic rewards, making it especially valuable for reinforcement learning environments where extrinsic rewards are sparse.

- By eliminating the need for external datasets and instead building rewards from the agent’s own experience, ONI offers a scalable approach that could adapt to a wide range of tasks.

- The asynchronous design keeps the policy training process uninterrupted, maintaining high throughput and efficiency throughout.

- ONI shows strong, competitive performance in challenging tasks within the NetHack environment, matching or even surpassing state-of-the-art methods in tasks like Oracle and multi-level exploration without relying on dense reward functions.

- Enough details are provided to reproduce the method. The method reproducibility appears to be good.

weaknesses:
- The method is primarily evaluated in the NetHack environment, leaving questions on how well ONI would generalize to other, potentially more diverse RL environments. It would be critical to show that the method can be applied to other environments and domains. In the current state, evaluation is too narrow to demonstrate the impact of this work.

- Asynchronous LLM feedback, while effective here, may face scalability challenges in more complex settings where feedback needs scale up significantly. The paper lacks detailed discussion on this issue. 

- The paper should discuss and report evaluation with different LLM. In the current state, the results cannot demonstrate that the method can be used with other LLMs. 

- LLM-generated annotations may carry inherent biases that can influence reward generation. The paper lacks a strategy for identifying or mitigating such biases.

- Comparisons with other intrinsic reward models and exploration techniques are minimal, limiting clarity on ONI's unique advantages or limitations relative to a wider range of baselines. I would suggest for instance comparing current paper with strong LLM-guided baselines, such as [“Guiding Pretraining in Reinforcement Learning with Large Language Models”, Du et al., 2023], [“Exploring Beyond Curiosity Rewards: Language-Driven Exploration in RL”, Bougie et al., 2024], or other recent papers.

questions:
- How would ONI perform in other environments beyond NetHack, especially ones with more complex or variable observation spaces?

- Are there any insights into how specific LLM choices impact ONI's performance, especially for larger models?

- Could potential biases in LLM annotations affect the intrinsic reward generation process, and if so, how are they mitigated?

- How does the method compare with stronger recent baselines that guide exploration with LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622769415

thread_type:
official_review

<thread_object>
creator: reviewer_cv7n

<content>
summary:
This paper introduces ONI, a distributed architecture that simultaneously learns a RL policy and an intrinsic reward function using LLM feedback. ONI is built upon Sample Factory, an asynchronous reinforcement learning framework, and incorporates an LLM server hosted on a separate node to annotate observations, facilitating the learning of an intrinsic reward model that improves the sample efficiency of RL algorithms in sparse reward environments. This paper evaluates several designs of the intrinsic reward function and demonstrates ONI’s strong performance in the NetHack Learning Environment.

strengths:
- The paper is well-written, and the method is clearly introduced.
- The limitations of previous approaches are effectively summarized, and the problem investigated is significant.

weaknesses:
- My primary concern lies in the novelty of the contribution. The proposed method, ONI, appears heavily intertwined with Sample Factory[1], adding only a minor extension. The distributed architecture should, in my view, be credited to Sample Factory rather than this work. Furthermore, both the LLM-based annotation and the intrinsic reward design have been previously introduced in existing research[2,3]. As a result, I believe this paper may not be suitable for acceptance at top-tier venues.
- Can the assumption of access to observation captions be relaxed, perhaps by leveraging a multi-modal LLM?
- The experimental domain is limited to the NetHack Learning Environment. Could the method be extended to other widely-used RL benchmarks, such as MuJoCo[4] or Atari[5]?
- The authors noted that increasing LLM annotations did not significantly improve performance. Could computational resources be optimized by reducing the number of annotations? What is the minimum level of annotation needed to maintain performance?
- The discussion on the method's limitations remains insufficient.
- The code is not available.

[1] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7652–7662. PMLR, 2020. URL http://proceedings.mlr.press/v119/ petrenko20a.html.

[2] Lee H, Phatale S, Mansoor H, et al. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback[C]//Forty-first International Conference on Machine Learning.

[3] Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166, 9 2023.

[4] Todorov E, Erez T, Tassa Y. Mujoco: A physics engine for model-based control[C]//2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012: 5026-5033.

[5] Bellemare M G, Naddaf Y, Veness J, et al. The arcade learning environment: An evaluation platform for general agents[J]. Journal of Artificial Intelligence Research, 2013, 47: 253-279.

questions:
Please take a look at the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638551057

thread_type:
official_review

<thread_object>
creator: reviewer_9m6i

<content>
summary:
The paper proposes a method to simultaneously learn an RL policy and intrinsic reward function using feedback from LLMs, which annotate the agent’s experiences. The method is shown to outperform competing approaches in the NetHack environment.

strengths:
ONI seems to be a novel approach with promising results for learning intrinsic rewards. Leveraging LLMs to learn or infer reward functions is an interesting and relevant contemporary topic.

weaknesses:
In general, the writing could be improved for clarity. There are some sections with unnecessarily long sentences, which impeded understanding. The work could be scoped out better to clarify exactly what the contributions are.

questions:
1.	What is the impact of limited annotation on the reward model? Also what is the impact of the quality of annotations?
2.	The lack of difference in performance between ranking and classification methods could be investigated further. Analyzing the distribution of intrinsic rewards generated by each method or comparing their performance on tasks with higher observation diversity could offer further insights. Have the authors considered this?
3.	It would be good to scope out clearly what the method is trying to achieve. For example, is the intrinsic reward used during learning fully learnt or is it just a current estimate?
4.	Is there a difference in the final reward model learnt via ONI and one learnt using say, a good amount of offline data? Does the reward from ONI converge? I wonder whether theoretical properties relating to this have been studied.
5.	The logic behind eq 6 should be clearly and explicitly mentioned immediately after the equation.
6.	Although it may not be directly relevant, it would be reassuring to include additional baselines (or modified versions of it) based on the reported related work.
7.	What does ONI stand for?
8.	The paper has some unusually long sentences (such as in the abstract), which should be split up for simplicity and clarity.
9.	In section 3, it would have been better to focus on the proposed approach rather than elaborating on the details in say, lines 142-148 for example. These details are not central to the understanding of ONI and could have been pushed to the appendix.
10.	URL in the abstract does not work
11.	$\eta$ is not rendered properly in Fig 5.2
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677091451

thread_type:
official_review

<thread_object>
creator: reviewer_4gbj

<content>
summary:
This work presents a distributed system and an online learning algorithm for developing a reinforcement learning policy that learns from an intrinsic reward function, which is trained/obtained using feedback from LLMs. The approach depends solely on online experience, removing the need for external datasets or source code, and is assessed in the NetHack environment, comparing its performance to that of the Motif algorithm.

strengths:
1. The proposed research problem—scaling up LLMs to generate dense intrinsic rewards for online learning—is compelling and significant for advancing LLM agents' ability to tackle complex, long-horizon tasks in real-world applications without assumption of access to expert demonstration.

2. The paper demonstrates strong engineering effort, employing online learning with help of LLMs to train an agent with up to 2 billion environment steps, a notable advance compared to prior work that required either only a few iteration of online training or a large offline dataset.

3. The experimental design effectively supports the method’s claims. The chosen testbed is well-suited, as its long-horizon nature necessitates LLM-generated intrinsic rewards to achieve successful outcomes.

weaknesses:
1. (main) The authors compare their work only with the Motif baseline algorithm. It would be beneficial to include comparisons with goal-conditioned reward design approaches as well. In the related work section, specifically in the goal-conditioned reward design subsection, the authors list relevant studies but do not critique their limitations or highlight the advantages of their approach. Adding these comment, as done in other subsections, would strengthen the discussion.

2. In Figure 5.4, it is surprising that a higher number of annotations does not improve performance. The author might conduct an ablation study to assess the relationship between annotation volume and performance—reducing annotated data until a performance drop is observed. This may reveal that fewer labeled data points are sufficient for training the reward function, potentially reducing computational resources.

3. Using HTTP for communication between the LLM server and the annotation process may incur high communication costs. Although the authors claim minimal overhead in line 189, an experiment to demonstrate this would provide clearer evidence.

questions:
I have outlined all my concerns in the weaknesses section. I would be happy to discuss further if the authors can address my questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732597543638

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response

comment:
We are grateful to all reviewers for their detailed and constructive feedback on our paper, and we are encouraged to see that reviewers find: 


- Our proposed research problem — utilizing LLMs to generate online dense intrinsic rewards — is compelling and significant for advancing LLM agents' ability to tackle complex, long-horizon tasks in real-world applications (Reviewer 4gbj, 9m6i, Cv7N, C2KC)
- ONI eliminates the need for external datasets and instead building rewards from the agent’s own experience, ONI offers a scalable approach that could adapt to a wide range of tasks (Reviewer 4gbj, C2KC)
- Our paper demonstrates strong engineering effort, the asynchronous design keeps the policy training process uninterrupted, maintaining high throughput and efficiency throughout (Reviewer 4gbj, C2KC)



We have added a number of experiments in response to reviewer comments. We summarize these here, and provide responses to individual reviewers below. 

* **Additional Baselines:** Reviewers **4gbj**, **9m6i** and **C2KC** all asked to see additional baseline comparisons, specifically to a goal-conditioned baseline. First, we would like to note that the specific LLM-based baselines suggested (such as the ELLM method of [Du et al, 2023]) are not computationally feasible in our setting, since they require an LLM call for each observation —and observations are in the billions here. Therefore, we implemented a modified version which does scale to our setting, where we replace the LLM embedding with a bag-of-words embedding using FastText, and also add an episodic term to the intrinsic reward. This reveals several interesting findings:
    * Directly using ELLM-BoW as in Du et al. 2023, without the episodic term, completely fails, highlighting that this episodic term is crucial. We present the results in Appendix C.6.
    * With our modification, ELLM-BoW provides strong performance on the Oracle and Staircase tasks, almost matching Motif and ONI without using an LLM. This shows that an LLM may in fact not be needed for these particular tasks, and simpler text processing can be sufficient, see Figure 5.1. 
    * Although ELLM-BoW matches ONI on Oracle and Staircase, it does worse than ONI on tasks where more sophisticated goal understanding is required. For example, we tested ELLM-BoW and ONI with two opposite goals: “collect gold, but do not kill monsters” vs “kill monsters, but do not collect gold”. ELLM-BoW fails to understand the semantic differences and results in agents with similar behavior, while ONI is able to capture the semantic differences and the obtained agents have different focuses. This stems from the nature of the bag-of-word embedding, which unfortunately is the only scalable variant, as we mentioned before. This shows that the LLM is indeed needed as the goal complexity increases. We also tried more sophisticated sentence transformers instead of bag-of-words embeddings, but these hurt the throughput too much (a 20-100x slowdown). 
    * The takeaway from these experiments is that ONI is the only method which can i) scale to high-throughput settings, ii) operate purely online, without offline data, and iii) handle complex goal specifications which require an LLM.

* **Annotation Ablation:** Reviewers **4gbj**, **9m6i**, **Cv7N** all asked for more details about how performance is affected as we decrease the annotation budget. We have added an additional set of ablations where we significantly reduce the number of annotations (by a factor of 0.1 and 0.01). We find that ONI-retrieval significantly drops in performance, whereas ONI-classification remains relatively robust. This provides more evidence for the utility of the parametric reward model, since it can generalize from a few annotations and reduce the need for annotation. 

We have also made some changes to the ranking algorithm: specifically, we found that with additional hyperparameter tuning, the differences between the offset and threshold variants become small. Therefore, we decided to stick with the threshold variant in order to introduce fewer changes compared to offline Motif. 



We believe have addressed all the questions raised by reviewers with additional experiments or thorough clarifications via separate responses to each reviewer. Please let us know if you have other questions, and we are happy to discuss them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732638246345

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response II -- System

comment:
Dear Reviewers,

Reviewer Cv7N brought up one concern that our system contribution is trivial, as it seems a simple add-on to sample factory. After some thoughts, we decided to write a common response for clarification, as we think the system contribution is important and need to be discussed.

To start, even though we are not implementing Sample Factory from scratch, the engineering effort for adding this async LLM sever and async reward model training is nontrivial. There are multiple difficulties that need careful design and treatment, otherwise the training would not scale or crash. Here are a few example questions we need to answer through the development of our codebase:

- **How to setup the LLM server? Should we use a child process or setup a remote server?** Spawning a child process and invoke a VLLM instance is the easiest solution, however, this means the APPO code and LLM code need to run on a same node, which prevents the use of large-size LLM e.g. 70B LLaMa that requires a single node's compute itself. To make the framework more general and enable future extensions to large VLMs, we chose the latter solution, which is way more difficult.

- **In which thread should we update the reward model? How to keep the LLM's annotation throughput as high as possible?** If we use the feedback processing thread that talks with LLM to update the reward model, it stalls the communication with LLM server and hurts the annotation throughput; but updating the reward model in the training thread will quickly overfit the small set of annotated data in the beginning. Therefore, we choose to tolerate the paused communication with LLM server for a short period of time -- updating the reward model in the feedback processing thread until we collect 25k annotations. 

- **How to enable thread-safe annotation data reading & writing, thread-safe reward model training and thread-safe checkpointing?** 
Due to the nature of async training, there are 2+ threads reading & writing the annotation hash table, 2+ threads updating the reward model & evaluating it to obtain intrinsic rewards for policy training. We need to make these operations thread-safe. Also, as the training takes up to 25+ hours, we need to save our models periodically outside the training/feedback processing thread. Unfortunately, neither torch.save() nor python pickle is thread-safe. Our system needs multiple thread locks that cover these 3 aspects, we need to carefully design & code them, also preventing deadlock. 

- **How to enable experiments on a Slurm managed cluster using such remote framework? How to enable debugging in an asynchronous environment?** This might sound easy, but in fact, it is more challenging than we initially thought, especially if you want to manage so many experiments where training an agent for a single NetHack task takes 25+ hours, with 2B env steps. We have 5 tasks (1 dense reward, 3 sparse reward, 1 reward free), 3 ONI methods, and 2 baselines (Motif, ELLM-BoW). We ended up with using heterogeneous jobs (https://slurm.schedmd.com/heterogeneous_jobs.html) that launch LLM server first, then launch the main training code with the URL address of the LLM server. As mentioned before, managing such large scale experiments is non-trivial, the volume of the experiments presented in our paper is large. We need to carefully code the logging of the LLM server and APPO server for debugging. 

Finally, for debugging in an asynchronous environment, the first set of questions are **How to attach a debugger to a python subprocess? Will it stall the computation for other subprocesses?** -- more questions are coming. The answers to those questions, at least, are not obvious to us in the beginning. We are using ForkedPdb suggested here: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess, which sometimes works but is also hard to use as other subprocesses will flush the IO. Even now, we don't have a perfect solution for python asynchronous training. (If reviewers have good suggestions, please do let us know.)

To summarize, we want to let reviewers know the engineering difficulty & effort we put are both nontrival. We have spent a lot of time to make sure the code quality is high. We did not discuss this too much in the paper, in order to put the paper's focus on the intrinsic reward design. We are working on refining the code and open source it later.  We strongly believe the system work is an important and valuable step, as it paves the road to later research in this direction. Therefore, we also (really!) hope and appreciate if you can see the long-term value of our work.

Best,
Paper 5187 authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission presents a framework for incorporating LLMs into an online RL setup by learning an intrinsic reward function concurrently with the policy. The idea is to asynchronously gather LLM annotations and distill them into a learned reward model, enabling sparse-reward problems to be solved more effectively. The authors compare three approaches (retrieval, classification, ranking) in the NetHack environment and claim these methods remove the need for pre-collected datasets while achieving strong performance.

The general idea of using LLM-based annotations for dense intrinsic reward design is interesting. However, there were concerns about lack of novelty and incremental contributions. The underlying innovations in system design are minimal compared to prior distributed RL architectures, and the method’s reliance on standard RL building blocks leaves little conceptual novelty. This is fine if empirical support outweighs the limitations. This work is narrowly evaluated on the NetHack environment. While challenging, this environment has a specific observation space.  It is not clear whether the proposed approach generalizes to more widely used RL benchmarks. This is its biggest limitations and why it didn't end up getting better scores.