Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

## Abstract

Abstract Automatically synthesizing dense rewards from natural language descriptions is a promising paradigm in reinforcement learning (RL), with applications to sparse reward problems, open-ended exploration, and hierarchical skill design. Recent works have made promising steps by exploiting the prior knowledge of large language models (LLMs). However, these approaches suffer from important limitations: they are either not scalable to problems requiring billions of environment samples; or are limited to reward functions expressible by compact code, which may require source code and have difficulty capturing nuanced semantics; or require a diverse offline dataset, which may not exist or be impossible to collect. In this work, we address these limitations through a combination of algorithmic and systems-level contributions. We propose ONI , a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function using LLM feedback. Our approach annotates the agent’s collected experience via an asynchronous LLM server, which is then distilled into an intrinsic reward model. We explore a range of algorithmic choices for reward modeling with varying complexity, including hashing, classification, and ranking models. By studying their relative tradeoffs, we shed light on questions regarding intrinsic reward design for sparse reward problems. Our approach achieves state-of-the-art performance across a range of challenging, sparse reward tasks from the NetHack Learning Environment in a simple unified process, solely using the agent’s gathered experience, without requiring external datasets nor source code. We make our code available at URL (coming soon).

## 1 Introduction

Reward functions are central to reinforcement learning (RL), and are often assumed to be given as part of the problem definition (Sutton & Barto, ).
These functions are written to describe the task at hand, and often involve tradeoffs between ease of task definition and ease of policy optimization. For example, assigning a reward of $+1$ for solving the task and 0 0 otherwise is simple to define and accurately reflects the task goal, but is difficult to optimize due to providing zero gradients almost everywhere.

These difficulties have motivated the use of intrinsic rewards to aid policy optimization
(Randlov & Alstrøm, ; Ng et al., ; Sorg et al., ; Singh et al., ). The reward designer can include additional reward shaping terms to create a denser learning signal, which can reflect task progress or guide the agent towards intermediate goals. However, designing intrinsic rewards can be remarkably challenging (Booth et al., ; Ibrahim et al., )
and places increased demands on human experts to provide task-specific knowledge.

Recently, several works have been proposed to leverage the vast prior knowledge encoded in large language models (LLMs) to automate the reward design process, based on a task description in natural language.
They can be broadly categorized into two families:

- 1. Generating the reward function’s code by LLM.
A number of methods have been proposed to automatically generate
executable code that computes the reward directly (Ma et al., ; Xie et al., ; Yu et al., ; Li et al., ).
While they have demonstrated success in complex continuous control tasks, they either require access to environment source code to include in the prompt, or a detailed description of input parameters and reward function templates.
Furthermore, they are limited to reward functions compactly expressible via code, describing explicit logic; and it is unclear how these approaches can easily process high-dimensional state representations such as images, or semantic features such as natural language.
- 2. Generating reward values by LLMs.
Motif (Klissarov et al., ) is a typical example of this category.
It ranks the captions of pairs of observations using an LLM and distills these preferences into a parametric reward model. Motif does not require access to environment source code nor numerical state representation, can process semantic input features, and can scale to problems requiring billions of environment samples.
Nevertheless, it also suffers from two important limitations. First, it requires a diverse, pre-existing dataset of captioned observations which are used to elicit preferences from the LLM. In many situations, such a dataset might not exist, and collecting it can increase the sample complexity.
More importantly, collecting a diverse dataset often requires a non-trivial reward function that is feasible to optimize,
which is the primary problem we aim to solve with intrinsic reward functions in the first place.
Second, it involves a complex three-stage process, which sequentially annotates observations using an LLM, trains a reward model, and finally trains an RL agent. This is still time-consuming, given that the LLM annotation process can take several days’ worth of GPU hours,
and is done prior to training the reward model and RL agent.
Alternatively, Chu et al. () query the LLM to directly label observations as having high or low reward at each timestep.
However, querying an LLM for every observation is computationally infeasible for many RL applications, which involve
millions or billions of observations.

As a consequence, it would be desirable to have *an integrated solution* that offers:

- (1)
*concurrent and fast online learning of both the intrinsic rewards and the policy
that requires no external data nor auxiliary reward functions,*
- (2)
*expressible reward functions that can capture semantic features
that are difficult to process with compact executable code.*

In this work, we present ONI, a distributed online intrinsic reward and agent learning system.
ONI assumes access to captions of observations, similar to previous work Klissarov et al. (); Chu et al. ().
The captions of collected observations are annotated by an asynchronous LLM server, and both the policy and intrinsic reward model are simultaneously updated using LLM feedback.
ONI removes the dependency on external datasets beyond the agent’s own experience and enables large-scale RL training with ease. Such a learning framework allows us to systematically compare different algorithmic choices for synthesizing LLM feedback. Specifically, we explore three methods: the first one is retrieval-based and simply hashes the annotations;
the second builds a binary classification model to distill the sentiment labels returned by the LLM;
and the third sends pairs of captions to the LLM server for preference labeling and learns a ranking model, similar to Motif. By carefully comparing the three proposed algorithms, we provide valuable insights into several important questions regarding intrinsic reward design.
We demonstrate that ONI is able to match Motif’s performance across a range of challenging, sparse rewards from the NetHack Learning Environment (NLE) (Küttler et al., ), solely using the agent’s gathered experience in a single, unified process.

## 2 Background

We consider a partially observed Markov decision process (POMDP) setting where the problem is defined by $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},p_{0},P,O,r,\gamma)$.
At each episode, an initial state $s_{0}\in\mathcal{S}$ is sampled from the initial state distribution $p_{0}$.
At each time step $t$, the agents observes $o_{t}\in\mathcal{O}$
which is computed by the emission function $O(s_{t})$, and takes an action $a_{t}\in\mathcal{A}$.
This action causes the environment to transition to a new state, $s_{t+1}\sim p(s_{t},a_{t})$. A new observation $o_{t+1}$ and a reward $r_{t+1}=r(o_{t+1})$ is given to the agent, and the process continues. The goal of the agent is to learn a policy $\pi:\mathcal{O}\rightarrow\Delta(\mathcal{A})$ which maximizes the expected return $\mathbb{E}_{\pi}\left[\sum_{t}\gamma^{t}r_{t}\right]$. In this work, we additionally assume each observation $o_{t}$ includes a textual caption $c_{t}$, which could be empty.
For observation spaces without textual captions, $c_{t}$ could in principle be provided by a captioning model as well.

In many situations, the extrinsic environment reward $r$ is sparse, and the resulting return objective is challenging to optimize. We therefore consider methods which make use of an auxiliary intrinsic reward $r^{\mathrm{int}}$ and define a composite surrogate reward:

$$ $\bar{r}(o_{t})=r(o_{t})+\beta\cdot r^{\mathrm{int}}(o_{t}).$ (1) $$

A key research question is how to define or learn the intrinsic reward $r^{\mathrm{int}}$.
A first option is to manually define $r^{\mathrm{int}}$ based on task-specific goals, for example a measure of the distance between the agent’s current state and the goal state. However, handcrafting the intrinsic reward function can require significant domain knowledge and must be redone for each new task.
A second option is to define $r^{\mathrm{int}}$ to measure some notion of observation novelty, which encourages the agent to systematically explore the environment. This can work well in smaller environments, but fails in ones that cannot be exhaustively explored in a tractable amount of time.
A third class of methods, which we focus on in this work, leverage LLMs to automatically synthesize $r^{\mathrm{int}}$ to reflect prior knowledge about the task. We discuss all three classes of methods in Section 4.

## 3 Online Intrinsic Rewards

### 3.1 System design: distributed PPO with LLM annotations

This section outlines the system we have built to learn online intrinsic rewards
alongside the policy optimization.
The engineering and design here is an important piece of our research,
as the rest of our experimental studies are conducted within this system
and influenced by the throughput of its interacting components.

Our core system illustrated in Figure 3.1
is built on top of the Sample Factory library v1.0 (Petrenko et al., )
and their asynchronous variant of proximal policy optimization
(Schulman et al., ), referred to as APPO.
APPO operates on a single machine and concurrently runs many
environment instances while asynchronously updating policy and value
estimates with the usual PPO rules, and
adequately handles policy staleness and data transfers between these components.
Concretely for NetHack, by running 480 environment instances
APPO collects approximately 32k
environment interactions per second on a Tesla A100-80GB GPU with 48 CPUs.

Figure: Figure 3.1: Overall system diagram of ONI. Our additions to Sample Factory are highlighted in red. We added an asynchronously executing LLM server and learned reward function, and connect them back into the main learning process in a way that does not hurt the overall throughput of the policy and value learning.
Refer to caption: x1.png

For online reward learning via LLM annotations in this system(^1^11 The prior work by Klissarov et al. () in the offline setting was able to connect their intrinsic reward function into Sample Factory’s APPO implementation with a few lines of code, as it just needed to load the PyTorch module of the learned reward function.),
we added 1) an LLM server hosted on a separate node,
2) an asynchronously running process that passes observation captions to
the LLM server via HTTP request,
3) a hash table that stores the captions and corresponding LLM annotations, and
4) and learning code that dynamically updates a reward model
that learns on these annotations.
Without being asynchronous, these components
have the potential to block the concurrent execution and
can reduce the throughput of the overall system,
because calling into an LLM and updating the reward model are time-consuming.
We added them in a way that retains most of the throughput (!),
approximately 80-95% of the original:
the average throughput is 30k environment interactions per second if we do not train
any additional reward model to distill the LLM annotations,
and 26k if a classification-based reward model is learned at the same time (see Section 3.2 for the description of those approaches).
The throughput of LLM annotation depends on the actual LLM and prompt we are using.
For the reader’s reference, when hosting a LLaMA-3.1-8B server on a Tesla A100-80GB GPU
using the prompt displayed in Appendix B, ONI annotates
approximately 810k NetHack captions over the whole training process (2B environment steps, 19-20 hours).
The rest of this section overviews the relevant components and
how we have modified them.

#### Background: APPO’s Distributed Execution and Shared Memory

APPO has three main types of workers:

- 1.
the learner worker that coordinates most of the operations, updates
the policy and value models, and sends the latest policy ID to the
other workers so that they can retrieve it;
- 2.
the rollout workers that run copies of the environment, execute
actions in them and save the observations; and
- 3.
the policy workers that query the policy on new states.
The policy workers are separate from the rollout workers so they can efficiently batch
across observations.

The workers here are individual processes forked off from a parent process.
These all have shared CPU and GPU memory buffers, and
efficiently communicate mostly via Python’s multiprocessing queues,
using pointers to locations in the shared memory.
We next describe our new LLM worker, and how we connect it back into the learner.

#### Our New Asynchronous LLM Worker and Remote LLM Server

This worker has input and output queues
that communicate
back with the learner process, which we use so it does not block the
main execution of the system.
The LLM worker awaits new observations to label from the input queue,
formats them into
prompts, calls into an LLM, and returns the annotations in the
output queue.
Additionally, the prompt templates and other LLM options are configured by the LLM worker.
We use an annotation and message format that support all the intrinsic reward
labels that we consider in Section 3.2.
As Sample Factory already utilizes most of the free CPU and GPU
capacity of the system, we opted to call into the LLM via an
HTTP/REST interface rather than loading it in this process directly.
The communication here adds minimal overhead, and also makes us not
need to coordinate the shared memory of multiple GPUs between the main
APPO code and LLM.
Alongside every APPO run, we allocate an unloaded machine
with a new instance of VLLM (Kwon et al., ) to exclusively
serve the queries from the RL run.

One important design decision is that the asynchronous LLM server is only able to
process a small percentage of the overall observations encountered (^2^22often $\leq 0.04\%$ for a LLaMA-3.1-8B instance using a single A100-80GB GPU, or $\leq 0.02\%$ when using a V100-32GB GPU),
because the environment rollout workers have a much higher throughput than the LLM.
An alternative design would be to block the main APPO components and wait
for the LLM to label more messages, but we find it more realistic to continue
running the policy and label messages as the LLM throughput allows.
This also creates the problem of needing to decide on what messages to
send to the LLM: the last-in-first-out queue (LIFO), or some uncertainty-based selection.
For this work, we use LIFO for simplicity, but note that it would be interesting to investigate alternate approaches in future work.

#### Our Modified Learner

Lastly, we connect this new LLM worker back into the main learner,
and dynamically learn a reward model on the annotations obtained from it.
To do this, we modified the two threads of the learner worker:

- 1.
the training thread is the main thread that
a) aggregates the new trajectories from the latest environment interactions;
b) decides which new observations to send to the LLM;
c) updates the policy, value, and (now) our intrinsic reward model.
- 2.
The feedback processing thread (in fact, the initial thread of the learner worker) receives the latest
annotations from the LLM worker and updates the dataset (or hash table)
that the reward model is trained on.
This thread also initializes the reward model at the beginning.

To prevent overfitting the limited amount of annotations in the early stage of training,
the training thread only continuously updates the reward model after we receive $25$k annotations.
Before that, the feedback processing thread run a few updates of the reward model every time we receive new annotations.

Now that we have a flexible way of annotating messages and fitting a reward
model on top of them, we turn to defining the types of intrinsic reward functions.

### 3.2 Intrinsic Reward Functions

ONI offers flexible algorithmic choices for querying an LLM and distilling its feedback. In this work, we consider the following
three methods.

#### Retrieval

The first and simplest approach we consider is based on binary labeling and retrieval.
The LLM is asked to assign a binary label $y_{i}\in\{0,1\}$ indicating whether a caption $c_{i}$ is “helpful” or “unhelpful” for making progress on the task. The learner worker maintains a hash table $\mathcal{H}$ to store labeled pairs $(c_{i},y_{i})$, managed by the feedback processing thread.
In the training thread, each time the RL agent receives an observation $o_{t}$ with caption $c_{t}$, we check if $c_{t}\in\mathcal{H}$ and
the intrinsic reward is defined as:

$$ $r^{\mathrm{int}}(o_{t})=\begin{cases}\mathcal{H}(c_{t})&\mbox{ if }c_{t}\in \mathcal{H}\\ 0&\mbox{ if }c_{t}\notin\mathcal{H}\end{cases}$ (2) $$

If $c_{t}$ is unlabeled, it is placed into a last-in-first-out (LIFO) queue $Q$ managed by the LLM annotation process,
and then sent to the LLM server.
The LLM continuously processes elements $c_{i}$ from $Q$ and returns their labels $y_{i}$ to the data processing thread of the learner worker, where the pairs $(c_{i},y_{i})$ are added into $\mathcal{H}$. As described in Section 3.1, the training thread and the feedback processing thread run asynchronously. Therefore, editing the hash table does not slow down policy training. This retrieval-based approach does not generalize to observations with unlabeled captions. However, the resulting intrinsic reward is simple and hyperparameter-free, and may work well when the set of captions belongs to a relatively small set.

#### Classification

The second approach we consider is based on binary labeling together with training a classification model. Similarly to above, we label observation captions $c_{i}$ with labels $y_{i}\in\{0,1\}$ indicating whether they are helpful or unhelpful via LLM.
We simultaneously train a binary classification model
to predict $y_{i}$ from $o_{i}$. More precisely, we model $P(y=1|o)$ by $r^{\mathrm{int}}_{\phi}:\mathcal{O}\rightarrow[0,1]$,
which is then used to compute the binary intrinsic reward by thresholding it at $\eta$:

$$ $r^{\mathrm{int}}(o_{t})=\mathbb{I}[r^{\mathrm{int}}_{\phi}(o_{t})>\eta],$ (3) $$

where $\mathbb{I}$ is the indicator function. We study the impact of $\eta$ in Section 5.2. Unlike the previous approach, this method has potential to generalize to observations whose captions are similar, but not identical, to the captions labeled by the LLM. However, like the previous approach, it will assign a same reward to observations which are slightly positive (such as finding a few gold pieces in NetHack) and very positive (finding hundreds of gold pieces or a rare artifact).

#### Ranking

The third approach we consider is based on ranking observations via pairwise classification, which is the approach taken by Motif. Here, pairs of observations $(o_{1},o_{2})$ are sent to the LLM, which returns a preference label $y\in\{1,2,\varnothing\}$ indicating whether $o_{1}$ or $o_{2}$ is more desirable for accomplishing the task, or if they are equivalent.
A reward model $r^{\mathrm{int}}_{\phi}:\mathcal{O}\rightarrow\mathbb{R}$ is trained by minimizing the negative log-likelihood :

$$ $\displaystyle-\mathbb{E}_{(o_{1},o_{2},y)\sim\mathcal{H}}\Big{[}\Big{(}\mathbb {I}[y=1]+\tfrac{1}{2}\mathbb{I}[y=\varnothing]\Big{)}\log P_{\phi}(o_{1}\succ o _{2})+\Big{(}\mathbb{I}[y=2]+\tfrac{1}{2}\mathbb{I}[y=\varnothing]\Big{)}\log P _{\phi}(o_{1}\prec o_{2})\Big{]}$ (4) $$

where we use average log-likelihood when $y=\varnothing$(^3^33In the equivalent cross entropy minimization formulation, this amounts to using a uniform target $[\frac{1}{2},\frac{1}{2}]$ when $y=\varnothing$.)
and use the Bradley-Terry model Bradley & Terry ()
$P_{\phi}(o_{1}\succ o_{2})=1-P_{\phi}(o_{1}\prec o_{2})=\exp\big{(}r^{\mathrm{
int}}_{\phi}(o_{1})\big{)}/\big{[}\exp\big{(}{r^{\mathrm{int}}_{\phi}(o_{1})}
\big{)}+\exp\big{(}r^{\mathrm{int}}_{\phi}(o_{2})\big{)}\big{]}$.
Motif computes the mean $\mu_{\mathcal{D}}$, standard deviation $\sigma_{\mathcal{D}}$, and a fixed quantile $\nu_{\mathcal{D}}$ of $r^{\mathrm{int}}_{\phi}$ over the offline dataset of annotations $\mathcal{D}$. During RL training, it normalizes and thresholds $r^{\mathrm{int}}_{\phi}$ to give the reward

$$ $r^{\mathrm{int}}_{\text{motif}}(o_{t})=\mathbb{I}[(r^{\mathrm{int}}_{\phi}(o_{ t})-\mu_{\mathcal{D}})/\sigma_{\mathcal{D}}>\nu_{\mathcal{D}}]\cdot(r^{\mathrm {int}}_{\phi}(o_{t})-\mu_{\mathcal{D}})/\sigma_{\mathcal{D}}$ (5) $$

For ONI-ranking, applying these steps directly is not possible since the annotation dataset is continuously changing.
Therefore, we replace $(\mu_{\mathcal{D}},\sigma_{\mathcal{D}})$ by a running mean and standard deviation $(\mu,\sigma)$ computed over the agent’s experience, and replace the quantile $\nu_{\mathcal{D}}$ by a quantile of the standard normal distribution.

## 4 Related Work

#### Exploration Based on Novelty Bonuses

Learning from sparse or otherwise difficult-to-optimize reward functions is a long-standing problem in reinforcement learning.
There is a large body of work which defines intrinsic rewards based on novelty bonuses (Schmidhuber, ; Kearns & Singh, ; Brafman & Tennenholtz, ; Stadie et al., ; Bellemare et al., ; Pathak et al., ; Burda et al., ; Shyam et al., ; Raileanu & Rocktäschel, ; Ecoffet et al., ; Agarwal et al., ; Zhang et al., ; Henaff et al., ; Lu et al., ). These methods tend to make minimal assumptions about the task at hand, operate online without requiring external data, and sometimes come with theoretical guarantees. However, since they are fundamentally tabula-rasa, they must rediscover much of the structure in the task that might already be encoded as prior knowledge in an LLM. Therefore, they tend to have difficulty exploring environments of very high complexity, such as NetHack, in a tractable amount of time (Klissarov et al., ).

#### LLM-aided Reward Design

In addition to Motif (Klissarov et al., ), several works have sought to leverage the prior knowledge encoded in LLMs to produce intrinsic rewards.
Eureka (Ma et al., ), Auto-MC (Li et al., ), L2R (Yu et al., ) and Text2Reward (Xie et al., )
all use LLMs to generate executable code which computes intrinsic rewards from the underlying environment state, conditioned on a task description. The generated reward function code is then iteratively improved based on aggregate statistics from agents trained with the current reward.
However, a disadvantage with intrinsic rewards represented as code is that they require access to an interpretable underlying state representation, and it is unclear how to leverage non-numerical features such as those provided by unstructured language captions.
The works of Kwon et al. (); Chu et al. () also successfully used LLMs conditioned on task descriptions to directly generate binary rewards in an online manner, and did not train a reward model. This was possible due to evaluating on environments and tasks which could be solved with a relatively small number of observations, whereas we consider complex open-ended environments with billions of observations, where labeling them all with an LLM would be computationally infeasible. Also of note is the work of Wu et al. (), which additionally conditioned LLMs on user manuals to define the intrinsic reward.

#### Goal-conditioned Reward Design

A different approach to reward function design is to define rewards as the distance between the agent’s current state and the goal. For example, one line of work learns a state embedding in a self-supervised manner which converts geodesic distances in the original space to Euclidean distances in feature space (Wu et al., ; Wang et al., ; Gomez et al., ).
Another line of work of (Fan et al., ; Rocamonde et al., ; Adeniji et al., ; Kim et al., ) leverages pretrained image and text encoders, and defines rewards to be some measure of similarity between embeddings of visual observations and embeddings of textual task descriptions.
Using a question generation and answering system, Carta et al. () extract auxiliary objectives from the goal description and construct intrinsic rewards.
An interesting combination of goal-conditioned and LLM-aided reward design is the work of Du et al. (), who used an LLM to generate candidate goals which might be relevant to the environment at hand, and then used distance in embedding space to define the reward.

#### LLM for RL Broadly

Another way of leveraging the prior knowledge encoded in LLMs for decision making is to use the LLM directly as a policy. This approach has been successfully used in robotics (Ahn et al., ; Driess et al., ),
as well as open-ended exploration in MineCraft (Wang et al., ). Both settings require the LLM to operate at a higher level of abstraction, by having it call upon a set of semantically grounded skills which handle the low-level sensorimotor activity. These are in turn produced by imitation learning on expert trajectories or hardcoded APIs.
Jeurissen et al. () prompt the LLM to choose a predefined skill to play NetHack.
The prompts are constructed to represent past events, current observation, and the task description and hardcoded available skills are also included.
More references on LLMs for decision making can be found in the survey paper of Cao et al. ().

## 5 Experiments

#### Environment

We use the NetHack Learning Environment (NLE) (Küttler et al., ) as our experimental testbed, since it is one of the most challenging open-ended, long horizon and sparse reward environments available, and was also used as the main environment in the prior work we compare to. NetHack is a classic dungeon crawling game which presents a number of interesting challenges for RL agents: it is procedurally generated, requiring generalization; rewards are sparse for most tasks, requiring exploration; the environment is partially rather than fully observable; transitions are naturally stochastic; episodes are very long, requiring tens to hundreds of thousands of steps to win the game; the dynamics are highly complex, involving large numbers monsters, objects, non-player characters and other entities. Succeeding in the game requires mastering and balancing diverse behaviors including exploration, resource management, object use, combat, puzzle solving and skill progression.

#### Tasks and Metrics

We evaluate our agents in two ways: how well they are able to succeed in tasks which have a (typically sparse) extrinsic reward, and how well they are able to progress in the game using the intrinsic reward only. For the former, we use one dense reward task and three sparse reward tasks used in prior work (Küttler et al., ; Klissarov et al., ), listed below.

- 1.
The Score task treats the in-game score(^4^44[https://nethackwiki.com/wiki/Score](https://nethackwiki.com/wiki/Score)) as a dense extrinsic reward.
- 2.
The Oracle task requires finding the in-game Oracle character, which resides deep in the dungeon. The agent receives a reward of 50 if it manages to reach the Oracle, zero otherwise.
- 3.
The StaircaseLvl3 and StaircaseLvl4 tasks requires reaching the third or fourth level staircase and zero otherwise—this requires exploring multiple levels in order to find staircases which lead deeper into the dungeon, while fighting or escaping monsters to survive.

Previous work found that the Oracle task can in fact be solved in unexpected ways via reward hacking; however, this still constitutes a challenging sparse reward problem and we include it for completeness.
We also train agents with the intrinsic reward only and measure the game progress via four metrics:

*experience level,   dungeon level,   gold,   and   scout (number of unique locations explored)*.

Removing the extrinsic reward gives us a clearer picture of what the intrinsic rewards are prioritizing.

#### Methods and Hyperparameters

We instantiate ONI with the three reward functions described in Section 3.2. We name the three approaches ONI-retrieval, ONI-classification and ONI-ranking, respectively.
For policy learning, we use the Chaotic Dwarven GPT5 (CDGPT5) architecture (Myffili, ) used in prior work (Piterbarg et al., ; Kurenkov et al., ; Klissarov et al., ). Architecture details can be found in Section A.1.
All the methods are trained with two billion $(2\times 10^{9})$ environment steps. We train ONI-classification with
with classification threshold $\eta=0.5$, where the intrinsic reward coefficient is $\beta=0.1$ for the Score task and $\beta=0.4$ for all the other sparse-reward tasks and the reward-free agent. Similarly, ONI-retrieval uses $\beta=0.1$ for Score and $\beta=0.5$ for the others.
ONI-ranking is trained with $\beta=0.05$ and offset $\nu_{\mathcal{N}}=1.96$ ($97.5$-th percentile of a standard normal). Full details of the training process can be found in Section A.2.

#### LLMs

We use the LLaMA-3 herd of models (Dubey et al., ) as our LLMs. All prompts are listed in Appendix B. We initially reran the Motif baseline using the official code(^5^55[https://github.com/facebookresearch/motif](https://github.com/facebookresearch/motif)) and compared the performance of LLaMA-3.1-70B-Instruct and LLaMA-3.1-8B-Instruct (see Section C.1). We did not observe a significant difference in their performance on the Oracle or Score tasks.
Therefore, we use LLaMA-3.1-8B-Instruct in our subsequent experiments to reduce computation.

#### Baselines

We compare to agents trained with extrinsic reward alone and Motif (Klissarov et al., ). We use the default hyperparameters of Motif, listed in Section A.2.
We note that Klissarov et al. () found that other exploration methods based on novelty bonuses such as RND (Burda et al., ), NovelD (Zhang et al., ) and E3B (Henaff et al., ) did not improve over the extrinsic reward baseline on these tasks, hence we do not include them.

### 5.1 Main Results

Figure: (a) Performance on four different NetHack tasks.
Refer to caption: x2.png

#### Task Performance

We report the average performance
and $95\%$ confidence intervals computed via standard errors over $5$ seeds in Figure 1(a).
The extrinsic reward agent performs reasonably well on the dense Score task, but completely fails on the others due to reward sparsity. We find that, despite not requiring any external data, our ONI-classification agent is able to match Motif on all tasks except Oracle, where it still performs well.
We note that Motif requires pre-collecting data from the Score task even for sparse reward tasks—this assumes access to an additional dense reward function, which is an often unrealistic assumption. It also incurs additional one billion ($10^{9}$) environment samples prior to policy training.
All of our agents significantly outperform the extrinsic only baseline on the sparse reward tasks, demonstrating they are able to explore effectively while synthesizing their intrinsic rewards from online data alone.

Despite its simplicity, ONI-retrieval performs surprisingly well, and its performance is often close to that of ONI-classification. This is likely a consequence of many messages with positive valence being repeated in the early game of NetHack, such as “*You find a hidden passage*” that allows exploring the rest of the level, or “*You kill the* {monster}!” which leads to experience gain. ONI-classification, which also predicts binary rewards but is able to generalize to unseen messages, provides a modest but consistent improvement over ONI-retrieval across all environments. This suggests that learning a reward model is indeed helpful. We would expect this gap to increase in settings with added noise or caption diversity, since they increase the likelihood of observed captions being unique.

We do not observe any significant gains from using ONI-ranking over ONI-classification, despite it being more conceptually general and able to represent a continuous range of intrinsic rewards rather than binary values. This may be because our tasks take place at the very earliest part of the game of NetHack, where only a small fraction of all possible messages are observed, which would also explain the relatively strong performance of ONI-retrieval. We hypothesize that more benefits will appear in settings with higher observation diversity.

#### Game Progress of Intrinsic-Reward-Only Agents

Results for all methods trained in the reward-free setting are shown in Figure 1(b). All of our ONI methods are able to make meaningful progress across all metrics. Interestingly, different variants appear to prioritize different forms of progress: ONI-ranking performs best in terms of experience level, whereas ONI-classification performs best according to the other metrics. We include the results of Motif for reference, yet emphasize it is actually inapplicable in a truly reward-free setup since it assumes access to dense rewards.

### 5.2 Ablation Study

#### ONI-classification : the impact of the classification threshold

As described in Section 3.2, ONI-classification predicts binary rewards by modeling $P(y_{t}=1|o_{t})$ and then thresholding with $\eta$.
Instead of using binary reward, an alternative design choice is to use real valued reward

$$ $r^{\mathrm{int}}(o_{t})=P(y_{t}=1|o_{t}),$ (6) $$

where $r^{\mathrm{int}}(o_{t})\in[0,1]$.
Figure 5.2 shows that the performance of ONI-classification on the four NetHack tasks
is relatively robust to this hyperparameter $\eta$, and using $P(y_{t}=1|o_{t})$ as the reward leads to similar performance.
As the training progresses, our reward model outputs values close to 0 or 1(^6^66To increase the diversity of the captions in the training dataset, we do not requery the LLM server if a caption is already annotated before. Therefore, every caption in our dataset only has a single label, resulting in this phenomenon.),
and we hypothesize this is the reason why the final performance remains comparable.
Moreover, the natural classification threshold $\eta=0.5$ marginally outperform the other values in the reward-free setting.

Figure: Figure 5.2: (Top) Performance of ONI-classification is robust to different choices of the classification threshold $\eta$ on the four NetHack tasks. (Bottom) $\eta=0.5$ marginally outperform the other values for training intrinsic-reward-only agents. The result when using $P(y_{t}=1|o_{t})$ as reward (6) is marked with legend oni-classification (no eta).
Refer to caption: x4.png

#### ONI-ranking : the impact of sampling strategy for LLM annotation and reward training

Unlike ONI-classification that uses a LIFO queue to rank captions for LLM annotation, it is more subtle to design the most effective sampling strategy for ONI-ranking, where we need to construct pairs of captions to annotate and use for reward model training.
Here we study the effect of deduplicating captions before passing them to our pipeline. In either case, we maintain a message list $\mathcal{L}$, and sample pairs of captions $c_{1},c_{2}\sim\text{Uniform}(\mathcal{L}\times\mathcal{L})$ for annotation. The annotated message pairs are stored in another list which is sampled from uniformly when training the reward model.
In the first option, each time the agent encounters a message, we check if it is already stored in $\mathcal{L}$ and only add it if not. This is similar to the approach used in ONI-classification.
In the second option, we simply add all captions encountered by the agent into $\mathcal{L}$, regardless of whether they have already been seen before. This approach is similar to that taken by the original Motif work, which does not perform any deduplication of the offline dataset.

Figure 5.3 shows the performance and the number of captions stored in the replay buffer for both schemes. We see that for the deduplicated variant, the number of captions does not grow past a certain point (approximately $70k$ unique captions). For the non-deduplicated variant, the number of captions keeps growing linearly over time. The deduplicated variant fails to learn, which highlights the important effect which the annotation dataset can have. We hypothesize that the deduplicated variant may undersample captions that occur frequently in the agent’s experience, for which it is important to reliably estimate reward. For example, the blank message occurs very frequently during policy learning, but is only included in a small fraction of the pairs sent to the LLM for annotation, since it is sampled with the same probability as the other $\sim 70k$ other captions. In contrast, the duplicated variant samples the blank message and other frequent captions with much higher probability. Still, this remains a simple strategy, and designing more sophisticated sampling mechanisms (for example, that account for the epistemic uncertainty of the reward model) would be an interesting direction for future work.

Figure: Figure 5.3: Performance and number of captions in the replay buffer for agents trained with and without message deduplication. Keeping the natural distribution of captions, and not deduplicating, is important to enable our ranking-based method to learn.
Refer to caption: x6.png

#### Performance vs. LLM Annotation Throughput

We compared agents trained on the Score and Oracle tasks using either $1$ or $4$ Tesla V100-32GB GPUs in the LLM server node.
As shown in Figure 5.4, using 4 GPUs rather than 1 significantly increases the number of annotated observations. However, we do not find any significant change in performance between the two. Likewise, we have observed the performance is comparable when using A100-80GB GPU with higher throughput, see Section C.2. This suggests that many of the labelled examples may contain redundant information, which is not useful for updating the reward model. Designing more sophisticated prioritization schemes, which can select maximally informative examples to send to the LLM, constitutes an interesting direction for future work.

Figure: Figure 5.4: Performance remains comparable despite doubling LLM annotations, as seen in results with LLM server utilizing 1 GPU vs. 4 GPUs. The number of annotations received by ONI-ranking is lower than the other two, as the LLM server analyzes two messages for each annotation (see the prompts in Appendix B).
Refer to caption: x7.png

#### Impact of Intrinsic Reward Coefficient β 𝛽 \beta italic_β

For ONI-classification and ONI-retrieval, we have used different values of $\beta$ for the Score task and other sparse reward tasks in Section 5.1.
Throughout our experiments, we have found that
for these two methods,
larger values of $\beta$ lead to better performance for the sparse reward tasks, while smaller values of $\beta$ better balance between intrinsic and extrinsic rewards for the dense reward Score task. In comparison, ONI-ranking is relatively robust to this choice, where larger values of $\beta$ are still slightly favored for the sparse reward tasks.
We include details in Appendix C.3.

## 6 Conclusion

We have introduced ONI, a distributed online intrinsic reward and agent learning system. We showed that we are able to match the state of the art across a range of challenging sparse reward tasks from the NetHack Learning Environment, while removing the need for a large pre-collected dataset or auxiliary dense reward function required by previous work. We explored three different instantiations of our system of varying levels of complexity and generality, and study their tradeoffs. Our work paves the way for intrinsic reward methods which can learn purely from agent experience, are not constrained by external dataset size or quality, and can leverage high-performance RL training.

## Appendix A Experimental Details

### A.1 Architectures

Our architectures largely follow those used in Klissarov et al. ().

#### Our policy network

uses the Chaotic Dwarven GPT5 architecture originally introduced in Myffili (). This architecture combines convolutional layers processing the top-down visible map centered at the agent with fully-connected layers processing messages and bottom-line statistics including hit points, experience, hunger level and the like. The convolutional encoder has $3$ convolutional layers with $32,64,128$ feature maps respectively, interleaved with exponential linear unit (ELU) non-linearities (Clevert et al., ). Messages and bottom-line statistics are each processed with 2-layer MLPs with $128$ hidden units each. All embeddings are combined, passed through a single-layer MLP with $512$ hidden units, and then passed to a recurrent GRU module (Cho et al., ) with $512$ hidden units. Finally, this hidden representation feeds into linear critic and actor heads.

#### Our reward model

The reward model of ONI-ranking is based on the encoder from Küttler et al. () that processes both state representation and messages. Messages are processed by a $5$-layer character-level CNN (Zhang et al., ) with $64$ feature maps at each layer. The first, second and last layers are interleaved with max-pooling layers with kernel size and stride $3$. The output is then passed through a $3$-layer MLP with $128,256,512$ hidden units at each layer respectively, and ReLU non-linearities, followed by a scalar output. The reward model of ONI-classification only process messages, using the same architecture described above.

### A.2 Hyperparameters

Following Klissarov et al. (), we scale the environment reward by 0.1 for the Score task and by 10 for the other sparse reward tasks, and use normalized intrinsic reward

$$ $r^{\mathrm{int}}_{\text{normalized}}(o_{t})=r^{\mathrm{int}}(o_{t})/N(c_{t})^{ z},$ (7) $$

where $N(c_{t})$ is the number of times the caption $c_{t}$ has been found in one episode. For all our experiments, we use $z=3.0$.

For ONI-classification and ONI-ranking, we train the reward model using the Adam optimizer Kingma () with batch size $256$.
ONI-classification is trained with learning rate $0.0001$, classification threshold $\eta=0.7$, $\beta=0.1$ for the Score task and $\beta=0.4$ for the others.
ONI-ranking is trained with $0.00001$, $\beta=0.05$ and $\nu_{\mathcal{N}}=1.96$ (97.5-th quantile of the standard normal distribution).
ONI-retrieval does not train a reward model, and we use $\beta=0.1$ for the Score task and $\beta=0.5$ for the others.

Table A.1 shows the APPO hyperparameters which are common to all experiments and Table A.2 includes the hyperparameters for the online LLM annotation.

**Table A.1: Common APPO hyperparameters across all experiments.**
| Hyperparameter | Value |
| --- | --- |
| Number of Parallel Environment Instances | $480$ |
| Batch Size | $4096$ |
| PPO Clip Ratio | $0.1$ |
| PPO Clip Value | $1.0$ |
| PPO Epochs | $1$ |
| Max Gradient Norm | $4.0$ |
| Value Loss Coefficient | $0.5$ |
| Exploration Loss | entropy |

**Table A.2: LLM annotation hyperparameters.**
| Hyperparameter | Value |
| --- | --- |
| LLM Model | LLaMA-3.1-8B-Instruct |
| Temperature | $0.1$ |
| Max tokens to generate | $4096$ |
| Annotation Batch Size | $100$ |

## Appendix B Prompts

Here we include all the LLM prompts we use. The system prompt and the prompt for ONI-ranking are identical to the ones used in Klissarov et al. (). We adapted these to produce labels for single observations for ONI-retrieval and ONI-classification.

System Prompt (used by all methods)

Prompt for ONI-retrieval and ONI-classification

Prompt for ONI-ranking

## Appendix C Additional Results

### C.1 LLaMA-3.1-70B-Instruct vs LLaMA-3.1-8B-Instruct

In Figure C.1, we compare the performance of Motif using two different sized LLMs on Score and Oracle (LLaMA-3.1-8B-Instruct and LLaMA-3.1-70B-Instruct). Interestingly, we do not observe a significant difference between the two. This is in contrast to the previous work of (Klissarov et al., ), who found a significant difference between using LLaMA-2-70B-chat and LLaMA-2-7B-chat. This suggest that the smaller 8B model is sufficient for evaluating messages on these tasks, hence we use it in our experiments.

Figure: Figure C.1: Performance of Motif using two different LLMs. Curves represent means and shaded regions represent standard errors over $5$ seeds.
Refer to caption: x9.png

### C.2 Performance vs. LLM Throughput: Tesla V100 vs Tesla A100 GPU

Figure: Figure C.2: Performance of ONI-classification is comparable when the LLM server uses a Tesla A100-80GB or V10-32GB GPU.
Refer to caption: x10.png

### C.3 Impact of Intrinsic Reward Coefficient β 𝛽 \beta italic_β

Figure: Figure C.3: For ONI-retrieval and ONI-classification, sparse reward tasks favor larger intrinsic reward coefficient $\beta$ while smaller values of $\beta$ lead to better results for the dense reward Score task. For ONI-ranking, we do not observe much difference for the Score task, but the sparse reward tasks still slightly favors larger $\beta$.
Refer to caption: x12.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I have outlined all my concerns in the weaknesses section. I would be happy to discuss further if the authors can address my questions.
soundness: 3
strengths: 1. The proposed research problem—scaling up LLMs to generate dense intrinsic rewards for online learning—is compelling and significant for advancing LLM agents' ability to tackle complex, long-horizon tasks in real-world applications without assumption of access to expert demonstration.

2. The paper demonstrates strong engineering effort, employing online learning with help of LLMs to train an agent with up to 2 billion environment steps, a notable advance compared to prior work that required either only a few iteration of online training or a large offline dataset.

3. The experimental design effectively supports the method’s claims. The chosen testbed is well-suited, as its long-horizon nature necessitates LLM-generated intrinsic rewards to achieve successful outcomes.
summary: This work presents a distributed system and an online learning algorithm for developing a reinforcement learning policy that learns from an intrinsic reward function, which is trained/obtained using feedback from LLMs. The approach depends solely on online experience, removing the need for external datasets or source code, and is assessed in the NetHack environment, comparing its performance to that of the Motif algorithm.
weaknesses: 1. (main) The authors compare their work only with the Motif baseline algorithm. It would be beneficial to include comparisons with goal-conditioned reward design approaches as well. In the related work section, specifically in the goal-conditioned reward design subsection, the authors list relevant studies but do not critique their limitations or highlight the advantages of their approach. Adding these comment, as done in other subsections, would strengthen the discussion.

2. In Figure 5.4, it is surprising that a higher number of annotations does not improve performance. The author might conduct an ablation study to assess the relationship between annotation volume and performance—reducing annotated data until a performance drop is observed. This may reveal that fewer labeled data points are sufficient for training the reward function, potentially reducing computational resources.

3. Using HTTP for communication between the LLM server and the annotation process may incur high communication costs. Although the authors claim minimal overhead in line 189, an experiment to demonstrate this would provide clearer evidence.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.	What is the impact of limited annotation on the reward model? Also what is the impact of the quality of annotations?
2.	The lack of difference in performance between ranking and classification methods could be investigated further. Analyzing the distribution of intrinsic rewards generated by each method or comparing their performance on tasks with higher observation diversity could offer further insights. Have the authors considered this?
3.	It would be good to scope out clearly what the method is trying to achieve. For example, is the intrinsic reward used during learning fully learnt or is it just a current estimate?
4.	Is there a difference in the final reward model learnt via ONI and one learnt using say, a good amount of offline data? Does the reward from ONI converge? I wonder whether theoretical properties relating to this have been studied.
5.	The logic behind eq 6 should be clearly and explicitly mentioned immediately after the equation.
6.	Although it may not be directly relevant, it would be reassuring to include additional baselines (or modified versions of it) based on the reported related work.
7.	What does ONI stand for?
8.	The paper has some unusually long sentences (such as in the abstract), which should be split up for simplicity and clarity.
9.	In section 3, it would have been better to focus on the proposed approach rather than elaborating on the details in say, lines 142-148 for example. These details are not central to the understanding of ONI and could have been pushed to the appendix.
10.	URL in the abstract does not work
11.	$\eta$ is not rendered properly in Fig 5.2
soundness: 3
strengths: ONI seems to be a novel approach with promising results for learning intrinsic rewards. Leveraging LLMs to learn or infer reward functions is an interesting and relevant contemporary topic.
summary: The paper proposes a method to simultaneously learn an RL policy and intrinsic reward function using feedback from LLMs, which annotate the agent’s experiences. The method is shown to outperform competing approaches in the NetHack environment.
weaknesses: In general, the writing could be improved for clarity. There are some sections with unnecessarily long sentences, which impeded understanding. The work could be scoped out better to clarify exactly what the contributions are.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Please take a look at the Weaknesses section.
soundness: 2
strengths: - The paper is well-written, and the method is clearly introduced.
- The limitations of previous approaches are effectively summarized, and the problem investigated is significant.
summary: This paper introduces ONI, a distributed architecture that simultaneously learns a RL policy and an intrinsic reward function using LLM feedback. ONI is built upon Sample Factory, an asynchronous reinforcement learning framework, and incorporates an LLM server hosted on a separate node to annotate observations, facilitating the learning of an intrinsic reward model that improves the sample efficiency of RL algorithms in sparse reward environments. This paper evaluates several designs of the intrinsic reward function and demonstrates ONI’s strong performance in the NetHack Learning Environment.
weaknesses: - My primary concern lies in the novelty of the contribution. The proposed method, ONI, appears heavily intertwined with Sample Factory[1], adding only a minor extension. The distributed architecture should, in my view, be credited to Sample Factory rather than this work. Furthermore, both the LLM-based annotation and the intrinsic reward design have been previously introduced in existing research[2,3]. As a result, I believe this paper may not be suitable for acceptance at top-tier venues.
- Can the assumption of access to observation captions be relaxed, perhaps by leveraging a multi-modal LLM?
- The experimental domain is limited to the NetHack Learning Environment. Could the method be extended to other widely-used RL benchmarks, such as MuJoCo[4] or Atari[5]?
- The authors noted that increasing LLM annotations did not significantly improve performance. Could computational resources be optimized by reducing the number of annotations? What is the minimum level of annotation needed to maintain performance?
- The discussion on the method's limitations remains insufficient.
- The code is not available.

[1] Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7652–7662. PMLR, 2020. URL http://proceedings.mlr.press/v119/ petrenko20a.html.

[2] Lee H, Phatale S, Mansoor H, et al. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback[C]//Forty-first International Conference on Machine Learning.

[3] Martin Klissarov, Pierluca D’Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166, 9 2023.

[4] Todorov E, Erez T, Tassa Y. Mujoco: A physics engine for model-based control[C]//2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012: 5026-5033.

[5] Bellemare M G, Naddaf Y, Veness J, et al. The arcade learning environment: An evaluation platform for general agents[J]. Journal of Artificial Intelligence Research, 2013, 47: 253-279.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - How would ONI perform in other environments beyond NetHack, especially ones with more complex or variable observation spaces?

- Are there any insights into how specific LLM choices impact ONI's performance, especially for larger models?

- Could potential biases in LLM annotations affect the intrinsic reward generation process, and if so, how are they mitigated?

- How does the method compare with stronger recent baselines that guide exploration with LLMs?
soundness: 2
strengths: - ONI provides a practical and innovative way to use feedback from large language models as intrinsic rewards, making it especially valuable for reinforcement learning environments where extrinsic rewards are sparse.

- By eliminating the need for external datasets and instead building rewards from the agent’s own experience, ONI offers a scalable approach that could adapt to a wide range of tasks.

- The asynchronous design keeps the policy training process uninterrupted, maintaining high throughput and efficiency throughout.

- ONI shows strong, competitive performance in challenging tasks within the NetHack environment, matching or even surpassing state-of-the-art methods in tasks like Oracle and multi-level exploration without relying on dense reward functions.

- Enough details are provided to reproduce the method. The method reproducibility appears to be good.
summary: ONI introduces a novel distributed framework for developing intrinsic rewards in reinforcement learning by leveraging feedback from large language models (LLMs). This approach supports policy optimization in sparse reward scenarios without needing external datasets. By exploring several methods for reward modeling, ONI demonstrates impressive performance in the NetHack environment, closely matching the results of previous approaches like Motif while eliminating the requirement for pre-collected data
weaknesses: - The method is primarily evaluated in the NetHack environment, leaving questions on how well ONI would generalize to other, potentially more diverse RL environments. It would be critical to show that the method can be applied to other environments and domains. In the current state, evaluation is too narrow to demonstrate the impact of this work.

- Asynchronous LLM feedback, while effective here, may face scalability challenges in more complex settings where feedback needs scale up significantly. The paper lacks detailed discussion on this issue. 

- The paper should discuss and report evaluation with different LLM. In the current state, the results cannot demonstrate that the method can be used with other LLMs. 

- LLM-generated annotations may carry inherent biases that can influence reward generation. The paper lacks a strategy for identifying or mitigating such biases.

- Comparisons with other intrinsic reward models and exploration techniques are minimal, limiting clarity on ONI's unique advantages or limitations relative to a wider range of baselines. I would suggest for instance comparing current paper with strong LLM-guided baselines, such as [“Guiding Pretraining in Reinforcement Learning with Large Language Models”, Du et al., 2023], [“Exploring Beyond Curiosity Rewards: Language-Driven Exploration in RL”, Bougie et al., 2024], or other recent papers.

Meta-review: This submission presents a framework for incorporating LLMs into an online RL setup by learning an intrinsic reward function concurrently with the policy. The idea is to asynchronously gather LLM annotations and distill them into a learned reward model, enabling sparse-reward problems to be solved more effectively. The authors compare three approaches (retrieval, classification, ranking) in the NetHack environment and claim these methods remove the need for pre-collected datasets while achieving strong performance.

The general idea of using LLM-based annotations for dense intrinsic reward design is interesting. However, there were concerns about lack of novelty and incremental contributions. The underlying innovations in system design are minimal compared to prior distributed RL architectures, and the method’s reliance on standard RL building blocks leaves little conceptual novelty. This is fine if empirical support outweighs the limitations. This work is narrowly evaluated on the NetHack environment. While challenging, this environment has a specific observation space.  It is not clear whether the proposed approach generalizes to more widely used RL benchmarks. This is its biggest limitations and why it didn't end up getting better scores.

Decision: Reject