From Words To Rewards: Leveraging Natural Language For Reinforcement Learning
Authors: Belen Martin Urcelay
Andreas Krause
Giorgia Ramponi
Venue: Accepted by TMLR ()

Abstract: We explore the use of natural language to specify rewards in Reinforcement Learning with Human Feedback (RLHF). Unlike traditional approaches that rely on simplistic preference feedback, we harness Large Language Models (LLMs) to translate rich text feedback into state-level labels for training a reward model. Our empirical studies with human participants demonstrate that our method accurately approximates the reward function and achieves significant performance gains with fewer interactions than baseline methods.

Published in Transactions on Machine Learning Research (01/2026)
From Words To Rewards:
Leveraging Natural Language For Reinforcement Learning
Belén Martín-Urcelay
b edu
Deportment of Electrical and Computer Engineering Georyia Institute of Technology
Andreas Krausc
krausea@etha.ck
Department of Computer Science
ETH Zurich
Giorgia Ramponi
piorgia.ramponi@uue.
Department of Computer Science University of Zurich
Reviewed on OpenReview: /ttps://open/eee opw.reveeeer....e..om net/ forum? ? Gbz0pLANdf
Abstract
We explore the use of natural language to specify rewards in Reinforcement Learning with Human Feedback (RLHF). Unlike traditional approaches that rely on simplistic preference feedback, we harness Large Language Models (LLMs) to translate rich text feedback into state-level labels for training a reward model. Our empirical studies with human participants retonstrate that our method accurately approximates the reward function and achieves significant performance gains with fewer interactions than baseline methods.
1 Introduction
Reinforcement Learning (RL) (Sutton & Barlo 2018 is powerful framework for solving complex decision- making problems by training agents to maximize cumulative rewards through interactions with an environment.
RL has achieved remarkable success in a variety of domains, from games (Mnih et al. 2015) and robotics
(Kaufmann et al. 2023) to healthcare (Yu et al. 2021 and finance (Pendharkar 2022). Central to the RL
paradigm the concept of reward function, which provides the agent with feedback on actions and
guides its learning process.
Defining suitable reward function in real-world applications is often difficult impractical (Hadfild- Menell et al. 2017), limiting the deployment of RT. in scenarios where desired behavior is hard to specify precisely. To address these challenges, Reinforcement Learning from Human Feedback (RLHF) (Christiano et al. 22017) has emerged as a promising approach. Rather than relying on predefined reward functions, RLHF derives a reward signal directly from human input, ensuring better reflects human values and intentions. This strategy has been especially effective in domains where human judgment crucial for determining task success.
While most RLHF approaches rely comparing or ranking trajectories, humans naturally communicate intent through more nuancod textual descriptions (Cherry) 1966). Shifting from comparison-based feodback to textual input would allow for a richer expression of underlying goals (Melz et al. 2024). However, for such text-based feedback to be useful in RL, it must be effectively translated into a suitable reward model for planning. In this paper, we introduce a novel approach for learning reward models from natural language feedback. We bridge this gap by employing Large Language Models (LLMs) to map expressive human text into state-level labels, which use to update reward model. This allows the reward model to capture

---

Published in Transactions on Machine Learning Research (01/2026)
the context and subtleties of human preferences more effectively, leading to more robust and adaptable RL agents. Our contributions are as follows:
We propose an in-context learning approach using LLMs to map natural language feedback into labeled state-level examples for training a reward model.
We incorporate our reward modeling approach into an RLHF framework and validate its performance in a Gridworld environment through experiments with 26 human participants and in continuous environments. proactively steer agents
We show that human feedback, especially when provided with guidelines, toward unexplored high-reward states, avoiding common reward modeling challenges.
Our method achieves strong performance with as few as 10 human interactions, outperforming preference- based RL. In two enviromments (GllssiGrii (Gridworld and Rubik's cube), we additionally observe settings where it outperforms ground-tree -ruth reward baselines.
By harnessing human insights from textual comments, our method aims to bridge the gap between human preferences and machine learning, paving the way for more adaptable and hunan-aligned RL.
2 Related Work
2.1 Preference Based Reinforcement Learning
Traditional RL relics on explicit reward functions to drive the learning process. When the reward function is not known or difficult to construct, RLHF suggests we collect human feedback to model the reward. In Preference Based Reinforcement Learning (PbRL) (Busa-Fekete et al.) 2014 Christiano et al.) 2017), human oracles provide their preferences between pairs of trajectories. These preferences used to train reward model, enabling the deployment of standard RL algorithms to find the optimal policy. Unfortunately, depicted in Figure [ relying solely on comparisons misses out on valuable information about finer details of the reward (Basu et al. 2018 Peng et al. 2024). We propose using natural language feedback to overcome this limitation.
2.2 Learning from Natural Human Feedback
A natural way for humans to interact and express their intentions is through lext. Consequently, there is much interest leveraging natural language in RL. One common strategy is to map natural language instructions to trajectories features. To achieve this mapping, previous works limit the instructions to finite set
[Goyal et al., 2019 Bahdanau et al. 2019 Nguyen et al. 2021 Lin al. 2022). force specific sentence structure, e.g., "Go to X (Fu et al. 2019) These restrictions simplify the mapping process, but, unlike our algorithm, they also limit the flexibility of the language used. Another approach, which allows for more general language, employs Neural Networks (NN) to map from natural language to rewards (MacGlashan et al. 2015 Tung el. al.; 2018 Narasimhan et al. 2018: Yang et al. 2021). While this approach achieves good performance, requires a large labeled dataset to train the NNs. To avoid curating extensive datasets, recent research focuses on exploiting pre-trained models. Pre-trained valence analyzers (Hutto & Gilbert 2014) translate text feedback into sentiment score. At each iteration, the sentiment score of the whole text drives the Bayesian update of the reward model (Sumers et al. 2021 A single sentiment score may not
Preference Feedback (PbRL)
Text Feedback (Ours)
"The trajectory starts well, but should pass above the tree to avoid the obstacle."
OR
At most bit information (just A vs B)
Highly informative
Figure 1: Traditional RLHF uses binary preferences, providing just one bit per query. We propose natural language feedback, offering richer, human-readable signals that reduce interaction needs.
2

---

Published in Transactions on Machine Learning Research (01/2026)
capture all muances in human text, e.g., "the start is good, but the end is bod" Our experiments show how having a more fine-grained interpretation of the leedback is beneficial.
2.3 Language Models in Reinforcement Learning
LLMs have recently emorged as powerful tools for natural language processing, providing novel approaches in the field of RL. One direction treats pretrained LLMs as proxies for reward signals, directly querying whether an outcome satisfies a language description of an objective (Kwon et al., 2023). While promising, this binary feedback is limited in expressivity, an inherent information bottleneck that our method aims to overcome. Another approach harnesses LLMs' general knowledge to provide common sense priors (Li et al.)
2023
Ahn et al. 2022: Peng et al. 2024) that bias agent actions, for example, identifying hazardous states from metadata (Choi ct al.) 2022). These methods complement our approach, which aims to learn subjective human intentions that are unknown to LLMs.
A third line of work employs LLMs to translate natural language descriptions of goals into reward functions in the form of code (Zhu et al., 2025). This approach faces two key challenges: (1) it is often casier to criticize outcomes than to articulate goals precisely, leading to underspecified tasks, and (2) generating reward models from descriptions alone frequently results in reward misalignment, especially when intent is underspecified. To address these issues, rescarchers have developed itcrative revision approaches using either pre-defined reward evaluation functions (Yu et al. 2023 Ma et al. 2024 Xie et al. 2023) or binary preference feedback (Yu et al. 2024a Sun et al., 2025) to judge the performance of the generated reward functions. However, reward evaluation functions rarely exist in practice, and we demonstrate that binary feedback provides limited information. Furthermore, all these methods depend on -eneratee code snippets which may be incorrect
LLu et al. 2024) and unreliable (Yu et al. 2024b).
Our approach takes different route: we use LLMs to convert free-form human text feedback, which provides more muancdd evaluations than preference feedback, into training data for reward modeling. While there are a few experiments in which authors provide text feedback (Ma et al. 2024 Xie et al. 2023), we collect and analyze text feedback from a broad set of independent human participants, strengthening the real-world applicability of our findings. Additionally, instead of relying on LLM --M-enerrtted diversity to avoid local maxima, our method uses text feedback guide exploration towards relevant states.
3 Problem Setting
In this paper, we consider an agent who interacts with an environment aiming maximize an expected reward. We describe the interactions between the agent and the environment as an episodic Markov Decision Process without reward function (MDP)R) (Puterman] 2014h. Formally, an episodic Markov Decision Process without reward function (MDP) is tuple M (S. A, T)) S is the state space where each state s E S captures the enviromment configuration. For example, the position of an agent in a grid, the position of a car and all surrounding objects in a self-driving task or the prompt and partial answer in LLM finc-tuning. A is the set of actions that the agent can perform in the environment: for example, an action could be a step to the right on the grid, accelerating a car or generating a specific next token in LLM fine-tuning. P: x A - A(S) captures the transition probabilities, mapping state-action pairs to probability distribution of the next state over S, where A(S) denotes the probability simplex over S, and T is the time horizon. P captures the environment dynamics, such as the distribution of a car's position after acceleration or how text evolves after token generation. The reward function, which maps state-action pairs to reward r S x A - R, is unknown to the agent. Instead, the agent learns a reward model F: S + R based on human feedback. The unknown reward function reflects how well state-action pairs align with the human's implicit preferences or goals.
At each step, the agent performs an action according to policy म : S - A. The goal is learn the policy x" that maximizes the expected return from the current. state s,
(i(s)) max r(s, a
maxr(s,
Σ P(s'ss, (s'), S'ES
(1)
3

---

Published in Transactions on Machine Learning Research (01/2026)
Predicted Reward
State level rewards
Text Feedback
Reward Model
State
Action
Agent
Environment
Human
Figure 2: Block diagram of Reinforcement Learning from Human Text Feedback (RLHTF). The algorithm
consists of two iterative phases: (1) learning reward model (dashed lines) from state-level labels derived by LLM from evaluations in natural language, and (2) policy learning (solid lines), where agent is trained using standard RL algorithms that query the learned reward model.
Algorithm RLHTF
1: Inputs: number of interactions N, landmarks, fLLM
2: Initialize Policy mn and reward model Tn
3: for i= 0 to N -
Record trajectory following policy: t; + Te
Query human for feodback: f, t-t, Encode context: u, (fe, tú, landmarks)
Translate to stward pairs: (so, R3 fLLM(U:)
8:
Update reward model: Pit1 + reward _p fso, R))
9:
Update policy: T++1 < policy policy_update((r,
10: end for
where (z) max a), E [1:7] represents the number of timesteps left until the trajectory finishes, and represents the next possible state based on the environment's 's transition dynamics. We denote the state-action pairs visited when following x* the optimal trajectory T" = (s) a at]
We propose algorithm, Reinforcement Learning from Human Text Feedback (RLHTF), shown in Figure Analogously to the classical RLHF framework (Christiano et al. 2017), RLHTF consists of two phases that are executed iteratively: reward model learning and policy learning. Section g describes how the reward model F is learned from human text feedback. Since the agent does not have direct access to the true reward function r, the agent instead learns the policy म that maximizes the value function V/2 derived from the estimated reward model F. The agent dircctly queries the reward model, instead of the human ovaluators, significantly reducing time, energy; and monetary costs during policy learning (Christiano et. al. 2017). We employ standard RL methods. The algorithm is outlined in Algorithm I and publicly available
Our primary goal is to minimize the performance gap between this learned policy Ω and optimal policy n², which we denote as value gop and formally define as
T Σ
E
r (s),ai) - (Se, an)
where (St, a, a represents the trajectory when following R.
4 Learning a Reward Model from Human Text Feedback with LLMs
We break down the process of learning a reward model from human leedback into three steps: (1) Context. encoding, which gathers human feedback with environment information; (2) Translation function, which transforms the encoded context into structured signal; (3) Reward model update mochanism to incorporate
http://gghhab.cmm///leMM//////
4

---

Published in Transactions on Machine Learning Research (01/2026)
these signals. This decomposition has been shown to be effective in prior work (Metz et al. 2024 Here, we show how these steps can be implemented with text feedback.
4.1 Context Encoding
To encode raw data from the environment and interactions with evaluators into a format suitable for an LLM, we must construct a structured user prompt. This prompt should capture the agent's 's trajectory, human feedback, and any relevant environmental landmarks to provide full context for interpretation.
At each interaction, the agent follows the policy म. generating a trajectory (8:) A human evaluator observes the trajectory and provides tezt feedback f. These evaluations are flexible; for example, they may include criticisms of specific states ("the last step is horrible"), or suggestions for alternative, unexplored states ('go to the door"). The human text feedback is combined with information about the trajectory to create a user prompt
u = (feedback: f, trajectory: (8:)2 landmarks: [[namer, location)) ..., (namen, locationn)]
As shown in Figure G the user prompt may also include information about landmarks in the environment that help ground human feedback. Namely, we may include landmark names and their locations in the environment, providing reference points known by both the user and the LLM.
4.2 Translation function
We use LLMs' language processing capabilities to transform the information in the user prompt into labeled states, which are then used to train a reward model. We design a system prompt to guide the LLM in its role as translation function. Efficient system prompts must describe the LLM's role and the items the user prompt (Schulhoff et al.] 2024). As this context is environment dependent (c.g., the mcaning of clements in the states), the system prompt differs from environment to environment. However, the system prompt remains consistent across all user interactions. Appendix D includes all prompts verbatim.
There are three key components that make our system prompts efficient. First, to enhance reasoning, we employ chain of thought (CoT) prompting (Wei el al. 2022), asking the model to classify the feedback intent (e.g., evaluation vs. correction) before identifying the relevant states. Feedback is inherently (Metz et al. 2024); for example, "to the left of the lamp is good" could either be an evaluation validating a past action or an instruction suggesting a future correction. This intermediate classification step helps the translation function better interpret and adapt to human intent, leading to more precise state-level label assignments. Second, use few-shot prompting (Kaplan et al. (2000) and provide demonstrations to steer the model to better performance. By exposing the model to relevant cases, we reduce ambiguity and improve accuracy. Third, to ensure that the output of the LLM is reliably interpretable in downstream lasks, we enforce a structured format. Rather than relying on free-form text generation, which can be inconsistent, we employ function calling to guarantee a well-defined output, making easier to identify relevant states and rewards.
dependen
Given the appropriate system prompt, the LLM translates the prompt u(f, (8e) landmarks) into labeled dataset of states {sa, Rt) fLLM(U), where each output state So E S has a corresponding reward R e R. This dataset is used to train the reward model.
4.3 Reward Model Update
The goal is to learn model of the reward conditioncd on the stato-reward pairs (so, R} output by the LLM. In simple tabular settings, the agent may track the reward probability distribution for every stale and update it using Bayesian inference as the state-reward pairs are observed. In more complex or continuous environments, we may approximate the reward function with an NN. At each itcration, we expand the training dataset with the state-reward pairs generated by the LLM, and finetune the NN using supervised learning.
5

---

Published in Transactions on Machine Learning Research (01/2026)
5
Please enter your critique of the trajectory:
4
Answer "Steps and correct. Go above the clock and then throngh the toilet." (Describes specific states.) RLHTF with instructions
3
2
Answer 2: "Follow the gray path until you reach the star." (Uses unknown references, e.g., star)
1
RLHTF no instructions
Answer is terrible." (Describes whole trajectory) Sentiment Feedback
d
Figure 3: Gridworld environment. The aim of the agent is to follow target path (in gray) from the start (yellow circle) to the end (yellow star). The agent's trajectory (in blue) currently deviates from this path. Below, we show the critique input box presented to human participants, along with example responses from different feedback conditions.
5 Experiments with Human Evaluators
We recruited 26 human participants to evaluate approact In particular, apply RLHTF to the Gridworld environment shown in Figure 3. An agent aims to follow a target path 7" known to human evaluators but unknown to the agent. Standard RL approaches, like Goal Condition RL (GCRL), assume access to a reward function, which is precisely what our method aims to learn, making them inappropriate for comparison. Instead, we empirically compare against ground-truth feedback, which is rarely available practice, and state-of-the-art approaches for RL with human foedback:
True trajectory-level feedback:
The agent receives single ground truth accumulated reward for the entire trajectory, based the number of steps that match the target path. This accumulated reward is uniformly applied to all states in the trajectory.
True state-level feedback: The agent receives a ground truth reward for each state in the trajectory, indicating whether the state is on the target path.
Sentiment feedback: Evaluators convey their judgment implicitly through the sentiment of their text feedback. For example, an evaluator may write "Very bad" for poor performance or " "Amazing" when the agent closely follows the target path. The agent measures and uniformly applics the sentiment score reward across all states in the trajectory (Suners et al. 2021).
PbRL: Human evaluators observe two candidate trajectories and select the one they judge to be closer to the target (Christiano et al. 2017).
Experiments with true environment feedback were simulated 50 times. For the rest of the experiments, we followed A/B testing guidelines and randomly assigned evaluators eight different target. paths among two distinct feedback conditions. This resulted in approximately 50 experiments per feedback type. All experiments start with the same reward model prior and consist of interactions, during which the agent executes the current optimal trajectory and receives the corresponding feedback to update the reward model. Further implementation details, experimental design and participant information are provided in Appendix B.
Since a state either on the target path or not, in the RLHTF setting we prompt the LLMs to out binary rewards R' E (0, 1}. All prompts are available verbatim in Appendix D Additionally, contextualize feedback, the environment includes randomly placed landmarks, such a chair or clock. These landmarks serve reference points that both evaluators and LLMs observe, enabling the LLMs to translate feedback such "Go above the clock" in Figure 3 into positive reward at position 'b3!
Once feedback is collected, the agent updates the reward model accordingly. The agent tracks the reward probability distribution for every state, which we model as a beta distribution. In all scenarios, we initialize
The study was categorized minimal research qualified for exemption by Institutional Review Board (IRB).

---

Published in Transactions on Machine Learning Research (01/2026)
Aversge Value Standard Eror
True ectory level
True state level
Sentiment
PoRL
RLMTT no instructions (ours) RLHTF with instructions (ours)
Interaction
Figure 4: Performance comparison of algorithms in the Gridworld When participants reccive instructions how lo phrase their leedback, RLHTF outperforms other human feedback based methods and even surpasses true environment. feedback in early iterations.
the reward distributions 3(0.5, 0.5) to introduce bias lowards binary rewards: 0 (negative) or 1 (positive). Then the agent performs the trajectory that maximizes the current reward model. As the time horizon is finite (T= = 10), we solve for the optimal trajectory with dynamic programming. The agent receives foodback based on the executed trajectory. For analytical tractability, we model the distribution of the observed rewards R' with the conjugate prior, ie., as Bernoulli distributions.
5.1 Feedback Granularity and Performance
Figure 4
shows the evolution of agent error across successive interactions with evaluators. Here, ervor is defined as the number of steps in which the agent deviates from the target path. Statistical significance after four interactions was assessed using Welch's 's t-test, as detailed in Appendix B.
All fcodback methods lead to performance improvements, but the time granularity [Mttz et al. 2024 i.c., the resolution at which feedback is provided (trajectory vs. state), plays decisive role learning efficiency. In our experiments, both the true trajectory-level reward and sentiment. feedback operate at the Irajeclory-level, offering single evaluation for an entire trajectory. This coarse granularity limits the agent's s ability to discern which specific states contributed to success or failure, thereby slowing lcarning. In contrast, true state-level reward and RLHTF operate at stale-level, providing more precise guidance during training. The impact of these gramularity differences is particularly evident when comparing the first two bars Figure 4 after four iterations, true state-level feedback yields less than half the error true trajectory-level feedback. This statistically significant (p < 10-10000 10)15) improvement reinforces the advantage of detailed information about each state's contribution to accelerate learning. Similarly, our algorithm RLHTF (5th bar) outperforms the sentiment baseline (3rd bar) across all interactions, reducing the error by more than half when leveraging the fine-grained information in the text. Additionally, RLHTF remains competitive with the state-of-the-art PbRL algorithm, underscoring the effectiveness of fLLM in extracting actionable information from human text feedback.
5.2 Human Feedback Adaptability to Instructions
In our initial evaluation, found that without instructions on how to construct the text feedback, participants often included information that the LLMs could not interpret. For example, when presented with Figure R one participant wrote: "follow the gray path until you reach the star." However, the LIM has no knowledge of the gray path or the star; in fact, this is precisely what the agent is attempting to learn. Participants may also refer to history not provided in the prompt ("This is not even as good as the initial one. Retract to your first attempt.") or actions outside of the reward model scope ("go up more on the right.").
To address this issue, we showed the evaluators the following list of five instructions on how to provide effective feedback:
7

---

Published in Transactions on Machine Learning Research (01/2026)
Correctly labeled positive
Correctly labeled negative
Average Counts per Expertment
Incorrectly labeled positive Incorrectly labeled negotive
20
Sentiment
PbRL
RLHTF no Instructions (ours)
RLHT- with Instructions (ours)
Figure 5: Average count of state-labels generated per experiment, categorized by correctness (correct in blue, incorrect in red) and predicted labels (positive in solid, negative in hatched). A state-label is correct if the label accurately identifies whether the state is in the target path or not. Providing brief instructions to human evaluators (RLHTF with instructions) substantially increases the proportion of correctly identified states.
The agent has limited capabilities, so for understand you correctly you should restrict your feedback. Namely, the agent does not understand:
Additional Instructions:
Do not compare trajectories: treat each path individually, without reference to previous attempts.
For example, avoid feedback like: "Now it is worse, go back to the previous trajectory".
Do not refer to the position of the star, yellow circle, or gray road. The agent doesn't 't know their locations; in fact, the agent is trying to learn where these are. For example, avoid feedback like:
"Follow the road until the star".
Avoid specific movement descriptions (go up, turn right). For example, avoid feedback like: "go up, up, right, up" or "turn right later".
Instead, the agent understands well:
Description of states: Position in the trajectory. e.g. "At the beginning is wrong", "slep nunber 6 is good", or position with respect to the furniture, e.g. "You should go to the left of the couch". Sentiment: It works especially well if you explain what states are good or bad. c.g. "The first half of the trajectory bad. Above the TV is good."
We conducted 51 exporiments in the RLHTF with instructions condition. Remarkably, providing participants with just two paragraphs instructions increases the comprehensibility of the feedback to the LLM. For example, the same setting of Figure 3a participant wrote: "steps and are correct. Go above the clock and then through the toilet." This allows fLLM to better extract relevant spatial cues and assign rewards accordingly, shown in Figure 5 We evaluate whether each state-reward pair, automatically generated from human feedback, is correct by comparing it against the known ground truth path. Formally, generated state-reward pair (s,, R) is considered correct if R = 11s, E ET where 1{-} is the indicator function, and 7" is the ground truth target path. Providing brief instructions to the evaluators increases the accuracy of LLM-generated state-labels by 19.1%, indicating that the same amount of human effort yields substantially clearer, more actionable supervision. In contrast, the sentiment baseline receives mostly negative feedback and attains only 35.4% positive precision on the states in 7* PbRL assigns rewards every differing state across pair of trajectories, which inflates label counts with redundant weakly informative items and results in 6.9% lower label accuracy than RLHTF with instructions (63.7% vs. 70.6%). Moreover, RLHTF with instructions encourages evaluators to provide more constructive guidance: it is the only condition in which the amount of positive leedback surpasses negative (1.25x). These trends indicate that instructions help steer feedback, improving both reward attribution and, consequently, policy success rates. Moreover, as [ [B.I] providing guidelincs significantly improves consensus: doubles the agreement between human evaluators' foedback interpretations and increases the consistency of interpretations across different LLMss.
detailed Appendix

---

Published in Transactions on Machine Learning Research (01/2026)
After single instance of feedback, Figure shows that RLHTF with instructions reduces the task error by 42%, even outperforming ground-truth state-level feedback. This improvement occurs because RLHTF proactively guides exploration towards the high-reward regions mentioned by the human feedback, which might not be covered by the current trajectory. In contrast, true environment foedback is purely reactive, offering evaluations only for states actually visited. Although ground-truth state leedback eventually surpasses RLHTF after 4 interactions, our method outperforms other RI. algorithms without explicit rewards. A key advantage of RLHTF is its use of LLMs to perform reward attribution. Unlike previous approaches, which update all states in trajectory (sentiment) or all differing states between two trajectories (PbRL) with single reward, RLHTF distributes rewards more precisely. This precision allows RLHTF to reduce error more quickly and substantially, significantly outperforming Sentiment (p < 10-13) and PORL (p < 0.0025) algorithms. After just 4 instances of human feedback, RLHTF with instructions reduces the error to approximately one-third of its original value.
Beyond task error reduction, Table shows the success rate, defined as the percentage experiments which the learned reward model, after four pieces of leedback, leads to policy that exactly follows the target path. Strikingly, RLHTF with instructions matches the success rate of true state-level feedback, achieving 25.5% success rate. This level of performance is particularly impressive given the combinatorially large space of possible trajectories and the fact that only four pieces of feedback are provided. In contrast, baseline methods (true trajectory-level feedback, sentiment feedback, and PhRL. fail to produce a single perfect path. This result demonstrates that guided human feedback, when processed by RLHTF, can rival access to privileged ground-trute state rewards in our experimental setting.
Algorithm
Success Rate
True trajectory-level
0%
True state-level
25.0%
Sentiment (Sumers et al. 2021)
0%
PbRL (Christiano et al.
2017)
0%
RLHTF instructions (ours)
0%
RLHTF with instructions (ours)
25.5%
Table 1: Percentage of learned policies that perfectly follow the target path after four rounds of feedback. RLHTF with instructions matches the performance of true state-level feedback, while all other methods fail to achieve any success.
As LLMs become more powerful, their ability process human feedback improves, further boosting the performance of RLHTF in the Gridworld experiments. As shown in Appendix [B.] upgrading from GPT-4o (used in our main experiments) to the newer GPT-4.5 or Sonar-reasoning models further reduces the error by 23% and 30,, respectively. This result suggests that, unlike the baselines, RLHTF in settings similar to those tested here may continue to benefit from future advances in LLM capabilities, further strengthening the potential of natural language feedback in RL.
6 RLHTF in Continuous and Structurally Complex Environments
In the previous section, demonstrated that RLHTF efficiently learns reward functions in discrete, tabular domains. Here we analyze method's performance in two settings that are continuous and structurally compler. We focus on two representative challenges: (i) precise motor control in the MuJoCo Reacher task, where rewards must be inferred from a continuous observation space, and (ii) goal- condiiiooo manipulation in a Rubik's Cube environment, where the reward hinges on matching human-specified color patterns. Across these experiments, we compare RLHTF to preference-basee based RL (PbRL) and an oracle with access to the true reward.
9

---

Published in Transactions on Machine Learning Research (01/2026)
6.1 Generalization to Continuous Environments
lest generalization beyond labular set sings, we apply RLHTF a continuous environment of the physics simulator MuJoCo (Todorov et al. 2012). Namely, we apply RLHTF to Reacher, two-jointed robot arm. The aim is to apply appropriate torques to the hinges that the robot's fingertip reaches target. The human evaluators watch and critique video of the robot arm moving under the current policy. To aid human evaluators in their assessment, we incorporate a timestamp and some visual landmarks (colored circles) into the environment. Figure G shows the modified Reacher environment. These modifications aid the human evaluator to refer to specific locations ("Go to the left of the blue circle.") or moments in the trajectory ("The movement from frame 4 to wrong."). We emphasize that our method does not require the human explicitly understand or annotate states such as joint angles or torques. Instead, human evaluators provide natural language feedback, and our method infers meaningful statc-level reward signals from contextual and temporal cues naturally present in the text.
User prompt
ChatGPT
Human Feedback
to the
blue point."
Positive reward at state
Fingertip position: 20p
Landmarks
Angle of arm: Speed: [0, 0]
Yellow: 18b
Orange: 5b
Blue: 20q
White:
Fill missing entrics by sampling uniformly at random from the trajectory
Trajectory State (fingertip position angle of arm spood)
Video current policy in reacher environment
Positive reward at state
. Fingertiip position: 20p
timeframe the vidco.
Anglc of arm: (5.8, 1.6]
Speed: [0,
Figure 6: Generation of state-reward pairs from human feedback and observations using LLM (Step in Algorithm in modified Reacher environment. The aim of the robotic arm is to stay close to the red ball.
The LLM processes three inputs: human feedback, landmark locations and the sequence of filtered states (where information about the target is removed, see Appendix B.2), making up the vidco. The LLM's task is to interpret these inputs deduce what states are described as positive or negative. Figure shows how the state-reward pairs are generated. The LLM outputs pairs of filtered states and binary labels. Each label indicates whether the corresponding state is positive negative. If filtered state is returned partially, missing elements are filled in by randomly sampling from the set of observed trajectory states. We approximate the reward function as fully connected NN, which takes a state vector input and outputs a scalar corresponding to the predicted reward. At each iteration, we expand the training dataset with the state-reward pairs output by the LLM, R.. Then, we employ stochastic gradient descent to fine-tune the NN with the expanded training dataset. This supervised learning process uses a cross-entropy loss function and is further detailed in Appendix B.2
We suggest that feedback in natural language is more informative than preferences between two trajectories, and thus RLHTF requires fewer interactions with humans to achieve accurate reward model. To verify the hypothesis, we compare the evolution of the reward model with RLHTF versus PbRL, as shown in Figure 7 While both feedback types improve the reward model, lext leedback results in a more precise localization of the target. By the tenth interaction, only a small area around the target receives a high reward with RLHTF, reward model more uncertain about the target location, giving high reward
whercas PbRL results large areas of the environment. Figure suggests that natural language feedback enables the reward model to identify the target more quickly and precisely.
10

---

Published in Transactions on Machine Learning Research (01/2026)
10
0.8
0.7
0.6
(b) RLHTF after (c) RLHTF after (d) RLHTF after
query
5 queries
10 queries
0.4
0.3
0.2
(a) Initialization
0.0
(e) PbRL after
(f) PbRL after
(g) PbRL after
query
queries
queries
Figure 7: Reward model visualization for RLHTF PbRL. The top illustrates the evolution of the reward estimations as more text feedback is gathered, while the botlom row shows the corresponding evolution with preference feedback. The blue star marks the target location and darker colors indicate lower predicted rewards. Notably, RLHTF quickly converges to a more accurate reward model than PbRL, as evidenced by the more localized high-reward region around the larget.
We also compare the agent's performance, in terms average distance to the target, when there is a budget of 10 human interactions. Figure Sa shows how the reward evolves the agent learns. Although RLHTF is not effective directly observing the true environment feedback, RLHTF performs much better in regimes with low feodback than PbRL. In fact, in experiments RLHTF increases the reward by 40% with only 10 human inputs, while PbRL at first repeatedly executes the trajectory with the arm fully bent, and it needs many more pairwise comparisons to approach the target.
16
Average Reward
14
True
Reward
RUHTF PbRL
10
RUHTF (6 10
PAAL 10 10
1
True (6 10-1)
True 15 10
10000
20000 30000
40000 50000 60000 Episodes
70000 80000
10000 20000
30000
40000
50000
60000
Episodes
(a) Reacher environment. RLHTF demonstrates strong performance in continuous euvironments, increasing the reward by 40% with only 10 humman inputs.
(b) Rubik environment for task where the optimal reward is 7.4. Unlike true reward, which can lead local maxima, RLHTF leverages dynamic humman feedback that adapts to agent progress, guiding explo- ration and overcoming reward modeling challenges.
Figure Reward evolution with episodes of the REINFORCE algorithm. We compare the performance for RLHTF with 10 pieces of text feedback provided at the vertical dashed lines, PbRL with 10 trajectory comparisons, and true environment.
11

---

Published in Transactions on Machine Learning Research (01/2026)
6.2 Towards Adressing Certain Reward Modeling Challenges
To highlight RLHTF's strengths in environments where reward design challenging, we evaluate RLHTF the Rubik's Cube environment (Gröling) 2022). This environment involves six-sided cube where cach side has 3 x grid of squares, each laking one of six colors. The goal of the agent is to manipulate the cube that the front face matches a pattern specified by human user.
Consider the task of obtaining an orange 'X' pattern on the front face of the Rubik's cube, i.c., both diagonals must be orange. In RLHTF, the evaluator watches a video of the trajectory following the current agent policy, and then provides text feedback. This feedback, along with the sequence of nine color grid squares observed on the front face, is input to the LLM. The LLM processes this information and outputs sets of 9 color grid squares, each accompanied by label indicating whether the set is positive or negative. The labeled dataset is then used to update the reward model. In the case of PbRL,, we query the evaluator for their preference between two different videos. The candidate videos are chosen by sampling trajectories and identifying the pair with the highest variance in preference estimations among the reward models in an ensemble. The true reward is computed by assigning a +2 score for each correct orange square in the diagonals and averaging this score over the ten timesteps in each trajectory.
Figure 8b compares the performance of RLHTF, PbRL, and true reward. After only ten human interactions, RLHTF successfully produces the desired orange 'X' pattern on the Rubik's Cube front face. In contrast, PbRL shows mcaningful improvement beyond its bascline performance. This limitation ariscs because preference-based feedback encodes at most one bit of information per interaction, requiring many more interactions to achieve the desired performance. Using true rewards, the agent gets stuck in a local maximum, where only three out of the five diagonal squares are correctly orange. Even after manipulating the learning rate, the agent fails to produce the desired pattern. This shows how reward design in RL problems is challenging and often leads to unwanted behavior, even constrained environments. In contrast, evaluators in RLHTT naturally adapt their feedback according to the agent behavior, mitigating such misalignments. For example, evaluator may write "you are doing it wrong, the top right corner should be orange and not red", thus encouraging exploration when stuck in local maximum. This adaptability bypasses the need for complex reward design and extensive hyperparameter tuning. Human text feedback and additional experiments with a different target pattern are provided in Appendix B.3
7
Future Work
While our framework introduces a new paradigm for incorporating natural language human feedback into reinforcement learning, its current implementation presents several limitations. First, our prompts must be adapted to each environment, similar to how traditional RL requires handcrafting reward functions. Although we demonstrate some degree of generalizability across three distinct environments, fully task- agnostic prompting remains an open challenge. The system prompts require manual design to ground the LLM in the specific context of cach environment. In our experiments, using off she-tff- object detectors (i.e., YOLO) to automatically detect and name landmarks, which are then inserted into the prompt. However, the core system prompt still requires manual design. To further automate the system prompt generation, future work could leverage metadata (e.g., simulator docunentation or object annotations) to automatically generate descriptions or relevant items, (e.g. the description of state elements). Alternatively, future work could explore automatic prompt optimization techniques (Ramnath ct al., 2025), that use evolutionary strategies to discover effective prompts without human intervention. We focus on three environments chosen for their interpretability and relevance to key RL challenges. Future work will extend our approach to more complex tasks, such as LLM finetuning, where an evaluator could critique stylistic elements ("too formal") or specific content ("the third paragraph is confusing"), going beyond the binary preferences typically used in existing RLHF pipelines.
partially reduce this burden by
Second, although LLMs can interpret rich feedback, their performance is constrained by the content and clarity of the prompt. When critical information is missing or ambiguous, the LLM may hallucinate or misinterpret the intent. We find that providing instructions human evaluators significantly reduces these issues, though it does not eliminate them entirely. Multimodal verification could help reduce hallucations further. Future work
12

---

Published in Transactions on Machine Learning Research (01/2026)
could deploy an ensemble method, where each piece of human feedback is interpreted by several models, and only outputs with majority agreement are retained. Our analysis suggests that keeping only the state-reward pairs on which the majority of models agree will filter many of the incorrect interpretations. Similarly, our method shows consistent empirical improvements, but we lack formal guarantecs of performance.
Lastly, our experiments focus on state-level leedback. However, our approach naturally extends to action-level feedback by appropriately prompting the LLM, e.g., an evaluator on the Gridworld environment might say go up steps.' This capability opens the door to broader applications in RL settings.
8 Conclusions
Our work leverages LLMs to extract state-level rewards from natural language feedback, addressing the challenge of reward attribution and capturing more muanced information than simple binary comparisons or single sentiment score. Morcover, our experiments with real human participants, a contribution not common in this line of research, take a step loward real-world applicability.
Acknowledgments
Belén Martín Uucelay supported by the Rafael del Pino Fellowship. The human-participant experiments were supported by the Hasler Stiftung grant (project 2024-06-12-90). The authors thank the action editor and the anonymous reviewers for their insightful comments and constructive feedback.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do i can, not as i say: Grounding language in robotic affordances. Proceedings of Machine Learning Research, pp. 287-318, 4 2022. URL http://arxiv. http://aaxvvv...o/aaaaa at////aa/22/ 22699.161191
Dzmitry Bahdanau, Arian Hosseini, Felix Hill, Pushmeet Kohli, Jan Leike, Edward Hughes, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. In Proc. of International Conference on Learning Representations, 2019.
Chandrayee Basu, Mukesh Singhal, and Auca D. Dragan. Learning from richer human guidance: Augmenting comparison-baseer based learning with feature queries. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Roise Robot Interaction, HRI '18, pp. 132-140, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450349536. doi: 00.1155/11112213317222 URL https://doo.org///
1145/3171221.8178
Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: Evolutionary direct policy search using a preference-bact racing algorithm. Machine Learning, volume 97, pp. 327-351. Kluwer Academic Publishers, 10 2014. doi: 10.1007/
s10994-014--458-8.
Colin Cherry. On human communication. MIT press Cambridge, 1966.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models task-specific priors. 10 2022. URL
12530
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proc. of Advances in Neural Information Processing Systems, volume 2017 2017-Decem, pp.
2017.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. In Proc. of International Conference on Learning Representations, pp. 1 14, 2019.
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. Using natural language for reward shaping in reinforcement learning. In Proc. of International Joint Conference on Artificial Intelligence, volume 2019-Augus, pp. 2385 2391, 2019. ISBN 9780999241141, doi: /11.9966//iaai111111//
Mare Gröling, Gym rubik's cube environment, 2022. URL httt::://gghhubb..ommmmmgoorr
Dylan Madfield-Menele Stuart J Russell, Pieter Abbeel, and Anea D Dragan. Inverse reward design. Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 6765-6774, 2017.
CJ Hutto and Eric Gilbert. Vader: A parsimonious rulo-based model for sentiment analysis of social media text. In Prof. of Conference on Artificial Intelligence, pp. 216-22222 2014. URL htttp://semtic..nt/
Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012,, Yonghye Kwon, Kalen Michael, TaoXie, Jiacong Fang. imyhxy, Lorna, Zeng Yifu, Colin Wong, Abhiram V, Diego Montes, Zhiqiang Wang, Cristi Fati, Jebastin Nadar, Laughing, UnglvKitDe, Victor Sonck, tkianai, yxNONG, Piotr Skalski, Adam Hogan, Dhruv Nair, Max Strobel, and Mrinal Jain. ullrralticesyylloe5: v7.0 YOLOv5 SOTA Realtime Instance Segmentation, November 2022. URL https: http:///ooo.o//10..000 5281//enooo http://oorrrr......////oooo.o. 7347926.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/ abs//200108336 2020. URL. https://api sew.aticccaanitccoocrrrcr wwwpcchhlarrroo/orrrpreeee : 210861095
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza, Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982 987, 2023.
Minac Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In International Conference on Learning Representations, 2023. URL httt:://aaxxi..rg/bbbb/////
Belinda Z. Li, William Chen, Pratyusha Sharma, and Jacob Andreas. Lampp: Language models as probabilistic priors for perception and action. 2 2023. URL
Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context. In Proc. of the Association for Computational Linguistics, volume 1, pp. 8546-8560, 2022. ISBN 9781955917216. doi: 10. /2tp3//11112222aallooooo
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602.1
Zhijic Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift finger anymorc? assessing the quality of code generation by chatgpt. IEEE Transactions on Soflware Engineering, 50(6): 1548- 1584, 2024. doi: T0.1099/SSE.222433322223
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. 5 2024. URL
James MacGlashan, Monica Babeg-Vroman, Vroman, Marie DesJardins, Michael L. Littman, Smaranda Muresan, Shawn Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. Grounding english commands to reward functions. Robotícs: Science and Systems, 11, 2015. ISSN 2330765X. doi: 10. N0.15607/RSS.2115.XI...
Yannick Metz, David Lindner, Raphaël Baur, and Mennatallah El-Assady. Mapping out the space of human feedback for reinforcement learning: A conceptual framework. arXio preprint 21. 2024.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Arlificial Intelligence Research, 63:849 874, 2018. ISSN 10769757. doi: 10.1613/
1.1263.
Khanh Nguyen, Dipendra Misra, Robert Schapire, Miro Dudik, and Patrick Shafto. Interactive learning from activity description. Proceedings of Machine Learving Research, 139:8096 8108, 2021. ISSN 26403498.
Parag C Pendharkar. Reinforcement learning in financial market applications: a survey and research agenda.
Artificial Intelligence Review, 552)):1277-1306, 2022.
Andi Pong, Ilia Sucholutsky, Bolinda Z. Li, Theodore R. Sumcrs, Thomas L. Griffiths, Jacob Andreas, and Julie A. Shah. Learning with language-guided state abstractions. In International Conference on Learwingg Representations, 2 2024. URL http:://aaxiw...rrg/g//.////// abt////aa//222..2 18759
Martin L Puterman. Markow decision processes: discrete stochastic dynamic programming.. John Wiley & Sons, 2014.
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, et al. A systematic survey of automatic prompt optimization techniques. arXiv preprint Xri:::2502. 16923, 2025.
Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, H Han, Sevien Schulhoff, et al. The prompt report: A systematic survey of prompting techniques. arviv preprint 2406. 06608, 5, 2024.
Thoodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, and Thomas L. Griffiths. Learning rewards from linguistic feedback. In Proc, of Conference on Artificial Intelligence, volume 7, pp. 6002-6010, 2021. ISBN 9781713835974. doi: 10.1609/aaaa v3i7.13517
Shengjie Sun, Runze Liu, Jiafei Lyu, Jing- Wen Yang, Liangpeng Zhang, and Xiu Li. large language model-driven reward design framework via dynamic feedback for reinforcement learning, Knowledge- Based Systems, 126:11 2025.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In Proc. of International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1109/IROS. 2012.638610
Hsiao Yu Tung, Adam W. Harley, Liang Kang Huang, and Katerina Fragkiadaki. Reward learning from narrated demonstrations. In Proc. of Conference on Computer Vision and Pattern Recognition, pp. 7004-7 7013, 2018. ISBN 9781538664129. doi: 20000//CVP..201.00000...
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting clicits reasoning in large language models. In Proc. of Advances in Neural Information Processing Systems, 1 2022. URL lhtt:://aaee www.riii...//bbbb. httgg/bbb/2201.. 11903
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229 256, 1992. ISSN 0885-6125. doi: 10.1007/b/009996600
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. arXiv preprint arXin:2309.11488, 2023.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Tsung Yen Yang, Michael Hu, Yinlam Chow, Peter J. Ramadge, and Karthik Narasimhan. Safe reinforcement learning with natural language constraints. In Proc. of Aduances in Newral Information Processing Systems, volume 17, pp. 13794 13808, 2021. ISBN 9781713845393.
Chao Yu, Hong Lu, Jiaxuan Gao, Qixin Tan, Xinting Yang, Yu Wang, Yi Wu, and Eugene Vinitsky. Few-shot in-context preference learning using large language models. 10 2024a. URL http://arxxw.org//aaata 2410
17233
Chunxiao Yu, Jinlong Liu, Shamim Nemati, and William M Gregg. Reinforcement learning in healtheare: A survey. Xiv preprint orXiv: 08796, 2021.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserral Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Hoess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language rewards for robotic skill synthesis. In Jie Tan, Marc Toussaint, and Kourosh Darvish (eds.), Proceedings of The 7th Conference Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp. 374 404. PMLR, 06 09 Nov 2023. URL
mlr wwwppppweaeeeeeeee watpe//222/yyu233. htmmlll..mmmmm1..m
Xiao Yu, Lei Liu, Xing Hu, Jacky Wai Keung, Jin Liu, and Xin Xia. Fight fire with fire: How much can we trust chatgpt on source code-related tasks? IEEE Transactions on Software Engineering, 50(12):3435 3453, 2024b. doi: 00.109//SE.2022444922222
Guobin Zhu, Rui Zhou, Wenkang Ji, and Shiyu Zhao. Lamarl: Llm-aided multi-agent reinforcement learning for cooperative policy generation. TEEE Robolics and Automation Lellers, 2025.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Broader Impact Statement
A.1 Human Subject Research
Our work involves a study with human participants. The study was categorized as minimal risk research qualified for exemption status under 45 CFR 46 104d.2 by the Institutional Review Board (IRB). All 26 participants were adults, provided informed consent before participation, were able to withdraw at any time, and were equally compensated $10 for approxinately 30 minutes of their time (no volunteer work was used). No personally identifying information beyond coarse demographic data was collected, stored or shared.
The ultimate goal of our work is to reduce the need for complex reward engineering by enabling aligmment by non-experts from their feedback in the form of natural language. This new manner of alignment may contribute to the democrat ization of RL training. However, as with all alignment technologies, there is a dual-use risk where such methods could be employed to align agents with malicious instructions. Malicious actors could exploit natural language foodback interfaces to inject mislcading rewards, causing RL agents to learn harmful behaviors. All our experiments were done in simulated enviromments with no deployment to real-world systems high-stakes decision domains. Any future applications in safety-critical or socially sensitive settings would require additional domain-specific oversight and safeguards.
The proposed algorithm seeks to align RL agents with fewer human interactions. However, employing fewer evaluators for alignment may exacerbate bias. This reinforces the importance of hiring diverse evaluators across cultural, gender and language backgrounds.
A.2 Reliance on LLMs
This work uses LLMs as translation functions to map human feodback onto state-level rewards. However, LLMs are prone to hallucinations or biased interprelations, especially when presented with ambiguous feedback. We attempt to mitigate these risks by using few-shot prompting, environment-specffe prompts and providing evaluators with guidelines, however, misinterpretatioos still persist. Moreover, the performance of our approach depends on proprictary LLM models, which may limit transparency and reproducibility. To partially mitigate this, we release our prompts and code, and report results across multiple LLMss
B
Experiment Details
B.1
Gridworld
This section provides detailed information about the experimental setup, data collection process, and implementation choices for our experiments with human evaluators. In each trajectory, the agent takes 10 steps in the Gridworld enviroument shown Figure 3 At each step, the possible actions are either to move right or up. The agent movements are restricted action that would take the agent outside of the allowed region, the agent does not move.
5x10 grid. When the agent is a border and performs
We simulate scenario in which human wants a robot to navigate their house in a specific manner. To make the environment more realistic, we sample four images of furniture or household objects (e.g., sofa, chair, toaster and TV) and add them to the environment. These objects serve as landmarks, helping evaluators communicate their critiques more effectively by using them as reference points. The landmarks are detected with a YOLOv8 model (Jocher et al.) (0022) trained on the Microsoft COCO dataset (Lin et al. 2014), and their positions are fed the LLM for shared context with the evaluators. Namely, we use gpt-40 translate human feedback into state-level rewards.
We conducted our experiments with the assistance of human evaluators. The study was categorized as minimal risk research qualified for exemption status under 45 CFR 46 104d.2 by the Institutional Review Board (IRB). We recruited 26 participants, most whom were university students (46% female, 54% male). Table provides a breakdown of their geographic region were participants were born.
Following A/B testing guidelines, we randomly assigned evaluators to interact with different algorithms. Each evaluator provided faekback two out of the four algorithms: (1) either RLHTF with instructions or R..TFF
17

---

Published in Transactions on Machine Learning Research (01/2026)
Region
Participant Count
Asia
11
North America
6
Africa
2
South America
1
Europe
1
Table 2:
Breakdown of participants by region of birth.
without instructions, and (2) either Sentiment or PbRL. To minimize any potential bias due to familiarity with the interface or tasks, we randomized the order of these settings. Fach participant provided feedback for either different rooms or for a duration of 30 minutes, whichever occurred first. Participants were compensated with $10 for their work. In total, we collected 772 foedback samples from 26 participants. The average number of words per interaction is 17.48 and 17.30 words in the settings without and with additional guidelines, respectively. A breakdown of the number of experiments for each algorithm using human feedback is summarized in Table B The experiments with the true environment rewards were simulated 50 times.
Experiment Type
First
Second
Total
RLHTF with instructions
27
24
51
RLHTF without instructions PbRL Sentiment
23
22
45
24
32
56
24
17
41
Table 3: Breakdown of number of experiments for each algorithm type, cach experiment was donc for four interactions. Columns Pirst and Second indicate whether the feedback was collected in the first or second task performed by the evaluator, with totals shown in the last column.
We conduct Welch's I-fest. to compare the performance of different algorithms after receiving four pieces of feedback. Table E reports the resulting p-values. Our results indicate that RLHTF with instructions siguificantly outperforms both PbRL and sentiment feedback, which do not have direct access to the ground truth feedback from the environment. also significantly outperforms the setting where true trajectory- feedback is received from the environment. In the of RLHTF without instructions the feedback, the performance is statistically indistinguishable from PbRL (p 0.5531). Receiving the ground truth state=level feedback from the environment yields the best trajectories after four pieces of feedback, significantly outperforming all other settings, except for RLHTF with instructions where the difference is not statistically significant (p- = 0.1752).
how to construct
Table depicts the performance of the final policy learned from pieces of feedback. Most algorithms fail recover the exact target trajectory, achieving 0% success rate. Notably, both our method RLHTF with instructions and ground-truth state-level feedback result in perfect trajectories in 25% of experiments. This demonstrates that natural language feedback, when guided with simple prompting instructions, achieve results comparable to the often unrealistic assumption of state-level supervision.
RLHTF without
True trajectory
PbRL
Sentiment
True state
instructions
RLHTF with instr. RLHTF without instr. PbRL
0.0009
0.0024
3.4 x 10 14
1.9 x 10-8
0.1752
0.5531
2.1 x 10-6
0.0391
6.4 x 10-7
2.7 10-9
0.0021
4.8 x 10-7
Sentiment
4.5 10-5
2 10-23
True trajectory
1.2 10-16
Table 4: p-values for Welch's t-tests, showing that. RLHTF with instructions and true state-level rewards significantly outperform other methods with 4 pieces of feedback.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Which trujectory 1s better? blue, red)
Figure 9: Example of human interaction with PbRL in Gridworld.
Guidelines for Evaluators
For both RLHTF settings, we provided participants with the following guidelines:
Objective: The purpose of this experiment is to provide instructional feedback to artificial agent. The task for the agent is to navigate from yellow circle to yellow star along gray pathway. The agent will attempt this task by following a trajectory marked in blue. Your role is to offer written feedback that assists in correcting the agent's current, course.
Task Instructions:
1. Observe the blue trajectory that the agent has taken.
2. Provide your guidance and feedback on the agent's performance (e.g., "Do not go below the sofa. The end was very good").
Repeat times.
In the setting with instructions, participants also received the additional set of instructions outlined in Section 5.2 These instructions helped participants adapt their feedback more effectively, which in turn accelerated the learning of the reward model.
For the PbRL the participants were shown the example in Figure 9 and told:
Objective: The purpose of this experiment provide feedback an artificial agent. The task for the agent is navigate from yellow circle to yellow star along a gray path. Two agents will atlempl. this task by following trajectory marked in blue and red respectively. Your role is to select which of the two trajectories is better, that the agent can learn the correct path.
Task Instructions:
Observe the blue and red trajectories that the agents have taken.
2. Choose the best (0: blue, 1: red).
3. Repeat 4 times.
Lastly, in the sentiment. setting participants were told:
19

---

Published in Transactions on Machine Learning Research (01/2026)
Objective: The purpose of this experiment is to provide feedback to an artificial agent. The task for the agent is to navigate from yellow circle to yellow star along a gray path. The agent will attempt this task by following trajectory marked in blue. Your role is to offer written feedback that assists in correcting the agent's current course.
Task Instructions:
1. Observe the blue trajectory that the agent has taken.
2. Provide a sentiment rating (how good or how bad) for the trajectory, e.g. "it is horrible" or "amazing".
3. Repeat 4 times.
Additional Guidelines Effect Consensus
To quantify annotator variability, we analyse the state-labels obtained across evaluators when exposed to the same 8 Gridworld test rooms with identical initializations. For each pair of participants, we compute the Jaccard similarity between their sets of state-label pairs, capturing the fraction of overlapping annotations relative to the total annotations made. the setting without additional guidance, the average pairwise Jaccard similarity 0.1028, indicating very high variability in how non-experts describe the same trajectorics. When we provide additional guidelines to the evaluators, the average Jaccard similarity increases to 0.2241, agreement between evaluators roughly doubles, although substantial variability remains. This analysis confirms that human feedback can be highly noisy and underscores that our method is explicitly evaluated under such challenging, non-expert conditions.
We further hypothesize that these guidelines also reduce ambiguity in the human feedback, thereby increasing output consistency across different LLM models. To test this hypothesis, we input the same prompts and collected feedback to different models (sonar, gpt-4o, ol, 03-mini and gpt-3.5) and compared the output state-reward pairs. Figure 10 categorizes the generated state-reward pairs according to the number of models that agreed on them. We notice that by providing guidelines led to marked shift toward consensus. The proportion of outputs with majority agreement (at least 3 models) rose from 48.5% to 73.3%. Specifically, complete agreement across all five models nearly doubled (10.3% to 20.4%), while inconsistent outputs generated by only a single model dropped significantly (30.1% to 13.5%).
Additionally, Figure 11 shows that the correctness of the outpul state-reward pair highly correlated to its level of agreement between models. While 84% of the state-reward pairs generated by all the 5 models are correct, only 43% of the state-reward pairs generated by single model are correct. This suggests that multinodal verification could help reduce hallucinations.
Performance for Different LLMs
We assess the robustness of reward learning from human feedback by experimenting with alternative LLMss. Specifically, we reuse the human feedback collected during the RLHTF experiments with GPT-4o outlined in Section 5 and apply the same prompts to different base models. Figure [2a shows the average error evolution in the setting where human evaluators are not given additional instructions, while Figure 12b shows the performance when the feedback was given after observing additional instructions.
We observe correlation between LLM capability and RLHTF performance. GPT-3.5 Turbo maintains the worst average error across interactions in both settings. Perplexity's sonar-reasoning model performs the best, showcasing its ability to interpret free form human feedback. Comparing final errors at interaction 4, Sonar-reasoning reduces the average error by 33% relative to GPT-3.5 in the no-instructions setting (from 6.7 to 4.5), and by 52% in the with-instruuttins setting (from 4.7 to 2.3). The superior performance of more capable models is also reflected in the Gridworld success rates summarized in Table 5 where more powerful LLM models achieve the target path 2 to 3 times more often than GPT 3.5 in the case with instructions.
The error reduction pattern across interactions follows consistent trend: rapid improvement after receiving the first piece of text feedback, followed by more gradual improvements in subsequent. interactions. This
20

---

Published in Transactions on Machine Learning Research (01/2026)
30.1% %
13.6%
13.5%
10.3%
Appear in 1/5 models
18.6%
Appear 2/5 models
20.2%
Appear 3/5 models
21.4%
Appear in 4/5 models
Appear 5/5 models
20.3%
17.9%
34.1%
(a) Without Guidelines
(b) With Guidelines
Figure 10: Consensus output state reward pairs across five LLMs: sonar, gpt-4o, ol, 03-mini, gpt-3.5. Percentages reflect the share of all generated outputs, grouped by the number of models in agreement, We observe that providing guidelines to the evaluators increases the agreement between models.
LLM model
With Instructions
Without Instructions
gpt-3.5-turbo
9.8%
2.2%
gpt-4o
25.5%
0.0%
gpt-4.5
25.2%
8.8%
o1
27.5%
4.4%
03-mini
25.5%
0.0%
sonar
15.7%
2.2%
sonar-reasoning
31.37%
4.4%
Table 5: Gridworld success rate different L.M models with and without instructions.
pattern is maintained across all models, showcasing the robustness of RLHTF, though the rate of improvement varies, with newer models generally showing more substantial error reductions per interaction. These findings support our conjecture that RLHTF will likely continue to benefit from ongoing advancements in LLMs technology.
B.2 Reacher Environment
Each trajectory in the Reacher environment spans T 50 frames. The standard observation space contains information about the goal (target location) and the performance (distance to target). However, to prove that the LLM is capable of deducing the performance solely from the human feedback, we filter this information from the user prompt. The L.M obtains a filtered state that includes the fingertip's location, arm's joint angles, and angular velocities. The filtering and preprocessing of the observation are detailed Figure I3
We use the LLM gpt-40 to translate the human feodback state-reward pairs. To conduct fair comparison with PbRL,, we reconstruct the filtered states output by the L.M back into the complete original observation
21

---

Published in Transactions on Machine Learning Research (01/2026)
Positive
True Negative (TN)
False Positive (FP)
Count state-reward pairs
False Negative (FN)
200
Number of models agreeing on the state-reward pair (out of 5)
(a) Without Guidelines.
True Positive (TP)
1400
5566
True Negative (TN)
False Positive (FP)
Count state-reward pairs
1200
Negative (FN)
3000
Number of models agreeing on the state-reward pair (out of 5)
(b) With Guidelines.
Figure 11: Count of state-reward pairs for each level of model agreement, categorized by correctness (correct in blue, incorrect in red) and predicted labels (positive in solid, negative in hatched). A state-reward is correct if the reward accurately identifies whether the state is in the target path or not. The more models output a stato-reward pair the higher the probability that is correct.
GPT-3.5 Turbo
GPT-4a
Acerage Frrr Standard Froor
000
GPT-4.5
Standard
5006
Soner
Sonar
5000
5
Semer
Interaction
Interaction
(a) Without additional instructions.
(b) With additional instructions.
Figure 12: Performance for different LLMs with feedback from RLHTF experiments
22

---

Published in Transactions on Machine Learning Research (01/2026)
Default Observation
cosine of angle the arm cosine of the angle of the second arm
1.3184
1.3184
sine of the angle of the first arm sine of the angle of the second arm x- x-corrdinate the target
0.9996
Filtered State
0.9995
Angles of in degrees
[5.8, 11.6]
-0.0288
Fingertip position algebraic notation
x15
y-coordinate of the target
-0.0307
Angular speed in rads/sec
[0.62, 3.73]
angular velocity of the first arm angular velocity of the second arm
-0.1320
0.1214
x-value positionmmeeitp position
3.9565
y-value of positio0mmgmpp position 3.9607
Figure 13: Preprocessing of state observation in Reacher environment.
space. Using this full observations with their corresponding binary labels, we train a reward model in a supervised learning framework. The architecture of this reward model is a fully connected NN with one hidden layer consisting of 32 nodes, employing a ReLU activation function characterized by a leaky parameter 0.01. In the case of PbRL, three such NNs initialized at random to create ensemble. Before asking the human for preference, has most uncertainly, specifically, the pair for which there is the most disagreement among the ensemble's predictive outcomes. During policy training, the agent observes the average reward from the ensemble.
sample random trajectorics and then choose the pair whose preference
We perform the experiments three times, each with a different target location, for every algorithm under consideration. The human feedback was provided by one the authors of this paper. The shaded area Figure 7 shows the standard error between the three experiments. Note that the plotted lines have been smoothed using a convolution operation with window size of 100 episodes, which helps reduce noise and reveal clearer data trends.
B.3 Rubik's Cube
Each trajectory consists of len steps, during which the agent can perform one of 18 actions: rolating any layer clockwise or counterclockwise, or making no movement. The experiment begins with the Rubik's cube in its solved state, where cach face consists of squares of a single color. By default, evaluators view the cube from the front-facing layer, but they can use keyboard keys to adjust the angle and observe other sides of the cube.
The Gym Rubik's cube environment observation is a 54-element array with values ranging from to 5, representing the colors of the entire cube in numerical format. We slice the array to extract the 9 items corresponding to the front face and we reformat it into 3x3 grid. We also map the numerical numbers to letter representing the color; e.g., instead of '0' we write a 'W' to represent white. A list of ten 3 x 3 arrays, each representing the front face state at one timestep in the trajectory, is added to the user prompt, along with the text feedback. We don't include any landmarks in this environment.
Although the environment discrete and countable, the vast number of possible states of face makes tracking the reward distribution for each state impractical. To address this, we model both reward and policy functions with NNs. The policy function is parameterized by two-layer neural network, where the first layer maps the observation space to 228-dimensional latent representation, followed by ReLU activation. The output layer maps this representation to the action space dimensions, applying softmax activation to produce a probability distribution over actions. Similarly, the reward function is modeled as a feedforward neural network with a single hidden layer of size 8. The input first transformed through a fully connected layer, followed by a ReLU activation. The output layer then produces scalar reward value. The reward model is trained via supervised learning using the dataset generated by pre-trained gpt-40 model from the human feedback. We train the agent with the REINFORCE algorithm (Williams) 1992).
23

---

Published in Transactions on Machine Learning Research (01/2026)
(a) RLHTF with default learning rate
(b) True reward with tuned learn- ing rate
Figure 14: Obtained pattern in the front face when aiming to create an orange X (a) RLHTF achieves the desired pattern with 10 human feodback and default hyperparameters whereas (b) the true reward with tunod learning rate converges to a local maximum after 90,000 iterations of REINFORCE.
Figures 14 show the pattern the agent converges to when tasked with obtaining an orange 'X' on the front face of the cube. With the true environment reward, the agent converges to local maximum, resulting only three of the required squares being orange. In contrast, RLHTF successfully learns the desired pattern. To produce the results in Figures 14a and Sb one of the authors acted as the evaluator and provided the following ten pieces of text feedback:
The first states are very bad. The states at time7, and are all wrong. The goal is have an orange X (orange squared in both diagonals) in the front side.
2. The first 7 moves are wrong. At time 8 if you add three orange squares (two on the left corners and one in the middle) it would be good. The goal is to have an orange X by using 5 orange squares in the diagonals.
Until state 3 and including state 3 it is all wrong. At state we should have three more orange squares the left corners and in the middle. States 5, 6,7 and 8 are very bad. State with orange squares the left corners and the middle would be good. The goal is to have 5 orange squares arranged in the shape of an X.
Until state 5 and including state 5 it is all wrong. state we should have three more orange squares on the left corners and the middle. States 7 and are very bad. State 9 with orange squares on the left corners and the middle would be good. The goal is have 5 orange squares arranged in the shape of X.
5. From 0 to 3 (including both) the states are wrong. The states at time 6 and at time 9 are perfect. The states 5, and 8 are not completely correct as they need two extra orange squares on the right corners. The goal is to have orange squares the shape of an X.
From to 3 (including both) the states are wrong, The states and 10 are perfect. The states 7, 8 and not completely correct. The goal is to have orange squares in the shape of an X.
7. The states 6 and 10 are perfect. The states 7, 8 and 9 are not completely correct. The goal is to have 5 orange squares in the shape of an X.
The states and 10 are perfect. The states 7, and are not completely correct. The goal is have orange squares in the shape of an X.
9. The state 2 is very very bad. The states 6 and 10 are perfect. The states 8 and 9 are not completely correct. The goal is to have 5 orange squares in the shape of an X.
10. The states and are very very bad. The states and 10 are perfect. The states 7, and are not completely correct. The goal to have orange squares in the shape of an X. could be an orange X with all the other squares in red all the other squares in green.
24

---

Published in Transactions on Machine Learning Research (01/2026)
6
PA
Reward
RLHTF
PbRL
2000
16000
Episodes
Figure 15: Reward evolution with episodes of the REINFORCE algorithm in the Rubik environment when aiming to construct an Italian flag. We compare the performance for RLHTF with 5 text feedback, PbRL with 5 trajectory comparisons, and true environment reward.
We performed a similar experiment aiming recreate the Italian flag pattern (three vertical stripes: red, white and green) the front side of the Rubik's cube. The results shown in Figure 15 Remarkably, with only five pieces of human feedback, RLHTF nearly doubles the reward achieved by PbRL. However, RLHTF converges to reward of 6.67, not to the maximum achievable score of 7.88. This discrepancy may arise because the performance appears 'good enough' to the human evaluator, in the sense that already fulfills the task. In contrast, when having access to the true reward, the agent may continue learning until it discovers the optimal way to achieve the task. Nevertheless, this 'good enough' level is reached faster with just five pieces of human text feedback than when using the true environment reward at every REINFORCE step.
To better interpret performance trends, we smooth the rewards per episode in Figures 8b and 15 by doing convolution with a window size of 30.
C Policy Learning
The reward model learns measure how close the trajectory is to human intentions. Consequently, find the optimal policy, the agent may directly query the reward model, instead of the human evaluators. This approach significantly reduces time, energy, and monetary costs during policy learning (Christiano et al. 2017 The interactions are described by the solid arrows in Figure 2
The agents follow a greedy policy, they select actions that maximize the expected future rewards as in In simple tabular settings, we use dynamic programming to find the optimal solution. Tn more complex or continuous environments, we model the policy with fully connccted NN, which takes the obscrved state as input and outputs an action. We train the network using the REINFORCE algorithm (Williams) 1992), where the reward signal is given by the reward model. Following standard practices from RLHE, compute several REINFORCE epochs before querying the human evaluators and updating the reward model again.
We would like to highlight that our contribution lies in efficient reward modeling with fewer but richer human interactions. The agent learning phase, which takes most computational burden, follows standard RL algorithms, such as REINFORCE, whose computational complexities are well-ostablished prior literature. We run all our experiments on an Intel(R) Core(TM) 17-7800X CPU 3.50GHz processor, most learning was done in minutes and experiment took longer than couple hours.
25

---

Published in Transactions on Machine Learning Research (01/2026)
An agent is trying to learn and follow a specific path in @grid_hhighhX(grri__wdtt) grid map.
Your job to translate the feedback of the current trajectory into feedback types, locations the map, and a label.
the feedback type imperative, compute what locations in the map the instructions are referring to and label them as either goodd (go to) or 'bad' (avoid),
the feedback type evaluative, determine what locations in the map are being referred to by the feedback and whether the feedback positive or negative,
CoT
feedback type descriptive, compute what new locations the agent should have visited, and label them as positive,
Output format
the getReward function only return JSON file with the specified shape enclosed double quotes.
For example: if the user's input is (example feedback), then the output should be (example output), .utput)
Few
Another example: the user's input is (example_feedbacae then the output should be @tample_output)
Shot
Figure 16: System prompt for Gridworld.
Prompt Engineering
The system prompt design has a strong impact on the algorithm's performance. Next, we explain some the design choices.
We define the coordinates in the grid using chess style notation, i.e., a letter and number pair indicating the column and row respectively. This format ensures clear spatial referencing, even with older LLM versions like GPT 3.5, which often struggle with traditional Cartesian coordinates, e.g. (2,3), as it can be ambiguous whether the first number refers to the horizontal or vertical coordinate. With chess notation, we remove this ambiguity and improve the model's ability to interpret spatial information correctly.
Additionally, we follow Chain of Thought. ( (CoT) prompting (Wei al. 2022) to enhance reasoning when processing human feedback. Since foedback inherently context-dependent, we introduce an intermediato classification step where the LLM calegorizes the feedback into different lypes, such as goal description, imperative instruction trajectory evaluation. This categorization helps structure the interpretation of feedback based on its intent.
In accordance with few-shol prompling (Kaplan et al. 2020 we provide demonstrations to steer the model lo better performance. By exposing the model to relevant cases, we reduce ambiguity and improve performance.
Lastly, we use function calling to force ChatGPT's output to have a prespecified json format, which enables seamlessly extract the necessary information in rownstream tasks. The desired output format for the Gridworld, Reacher and Rubik's cube environments described in Figures I8 and 24 respectively. While
the full system prompts are shown in Figures 16
19
20
22
and
26

---

Published in Transactions on Machine Learning Research (01/2026)
Frid_height= grid_widht 10
max_coLLeeterr="]
example_ feedback "
("feedback": "it should not go below the bed",
"landmarks": ("clock": ("""f"""(""2""),,, "bed": ["2c", "2d""}.
"trajectory": ("1b", "1c", "1d", "1e", "1f", "1g", "2g", "3g", "4g", "]h"]]
exapple_output www.otuuuuutooooooooooooooooo ['location': [1cc '1d'J, 'label': 'NEG' feedback_type"; 'mperrttive))
example example_ feedback22baa2e-
("feedback": "the last couple steps are good",
"landmarks": ("clock": ["2f"], "bed": ["2c", "2d"]],
"trajectory": ["1b", ("'bb, "1c", "1d", "1e", "1f", "1g", "2g", "3g", "4g", "4h"]]
example_outpurz = ('location': ['4g' '4h'], 'label': 'POS, feedback_typpd 'evaluative')
Figure 17: Parameters for system prompt in Gridworld.
27

---

Published in Transactions on Machine Learning Research (01/2026)
"name": "getReward", "parameters":
"type": "object", "properties":
"locations": {
"type": "array"; "
"type": "string",
"pattern": [1-grrd h height][a--mma httpa/aaaaaaaaaaaaaaaaaaaaa col_letter]]
"description":
("Locations in the grid referring to feedback with row numbers and a lowercase letter "for columns. The rows are numbered from bottom to top (1 is the lowest row, increasing" "
"as you move upward), and columns are labeled from left to right (a-j). For example, '1b" "refers to the lowest row in the second column from the left.")
}
[
"type": "string",
"enum": ("POS", "NEG"),
"description": "The feedback's connotation, positive negative."
},
feedbackktype"::
"type": "string",
"enum": ("imperative", "evaluative", "descriptive"),
"description":
Imperative: Feedback includes instructions on what locations are good or should be avoided.
Evaluative: Feedback an assessment of the current trajectory.
Descriptive: Feedback is about modifications for an improved trajectory.
required": """uurred"""lccattton""""" ["locations", "label"]
Figure 18: Function calling to force output format in Gridworld.
28

---

Published in Transactions on Machine Learning Research (01/2026)
You will return state-reward pairs for two-joined robotic arm named 'Reacher-v4;
The arm aims to reach targets with its end effector (fingertip), and you must assess what states good bad based human observers' feedback. As input, you will receive:
1. The natural language comment by human observer who has seen the simulation.
2. The location of landmarks which are circles of different colors.
3. Trajectory of a simulation of the robot trying to reach target. Each timestep is described by:
a) The fingertip position letter representing column (left 'a' to right 'z') and number representing row (down '0' to up '26")
b) A item list of angles in degrees corresponding to the first and second joint respectively.
c) A value of 0 the second joint means the arm fully bent, while is completely straight when it -180 180.
d) A item list with angular speed on the first and second joint respectively.
For each set of observer comments, use the provided trajectory and landmarks to determine successful and unsuccessful states.
Follow this steps
1. Classify each section (sentence or linked group of sentences) in the feedback text as:
a) Goal description: It describes where the target is, or where the fingertip should go (e.g.: You
should to the pink dot).
b) Trajectory feedback: It criticizes the simulation observed (e.g.: The first two steps are wrong),
c) Trajectory suggestion: It describes ways to improve upon states in the simulation observed (e.g.:
Go a bit to the right of the state at time 23).
2. Generate state reward pairs depending on the feedback type
a) For Goal description: Provide "reward": +1, "angular_ "eggllarspeedd: [0, 0] at the location the described target position.
CoT
b) For Trajectory feedback:
2.1. Determine whether the feedback has positive ("reward": +1) or a bad ("reward": -1) connotation.
2.2. Determine what state states of the simulation it is referring to, and get the fingertip position, angles and angular speed those locations
For Trajectory suggestion:
2.1. Determine what state states of the simulation it is referring to, and get the fingertip
position, angles and angular speed of those locations
2.2. Correct the states suggested by the feedback and pair with a "reward": +1
Output format
Use the getReward function to only return JSON file with the specified shape.
Figure 19: System prompt for Reacher environment Part 1.
29

---

Published in Transactions on Machine Learning Research (01/2026)
Example inputs with expected outputs are provided below for guidance:
Example 1:
Input:
M'feedback': 'The last half very bad, it should go bit higher than the blue dot,
'landmarks': Y'yellow': 'b3', 'blue': '66,, 'white': 'm7; 'orange': 'c23"\,
'fingertip pingertippposition: ['k15;, k14; k1 k14; k13 [12],
'angle': [[00.7, -63.9), [33.2, 68. (35.7, (3.7, (38.3, -76.9), [40.8, -81.3), [43.2, -85.7],
'angular_sae :peadd: [[0.1, -0.06) [0.12, -0.12), [0.13, -0.17,, (0.15, -0.23,, [0.16, -0.29), [0.18, -0.34]] y
Expected Output:
\"referred_stene steps":
("fingertip_positionn. position": 'k14', "angle": [38.3, -76.9), "angular :ecillrrspeeed::: [0.15, -0.23], "reward": -11),
V"fingertip_positiee position": 'k13', "angle": [40.8, -81.3),
[0.16, -0.29], "reward": -11),
("fingertip position": '122' "angle": [43.2, -85.7). "angular speed": [0.18, -0.34], "reward": -1\,
V"fingertip_positionn" '17,, dangular_speed":: 0], "reward": 1V},
Few Shot
'18,, tingllarspeeede [0, 0], "reward": 1VJJV
Example
Input:
('feedback:: 'The fifth step but slower good. Stop at the last point. The goal go to the white
point.;
'landmarks': V'yellow': 'u,,, 'purppeu: 'k12;, 'white': 'w16'))
'fingertip position': ['k15', k14; k14 (114; (12'),
'angle': [[12.2, 34.2], [11 39.6
[8.2, 4124222421|433333000000222222239
'angular speed': [[0.1, -0.06, (0.12, 0.12], [0.13, 0.177, (0.15, -0.23), [0.16, -0.29), (0.18, -0.34]] y
Expected Output:
M"referred steps":
("fingertip_posiiteee 'k13', "angle": [5.9, 56.6], sangular_speed" [0.08, -0.15), "reward": 11),
\"ffngertip_poosiiior" position": '(12', "angle": [3.1, 62.3), "angular speed" [0, 0), "reward": 11),
V"fingertip
V"fingertip_ pittreerttpppsstiooonn 'w16' "angle": [13.0, 10.1], Pangular_speede [0, 0], "reward": 1V).JV
Figure 20: System prompt for Reacher environment Part 2.
30

---

Published in Transactions on Machine Learning Research (01/2026)
pattern - (tt---////000111220-22262/6666
FUNCTION_ STRUCTURUUUURERE
"name": "getReward",
"parameters":
"type": "object".
"properties":
"referred_steps"::
"array""
"items":
"object"
"properties":
"fingerti__posee p
"type":
"pattem":_paaten _patter,
"description": ("Location
fingertip row and top lowest row, increasing" you move upward), columns labeled from left right (a, 2).
columns. rows are numbered
"For example, refers the lowest row second column left.")
angle":(
"type":"arryy"
"items":
"number."
'format": "float".
"minimum":--18"
"maximum": 180,
"description": "Angle degrees for first and second joint of the arm."
"minitems":
"maxitems": 2,
"description": "A 2-item list angles degrees corresponding the first and second joint
respectively."
aangular_ (
"type": "array,, "items":
"type": "number"
"minltems":
"maxitema": 2,
"description": ("Vector of two elements corresponding the angular velocity of the first and second
arm respectively."
"If feedback refers location with positive reward, but without specifying the speed, set the
angular speed to 03.")
}.
'reward": "type": "integer",
"enum": [-1,
"description": "The reward value, which can be +1 for good performance or -1 for bad performance!"
).
"required": liigertip_positionn "reward"], Propertitt's 2,
"description": ("Dictionary containing information about the state and its reward." "
"Must have the 'reward' and Eingeripp_position keys and optionally other keys:" "'angle', or 'aagular__aa "aangllrr_veloitty" velocity.")
).
"description': "List of states described by the feedback along with their reward implied by the feedback."
required": cistarred_seppst
Figure 21: Function calling to force output format Reacher enviromment.
31

---

Published in Transactions on Machine Learning Research (01/2026)
You will assess the state-reward pairs of the front face of a Rubik's cube based on human observer feedback with the goal achieving a specific pattern.
Objective: Identify successful and unsuccessful states of the Rubik's front face cube based on observer comments after viewing 10-move simulation.
Input Format:
1. Observer Comments: Natural language comments by human observer who has seen the simulation.
2. Rubik's Cube Trajectory: States of the Rubik's cube for 11 timesteps:
You will receive 11 states (initial state 10 subsequent states).
Cube State: Defined by 3x3 matrix for the front face of the cube, each cell representing color (e.g., [[R, G, BJ, [W, Y, O), [B, R, G]]).
Processing Steps:
For each sentence or comment in the human feedback:
1.
Classify Human Feedback:
Goal Description: Describes the target pattern the cube (e.g.. "You should have all red squares on the middle column.").
Trajectory Feedback: Criticizes the observed simulation (e.g., "The first two steps wrong, but setup at time was good").
State Suggestion: Suggests corrections the cube's state (e.g., "At time 3 you should have another red square the top right corner").
2. Generate State-Ree Reward Pair:
Goal Description:
reward: +1
state: The target state described by the comment (a 3x3 matrix representing desired colors).
CoT
Trajectory Feedback:
Determine the connotation of the feedback (positive: reward: +1, negative: reward: -1), Identify the index of the specific state(s) the feedback refers to, and return the input state(s) corresponding to such index.
State Suggestion:
Identify the index of the state referenced,
Modify state as suggested.
reward: +1.
Return the Results:
Check your results
Output format
Use the getReward function to only return a JSON file with the specified shape.
Figure 22: System prompt for Rubik's cube environment Part
32

---

Published in Transactions on Machine Learning Research (01/2026)
Example inputs with expected outputs are provided below for guidance:
Example 1:
Input:
V'feedback':
'The top row should white and there should blue in the lower right corner;
'state0': [['R; 'B', ' [';; ['; 'B', WW]].
'state1': [['B; 'W; 'O',, ['G: Y, R1), [';;
['; BB W
Expected Output:
M"state": [[WW, WW; W'), ['G, Y,, 'R', rO; BB, 'B']],
"reward": +11
Example
Input:
Few
Shot
('feedback': 'The end is bad. The last state with yellow the bottom left, and another yellow the top the middle column of the front side would good.;
'state0': [['B; 'W, 'O'], ['G;, Y 0'0; 'R']],
'state1': [['B; 'W, 'R', ['G;, 'R' W'], ['0; 'B, 'G']],
'state2': [[ G,, 'R'], [' RR, W/], rO; 'B, G
Expected Output:
V"state": [['B; 'G, [G; 'R,, [',, 'B, 'G']]
"reward": -1]V,
V"state": : [['B; YY; 'R', ["), WY, IY; 'B, 'G]]]
"reward": +1]V
Figure 23: System prompt for Rubik's 's cube environment Part 2.
33

---

"type": "object", Published in Transactions on Machine Learning Research (01/2026)
FUNCTION STRUCTURE {(
"name": "getReward",
"parameters"::
"type": "object", "properties":
(
"type": "array, "items":
"properties":
"state": "array"
"description": "A 3x3 grid representing the state of the board. Each subarray corresponds to row from left to right, with the first subarray representing the top row, the second representing the middle row, and the third representing the bottom row.",
"type": "array"
[
"string,
"enum": ["W", "G", "O", "B", "R", "Y"]
},
"minltems": 3,
"maxitems": 3
"minltems": 3,
"maxitems":: 3
"reward": "type": "integer,, "enum":[1,
"required": ("state,, "reward"]
}.
"required": ["states"]
Figure 24: Function calling to force output format in Rubik's cube environment.
34


Reviews:
[{"review_id": "M0XElmojEZ", "reviewer": "Reviewer_bg8B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes RLHTF, a framework where humans provide free-form natural language (NL) feedback on trajectories, and a large language model translates this text, together with simple “landmarks” and trajectory information, into state-level labels used to train a reward model. The authors study the effect of feedback granularity in a small Gridworld with a user study, and present additional proof-of-concept experiments on a low-dimensional Reacher task and a reduced Rubik’s Cube “front-face pattern” task, showing that RLHTF can outperform sentiment-based and preference-based baselines with relatively few feedback interactions.\n\n**Key strengths**:\n1. Timely and interesting direction: using rich natural language as a higher-bandwidth reward signal.\n\n2. Clear high-level pipeline (context encoding → LLM translation → reward update).\n\n3. Some empirical evidence (user study, multiple toy domains) that structured NL feedback plus an LLM can be more effective per interaction than PbRL/sentiment.\n\n**Key weaknesses**:\n\n1. All environments are very simplistic; there is no convincing demonstration on a realistically challenging RL task.\n\n2. Baselines are narrow (mainly PbRL and sentiment); standard sparse-reward RL or GCRL baselines, which likely perform well here, are missing.\n\n3. Critical methodological components (landmark extraction, LLM translation function, reward model update) are not accurately described in the main texts and heavily rely on environment-specific prompt/landmark engineering.\n\n4. The writing repeatedly overstates what the experiments show, claiming to “prove” robustness in continuous and structurally complex environments, that RLHTF can “rival access to privileged ground-truth rewards,” and that it “overcomes reward modeling challenges” and will “continue to benefit from future advances in LLM capabilities.” These claims are based on a single small Gridworld, a simple 2-DoF Reacher task, and a heavily simplified Rubik front-face toy, all with carefully chosen baselines, which does not justify such broad, forward-looking statements.", "claims_and_evidence_yn": "No", "claims_explanation": "The central claims are not convincingly supported by the current evidence, for several reasons:\n\n1. **Toy-scale experiments vs broad claims**. The strongest claims (e.g., that RLHTF can “rival access to privileged ground-truth rewards,” “prove robustness” in continuous and structurally complex environments) are based only on a tiny tabular Gridworld, a very simple 2-DoF Reacher, and a heavily simplified Rubik’s “front-face pattern” toy. These settings are far too narrow and easy to justify such general statements about robustness, scalability, or practical impact.\n\n2. **Missing strong baselines**. The method is mainly compared to preference-based RL and sentiment-based rewards. There are no comparisons against standard RL or GCRL methods with simple sparse rewards, which could almost certainly solve these environments. As a result, the claimed advantages over “baseline methods” may simply reflect the choice of weak baselines, not a genuine advantage of the approach.\n\n3. **Under-specified methodology**. Critical parts of the pipeline are only described at a high level. Landmark extraction, the LLM “translation function” from free text to state-level labels, and the reward-model update procedure are all presented as black boxes with environment-specific prompt engineering and minimal algorithmic detail. This makes it hard to assess reproducibility or to understand when and why the method should work.\n\n4. **Overinterpretation of fragile results**. Several headline claims rely on very specific, fragile phenomena (e.g., slightly better early error reduction than ground-truth feedback in one Gridworld protocol, or escaping a local optimum in one particular reward shaping for the Rubik toy). These are then generalized into broad narratives about overcoming reward-design difficulties and benefiting indefinitely from future LLM advances, which goes well beyond what the data actually support.\n\nTaken together, the experiments provide some initial, suggestive evidence that natural-language feedback plus an LLM can be useful as a reward signal in simple settings, but they do not provide accurate, convincing, and clearly argued support for the stronger claims made in the paper.", "audience_yn": "No", "audience_explanation": "While the general direction is undoubtedly interesting and relevant to TMLR, this particular submission does not, in my view, produce findings that are useful or general enough for the journal’s audience. The experiments are confined to very simplistic, highly engineered settings, and the methodological details are too vague to make the approach directly reusable. As a result, it is hard to distill any clear, general statement or actionable insight that a TMLR reader could take away and apply beyond the specific toy setups considered here. The work feels more like an early research than a mature contribution that would meaningfully inform or influence the broader community.", "requested_changes": "1. **Stronger, non-toy experimental setting**.\nInclude at least one genuinely challenging RL task (e.g., high-dimensional / vision-based locomotion or navigation), make it the primary case study, and relegate most toy examples (Gridworld, simple Reacher/Rubik) to a supporting role.\n\n2. **Add appropriate, strong baselines.**\nCompare against standard RL / GCRL methods with reasonable sparse or shaped rewards (and tune them fairly), not just PbRL and sentiment. This is necessary to show RLHTF is useful beyond weak or contrived baselines.\n\n3. **Fully specify the methodology.**\nProvide enough detail to reproduce the method in the main text: how landmarks are obtained and encoded, the exact LLM prompting and output format for the translation function, and the full reward-model update procedure (architectures, losses, optimization, data aggregation).\n\n4. **Analyze robustness and human-effort trade-offs.**\nInclude experiments with non-expert / noisy feedback and quantify the cost of feedback (number of interactions, words, or time) versus performance, including more realistic PbRL regimes.\n\n5. **Align claims and writing with evidence.**\nRemove or substantially soften overstatements (e.g., “prove robustness,” “rival ground-truth rewards,” “overcome reward modeling challenges”) and clearly present the work as an initial proof-of-concept with explicit limitations.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OF3pROrYlV", "reviewer": "Reviewer_rnic", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes RLHTF (Reinforcement Learning from Human Text Feedback): using LLM to convert natural language feedback from humans into state labels to train a reward model. Specifically, the overall pipeline is divided into three steps: 1) Context encoding: packaging the execution trajectory, corresponding human feedback, and some information into a formatted prompt; 2) Translation: feeding the above prompt into the LLM to obtain data with state labels; 3) Training the reward model: training the reward model using the data obtained in Step 2. The above process can accelerate convergence and improve learning efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Clear method: Clearly explain the core of the method and the overall pipeline: Context encoding -> Translation -> Training the reward model\n2. Enough experiments: The paper compares the RLHTF with multiple baselines (true trajectory/true state, sentiment score, PbRL) in various environments, and reports metrics such as success rate, error/reward curve, and significance, and provides analysis and conclusions.\n3. Detailed analysis: Analysis of performance, efficiency, and details.", "audience_yn": "Yes", "audience_explanation": "RLHF (and human-computer alignment) is an important topic in itself, and readers will likely find it interesting. At the same time, the RLHTF proposed in this paper is a concise and clear method that helps achieve more efficient alignment, making it attractive to the community.", "requested_changes": "1. The proposed method relies on prompts, which are specifically designed and have limited generalization ability. This limitation is discussed in the authors' future work; further related preliminary experiments or a more in-depth discussion could be considered.\n2. The proposed method relies on the translation capabilities of LLM (step 2), a process prone to hallucinations. The authors currently mitigate this by providing guidance to evaluators, but this issue still requires further exploration. This approach also relies on human behavior. The authors could consider more mechanized approaches to alleviate this problem, such as back-translation consistency checks and multi-model consistency verification.\n3. Consider providing an ethical statement explaining the potential risk according to the 'Broader Impact Concerns'.", "broader_impact_concerns": "1. The article mentions recruiting volunteers for the experiment, but the authors have clearly stated that minimal risk research also provides compensation.\n2. Feedback from small groups of participants may introduce some biases (e.g.,  cultural/gender/language).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IhoF51W0Eo", "reviewer": "Reviewer_R64J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes  Reinforcement Learning from Human Text Feedback (RLHTF). It allows the LLM to recieve the human's feedback along with the reward to improve the performance. The proposed method contains two phase: Reward learning (described in Section 4) and policy learning. In experiments, the proposed method achieves better performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "All four contributions are clearly supported in this paper. For its constribution 1 and 2, the proposed RLHTF framework indeed ultilizes the human feedback to enhance the reward model in the RL fine-tuning. For its contribution 3 and 4, the empirical experiments support that the proposed method achieves better performance. As a result, these claimed contributions are supported by these evidences.", "audience_yn": "Yes", "audience_explanation": "As RLHF is a hot topic, many audience would be interested in knowing this result.", "requested_changes": "It is hard to connect Section 2 Problem Setting with existing LLM fine-tuning route. It would be better if the author could clearly clarify how each part is defined in RLHF. For example, what are the state, action, transition, and reward defined in RLHF and the proposed RLHTF setting. Currently, Section 2 is just a general description of RL, which is not very useful to understand the problem setting. \n\nOn page 3: \"Analogously to other RLHF algorithms,\". I am not sure if other RLHF algorithms will require to train a reward model. In many math reasoning tasks, the reward model is just a text parser to extract the correct answer from the model's response. It doesn't really need to train an additional model. I may suggest the author to further clarify here to explicitly state the detailed setting.", "broader_impact_concerns": "No any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)