ROSETTA: Constructing Code-Based Reward from Unconstrained Language Preference
Authors: Sanjana Srivastava
Kangrui Wang
Yung-Chieh Chan
Tianyuan Dai
Manling Li
Ruohan Zhang
Mengdi Xu
Jiajun Wu
Li Fei-Fei
Venue: ICLR 2026 Poster (2026)
Keywords: reward generation
LLMs for robotics
human evaluation
Primary area: applications to robotics, autonomy, planning
TL;DR: We aim to use LLMs to generate reward from unconstrained language with changing goals; this requires structure and domain knowledge in prompting, and rigorous human evaluation.

Abstract: Intelligent embodied agents not only need to accomplish preset tasks, but also learn to align with individual human needs and preferences. Extracting reward signals from human language preferences allows an embodied agent to adapt through reinforcement learning. However, human language preferences are unconstrained, diverse, and dynamic, making constructing learnable reward from them a major challenge. We present ROSETTA, a framework that uses foundation models to ground and disambiguate unconstrained natural language preference, construct multi-stage reward functions, and implement them with code generation. Unlike prior works requiring extensive offline training to get general reward models or fine-grained correction on a single task, ROSETTA allows agents to adapt online to preference that evolves and is diverse in language and content. We test ROSETTA on both short-horizon and long-horizon manipulation tasks and conduct extensive human evaluation, finding that ROSETTA outperforms SOTA baselines and achieves 87% average success rate and 86% human satisfaction across 116 preferences.

Published as a conference paper at ICLR 2026
ROSETTA: CONSTRUCTING CODE-BASED REWARD FROM UNCONSTRAINED LANGUAGE PREFERENCE
Sanjana Srivastava Kangrui Wang#12, Yung-Chieh Chan Tianyuan Dai Manling Li
Ruohan Zhang Mengdi Xu Jiajun Wu', & Li Fei-Fei!.?
"Department of Computer Science, Stanford University
"Together AI Department of Computer Science, Northwestern University
"Scale AI
$ DDepartment of Computer Science, University of Texas at Austin
"Inttitute for Interdisciplinary Information Sciences, Tsinghua University
Cluman-Centeerd Artificial Intelligence Institute Correspondence: ssrivastavv@toogetie. ai
ABSTRACT
Intelligent embodied agents not only need to accomplish preset tasks, but also learn to align with individual human needs and preferences. Extracting reward signals from human language preferences allows embodied agent to adapt through reinforcement learning. However, human language preferences are unconstrained, diverse, and dynamic, making constructing learnable reward from them major challenge. We present ROSETTA, a framework that uses foundation models to ground and disambiguate unconstrained natural language preference, construct multi-stage reward functions, and implement them with code generation. Unlike prior works requiring extensive offline training to get general reward models fine-grained correction on single task, ROSETTA allows agents to adapt online to preference that evolves and is diverse in language and content. We test ROSETTA on both short-horizon and long-hov horizon manipulation tasks and conduct extensive human evaluation, finding that ROSETTA outperforms SOTA baselines and achieves 87% average success rate and 86% human satisfaction across 116 preferences.
Origi net trok
Ex3
E-25
Bebs
Short hor
the
the bine, tirk aupp thteets too
ROSETTA
Lata cown.
Preference Grounding
tosrist 711111
Long horizon
Paze the apple the draves
Avel 1i,
Staging
apim_e e tanhiangoosy
tavel 5.05
Gevi yoh again
tta??
new spin stg?
whi Lgt
Move the bulls the second lnn
Coding
ball
targers withh
acalp bat.pos
[0, chist,
Figure 1: ROSETTA's three-module structure takes unconstrained, unseen natural language preference and generates reward code that incentivizes desired behavior. ROSETTA can adapt to diverse behav- ioral preferences, not only in position but orientation and speed. We validate ROSETTA's 's performance in two short-horizon and three long- aonghooizonn task-agnostic environments.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Hluman-centeree embodied intelligence requires that humans be able to guide embodied agents to align with their preferences (Bobu et al. (2024); [eike et al. (2018)). For agents operating in closed-loop interactions, this means aligning with each human preference in its most natural and unconstrained forms {Li et al. (022)) ) as shown in Fig. | Reinforcement learning (RL)-based embodied agents have demonstrated the ability to adapt to high-level tasks originating in human language given dense rewards (Srivastava et al. (202T), and the efficiency of specifying reward rather than collecting extensive training data makes RL promising testbed. However, humans have unique voices and changing goals. Adaptation requires handling unconstrained language and unseen goals that edit, build on, even contradict prior goals every step. Generating effective rewards under such conditions is an open problem.
Reward modeling enables creation of nuanced reward signals without expert shaping. While many existing methods are general and multi-task, often able to adapt to changing, unseen goals, they require extensive end-to-end training (Chen et al. 202T); D: Arora Doshi (2020)) Due to their large-scale training across diverse domains, foundation models (Bommasani et al. (2022) offer a compelling alternative to custom reward model training. Their code generation capabilities offer a natural conduit between language and dense scalar reward. However, prior methods like Eureka (Ma) jet al. 2024bb) and Text2Reward (Xie et al. (2024b)) that work for embodied agent manipulation are limited to clean and structured language, predetermined goals that remain fixed over the course of interaction with the human, and several rounds of feedback that aims only to refine that fixed goal, rather than assert the human's 's own evolving preferences and receive immediate adaptation.
In this paper, we aim to generate rewards for language preferences that do not obey these constraints. First, the underlying tasks requested in these preferences are not cleanly phrased. Individuals express themselves in whatever way is natural and efficient. Second, the preferences have dynamic and unseen content that requires single-step adaptation. This means that whatever goal comes up, we aim for good reward function in a single step. And third, evaluation of adaptation to naturally expressed preferences is itself' a bottleneck. is impossible measure alignment with training metrics such as success rates alone, as they do not address adherence to the preference, only to the goal in the reward function which may not match the preference.
To address these challenges, we present ROSETTA (Fig. 2p: Reward Objectives from Spontaneous Expression Translated Agents. ROSETTA is code-based reward generation pipeline that enables embodied agent adaptation to human natural language preference in a single step. ROSETTA generates rewards from preference statements that have no constraints on language, limited constraints on content, and compound over time in long interactions. It contains a Preference Grounding module to disambiguate and contextualize in prior preferences and execution, as well as a Staging module that repurposes LLMs' known planning abilities (Huang al. (2022a)) to add structure and stage rewards. Both use gpt 4o (OpenAl et al. (024a)). The Coding module then turns the staged reward specification into coded reward function with ol ol-mini (OpenAl et al. generates rewards even from preferences that change the entire shape of a task in just few words.
ROSETTA therefore
We also propose evaluation framework consisting of three key metric categories. Most important is alignment, measuring human satisfaction with the policy. Allowing unconstrained preferences means that unlike in prior work, training metrics only indicate how well the RL algorithm optimized the reward, not whether the reward itself is aligned with human intent. Measuring alignment requires direct human input (Conitzer et al. (222)); Delgado et al. (222)). but can be noisy and incomplete, motivating additional metrics. First, we do also want rewards that are suitable for RL. We therefore measure reward optimizability via policy success rate. Second, looking only the trained policy is insufficient for determining whether the reward reflects the preference. It may match semantics but not be optimizable, policy rollouts given to the human have some stochasticity, and the human may rate a behavior well because they like it even though they didn't 't ask for it. We therefore directly measure semantic match, measured by expert evaluation of the reward code itself.
We validate ROSETTA iteratively: multiple steps of taking language preference, generating a reward, training an agent, evaluating, and taking a new preference. We evaluate on 35 sequences of two to four preferences each, total 116 preferences, in five task-agnostic manipulation environments (Fig. [I]. ROSETTA successfully interprets ambiguous language, adapts to unseen preferences even after four interaction steps, and produces semantically matched and optimizable rewards that result in aligned
2

---

Published as a conference paper at ICLR 2026
policies. It outperforms Eureka and Texr2Reward (Sec. (.2)). structured approach for immediate and general adaptation. We present three main contributions:
demonstrating the importance of its
ROSETTA, a framework that generates code-based rewards. These enable embodied agents adapt unconstrained, diverse, dynamic human language preferences online.
An evaluation framework for measuring human-embodied agent interactions in the wild, consisting of metrics for alignment, semantic match and optimizability.
Evaluation in five manipulation tasks on 116 human preferences, showing ROSETTA's state- of-the-art performances: 86% human satisfaction, 78% semantic match, and 87% success rate. It shows 33% better human satisfaction than baselines.
RELATED WORK
Foundation models on human-robot interaction and control. foundation models have trans- formed the landscape of robotics by enabling natural language interaction. Early works focused on affordances and executable control code (Ahn et al. (2022): Liang et al. (2022) Huang et al. (2022b), establishing the foundation for language-guidn paradigms, developing systems that can interpret and incorporate real-time verbal feedback and corrections during task execution (Cui et al. (2023); Liu al. (2023); Zha al. (2023); [hh al. (2024) 2025)). These interactive approaches have been further extended to handle complex spatial reasoning and hierarchical task planning (Huang et al. (2023); Li ct al. (2023); Wang et al. (2023); Chen et al. (2023)). The field has also seen sigmificant advances owwn-ppeedeeeeeoo embodied control and personalized human-roba interaction systems (Wu et al. 2023a b): et al. (024b))
robotics. Recent work has explored more interactive
Foundation models in reinforcement learning. The emergence of foundation models has sparked new directions in reinforcement learning, particularly in the context of robotic tasks. Vision-language models (VLMs) have proven effective as both success detectors and zero-shot reward generators (Du
et al. (2023); Wang et al. (2024)). Developments in value learning have explored personalization of multi-objecttve rewards and in-context value function adaptation (Hwang et al. (2023) Ma et al. (2024a) ), capturing nuanced human preferences.
LLMs for reward engineering. The most relevant developments to our work lie in the direct application L.MMs for reward engineering. Recent approaches have demonstrated increasingly sophisticated methods for translating natural language into reward functions, ranging from basic reward shaping to full code generation (Xie et al. (2024a); Ma et al. (2024b); Yu et al.) (2023); Li et al. 2024c); Clark al. (2025); | Yu al. (2025)). Some approaches have explored automated reward generation through progress functions (Sarukkai et al. (2024) or evolutionary techniques guided by human feedback (Hazra al. (2024)) These methods require fixed task definition. They
also require pre- and hand-defined goal-specific supervision like fitness functions or goal-specific
information in the prompt.
Existing approaches require either pretraining or fixing the goal and using only feedback that is clean and helps achieve that goal. Though ROSETTA is a prompting framework like Eureka and Texr2Reward, its structured, domain knowledge-informed reward generation enable adaptation to unseen, unconstrained, dynamic goals without training. The core innovations lie in the problem setting of natural human preferences without constraint on preference; the learning task set up by our data; the conceptual structure underlying our approach; and the evaluation suite.
PROBLEM FORMULATION
Our aim is to turn human language preferences into semantically matched, optimizable reward functions that lead to aligned embodied agent policies. We aim to do so for ongoing chains of preferences, adapting to each individual preference while also adapting to the compounding meaning over time and avoiding degradation in later interactions. We formulate it the reward generation problem for unconstrained human language preference adaptation, extending the reward generation problem presented in Ma et al, (2024b).
3

---

Published as a conference paper at ICLR 2026
Favorite verient
Reward function verients
Images ajecttryy time
red on the botson purpla cabe."
Pick gaeen cubs.
(tvand graaped
swourd()
Grounding
Staging
2. Place on sed cub....
Domain knowledge verification
PActually
stepl...
Humen feedback
Structured feadback
Reward stages
ol ol-mini
gpt-4o
gpt-4o
ol-mini
</> S
Eorative conrection
01-mini
Figure 2: ROSETTA diagram. A trajectory from the latest policy and preference history given to the Preference Grounding module. The resulting grounded preference is given to the Staging module. The Coding module takes resulting staged reward plan and generates reward variants with various domain knowledge reasserted. Iterative error correction conducted.
The reward generation problem (Ma et al. (2024b)) takes a task description and generates a reward function R that maximizes known fitness function F(AAs (R)). where Ans (Rt) is learning algorithm run on world model M and reward to get policy æ. The fitness function when the task goal is predetermined and does not change. With unconstrained human preferences, the fitness function denoted F(t) is instead unknown and time-dependent. It can be used to aid the adaptation process across different t, but unlike Eureka or Text2Reward, we do not assume this. Instead, we aim for each F(() to be maximized, so that we minimize user queries for any one goal. We note that the fitness function exists only in the mind of the annotator and their satisfaction is the objective of the method, meaning human involvement unavoidable.
known
Reward generation for unconstrained human language preference adaptation. An adaptive embodied agent interacts with humans through multiple iterations of preference. At preference iteration t 0, the agent (0)) is trained optimize given task with text description ((0) Then at any following preference iteration E [1, 7J, human preference annotator watches rollout of policy (((-1) and gives language preference (() Given the latest policy π (1-1) = AM (
and history of human preference {7(0) (1) ...., (0) the objective is to output a reward function R(O) such that (Asr (R is maximized. In this problem, (+((() is the human's 's satisfaction level with a task execution controlled by (1) == Ar (R()). We once again note the importance of maximizing (t)) at every t, rather than using multiple (2) to maximize one F.
3.1 EVALUATION FRAMEWORK
ultimately human satisfaction that matters, meaning need to measure complex, noisy quantity that itself is given on the policy rollout video, a variable and open-ended stimulus. At the same time, ROSETTA'S 's reward functions need to be evaluated for semantic match and optimizability. We therefore propose three sets of metrics on the set of reward functions detailed as follows.
Alignment requires thorough human survey. We propose a) binary and ternary descriptive questions regarding the preference, such as "did the robot incorporate your preference that wasn't 't met in the previous video." " "did the robot contradict any part of your preference" etc. b) Binary and Likert questions that quantify satisfaction (Hyman & Sierra (016)). Full survey text in Sec. D.2.1
Semantic match requires establishing that the goal in the generated code matches the goal in the preference. Given that reward code is a long-form generation and therefore difficult to objectively evaluate (Rosati et al. (2024)), we propose expert text evaluation. ROSETTA'S 's preference-code pairs evaluated by roboticists, also with binary and Likert questions such "does the reward function miss part or all the reward plan" or "are there common sense errors". Full survey text in Sec. D.2.1]
4

---

Published as a conference paper at ICLR 2026
Optimizability measures whether the generated reward function is suitable for the learning algorithm Aar(:) (-). We use the success rate on evaluation rollouts to tell us how well the reward function can be optimized given adequate models and training algorithms.
4 APPROACH
We introduce ROSETTA, which made of three prompt-based modules: Preference Grounding (Sec. I Staging (Sec. (.2), and Coding (Sec. (S)c. As seen in Fig. 2 these modules are built on each other to get the final reward function. All prompts are included in Sec.
4.1 PREFERENCE GROUNDING
Generating rewards from unconstrained human language preference first requires that the preference be contextualized and disambiguated. Foundation models rely on the letter of the prompt, so the system needs match vague references, consider prior preferences, etc. For example, consider "no, get it to the center." "it" needs to be mapped to self obj. "center" needs to be mapped to "center self. target. Even "get" could be mapped "push". This preference was given on reward function that already aimed for the agent to push the ball to the center, so the "no" needs to be recognized as rejection of the performance "can it start further back?" and this needs to be retained despite not being mentioned.
not of the goal. Finally, the previous preference was
We therefore introduce Preference Grounding module that uses trajectory images, symbolic states, and single-sentence task description to get per-step language descriptions. From these, it generates a summary of task performance, grounded human preference, and new single-sentence goal for use in the next interaction step. The above example becomes "the agent did not push the ball to the center of the target as requested. should do so while still starting further back." Here we use gpt- go (OpenAl et al. (0224a)) given its strong vision- asiion-language capabilities.
4.2 STAGING
In unconstrained language preference, single sentence can completely change the task goal without providing guidance on steps take. However, dense reward often requires breaking the task into stages and rewarding each one. We automate staging by leveraging its similarity to task-planning foundation models are adept at planning (Huang al. (2022a). ) and unlike many settings that require fixed output structure, here we can allow the foundation model to stage in language space and capture nuances embedded in the language. Given "nestle the ball into the bin", the Staging module generates plan including "Stage 3: Reward the agent for placing the ball in the corner of the bin with a tight threshold". These function as concrete design notes for the Coding module.
The Staging module takes four inputs: existing environment code in Ma et al. (024b). generated task deseription, generated demo summary, and grounded preference. gpt-4o is used here as well.
4.3 CODING
The coding module implements the plan created by the Staging module. It takes several key elements: a history containing the staging plan, environment code, summarized documentation on environment attributes, Grounding outputs, desired function headers, and importantly, a checklist of domain knowledge. In ROSETTA, this includes knowledge on densifying reward, masking staged rewards, and considering problem geometry: all crucial to reward success. The full checklist is in See. C For long-horizon action primitive environments, the Coding module generates target positions, dense position reward, and success conditions for each stage. These are inserted into a task-agnostic template ensure action space adherence. In short-horizon continuous control, all code is generated.
The Coding module uses ol ol-mini. Many foundation models, even vision- vision-language models, halluci- nate spatial reasoning (Li et al. (2024a)). However, reasoning models (OpenAl et al. (2024b)) known not only for their coding ability but also their ability to apply long contexts and instructions. ol a1-ming n achieves this with relatively low cost.
5

---

Published as a conference paper at ICLR 2026
Alignment
Optimizabliity
40
TowrdRwwadd
Rurvlc
Behavior
Beluviat
Figure 3: ROSETTA vs baselines, compared on optimizability and alignment for goal, behavior, preferential, corrective, and context-dependent preferences. ROSETTA outperforms all preferences except corrective, as expected given Eureka and Rexrd are built for the corrective setting.
Alignment
Optimizahility
8OSBTTA
Neggowiding
Ne-stegiig
12
Ni-fellonepp
Goal
Hahavvor
Prof
Contest
Gool
Hehavior
Prof
Contest
Figure 4: ROSETTA vs ablations, compared as in Fig ROSETTA outperforms on alignment metrics for all preferences other than corrective. For optimizability. DOSETTA-no-grounde because ungrounded preferences often result in minimal edits the prior reward. If is optimizable, the success rate will be high despite it being misaligned. See Sec. 5.2.4 for analysis.
appears superior
Verification questions for domain knowledge. While reasoning foundation models are strong, long context can result in information loss (Chuang al. (2024)). Giving the same information in smaller chunks can promote adherence. At the same time, the foundation model is unlikely to completely change strategies once the initial code has been written. We therefore give domain knowledge in the initial coding prompt, then reiterate in sequence of verification instructions. We train and evaluate a subset of the resulting variants. As prior works (Xie et al. (2024b); Pan et al. iterative error correction the first and final variants. Generated reward code is run in the simulator, with errors corrected by ol -mini. Across our main set of preferences, 79.35% of generated reward functions ran before error correction, and 95.38% ran after.
(2024)),
4.4 POLICY TRAINING
ROSETTA results in V N reward functions, where V is the number of variants and N is the number of end-to-end generations. V fixed based on the number verification questions, whereas N is hyperparameter. We train at most three variants per generation given resource constraints. To select (1)) at iteration step f, we take the best policy from each reward variant's training run and let the preference annotator choose. Consider the set of variants R (4) = ROSETTA(A()) For all R E R()), we take policies Aas (R)x) where k* is the training iteration with the highest success rate; if success for all training iterations is O,
maximum average accumulated reward. The chosen policy is (1) Since the annotator chose its reward function R" is by definition the one that maximizes (r() (A)()))) Given the goal of investigating automated reward generation and selection, we choose hyperparameters a priori for each environment, but do not tune for any specific reward function. With sufficient resources, systematic hyperparameter tuning can be integrated.
EXPERIMENTS
ROSETTA aims handle natural human preference in way that is satisfactory that human. We investigate this through four research questions:
Q1: Does ROSETTA generate adaptive reward functions given a sequence of human preferences?
Q2: Can ROSETTA handle diverse, unconstrained human language?
Q3: Can ROSETTA handle dynamic, unseen preferences, even as they compound over time?
Q4: Does the proposed evaluation framework offer insights beyond standard training metrics?

---

Published as a conference paper at ICLR 2026
The robt needs amooliyy wilh loss Triccing when placing The sphare doen...
you my
oaser & slowsr you wort "linch"
could you carry ball the the oecond Nin..
during cropping the ball..
Figure 5: Visualization of diverse human preferences. Each column is one human preference, and each group of columns is preference sequence. Colored cells indicate applicable categories: orange for language, blue for content. A full list of preference categories can be found in Sec. D.3
5.1 EXPERIMENT SETUP
Training and environments. We conduct experiments in two settings. Short-horizon: 7-DoF operational space control, training with proximal policy optimization (Schulman et al. (2017), two task-agnostic environments (SphereAndBins and (allAndTTrrge)) each with preference sequences of length four. Long-horizon: hierarchical policy trained with soft actor-critic (Haarooj et al. (201 8a)) and primitive skills adapted from MAPLE (Nasiriany et al. (20226). three task-agnostic environments AndjectsAnDDrawerrrr each with five preference sequences of length
(ThreeCubes, ObjectsAndBins two or three. For both, we use ManiSkill 3 (Tao al. (2024)) and the Franka Panda (Haddadin et al. (2022) Annotators are sourced from Upwork (Upwork (n.d.) Full details in Sec. F.2
Baselines. We compare ROSETTA to Eureka (Ma et al. (2024b) and Text2Reward (Xie al. (2024b), Both use multiple rounds of language feedback to optimize fixed goal, with the language feedback being purposely written to help shape this reward. Eureka also uses multiple rounds of best-of-n sampling, supervised by hard-coded ground-trute fitness functions. We test each method by inputting a human language preference, then having that human select among policy videos from the generated functions. As noted in Sec. 3 consider human evaluation be the fitness function for all methods. We begin with the original ManiSkill reward, then evaluate every method after a single round of application to test immediate adaptation. We compare on both of our short-horizon task-agnostic environments, as well as two more used in Text2Reward that were not seen during ROSETTA 's development. We apply the training pipeline described in Sec. ol-mini is used for both methods, as they both involve code generation in every generation.
Ablations. We compare ROSETTA to three ablations: 1) ROSETTA-
no-gouiin. groundinn. which we give the original human preference directly to Staging. 2) DOSETTA-nO-staginn in which we ground the preference and give it directly KOSETTA-nO-followe in which we do not apply any follow-up questions.
version of Coding with all mentions of "stage" removed. 3)
Metrics. For aligmment, we consider percent preferences where the annotator was more satisfied and average Likert-scale satisfaction score. For semantic match, we consider seven-point Likert scores for each of Grounding, Staging, and Coding. We multiply these to get a Cascading score. For optimizability, consider the average training success rate and percent of preferences that have policy with success rate >50%. Detailed definitions of each metric are in Sec.
5.2 RESULTS
5.2. ROSETTA GENERATES ADAPTIVE REWARDS TO SEQUENCES OF PREFERENCES (Q1)
ROSETTA generates semantically matched, optimizable, and aligned reward functions for sequences of human preferences. As shown Tab, 2 improves satisfaction on 86.0% of preferences and achieves mean satisfaction score of 79.0%. It has even higher success rate and percent success >50,, and similarly successful performance on semantic match; all measured across both short- and long-horizon. Per-interaction step, we in Tab. T that performance maintained
context
7

---

Published as a conference paper at ICLR 2026
and changes layer each other, with at least 80% of preferences resulting in satisfaction and at least 75% satisfaction score in each step. There is some degradation in training metrics and degree of satisfaction, often caused either by the preference annotator doubling down on difficult preferences or hallucinations from prior steps contaminating the current prompt. The latter is a key failure mode of prompting with environment code that includes the prior reward function; examples in Sec. B
Comparing to baselines. We see in Fig. 3 that ROSETTA outperforms Eureka and Texr2Reward, particularly when the preference changes task requirements. We compare on preferences that are cor- rective (give corrections on the same task requirements), preferential (add/change task requirements), goal-related (address success definition), and behavioral (address other parts of reward landscape). While the basclines are similar corrective preferences, ROSETTA outperforms cases: preferential, goal-related, and behavioral. We expect that with
task fask-changing the correct (human-defined) fitness function each step, baselines would improve in performnace, but only if able to generate code that effectively specified the success condition of every preference. Doing so would require ROSETTA'S 's prompt components, making the resulting method a hybrid.
Cost comparison can be found in Sec. G
To test the value of the approach itself, test with open-source language models: Qwen2.5-VLL-72B Instruct for grounding/stagingg and Qwen3-Coder-480B: A3SB--Isstrut--PPP----------- for coding. ROSETTA with the proprietary models achieves mean best reward of 0.623; with open-source models, it achieves 0.741, giving preliminary evidence of the value of ROSETTA across LLMs. To test the value of the design choices vs. specific prompts, we test with the prompts paraphrased by qpt-4o and Gemini 2.5 (Taa (2025)). Gemini 2.5 paraphrasal leads to 0.584 mean best reward, while gpt -o leads to 0.658, suggesting that the specific prompt matters less than the conceptual structure.
5.2.2 ROSETTA EXTRACTS SIGNAL FROM DIVERSE, UNCONSTRAINER LANGUAGE (Q2)
Human can communicate preferences with diverse language categories as shown in Fig. ROSETTA has consistent performance across language features that are difficult for foundation models, such as specifie positions and context- deteextdeppendet. shown in Figs. 7 and The specifie positions category has complex and often evolving position descriptions. These are challenging for foundation models given the lack of spatial reasoning (Liao et al. (2024). The context-dependent category includes preferences that require prior context, c.g. sequence like "red top" then "other actually." Without context, the second preference would be impossible to interpret even for a human. ROSETTA generates productive reward functions even in these challenging cases, achieving 60% success rate and 92% satisfaction score on specific position preference, and 73% success rate and 76% satisfaction score context-dependent preference. Fig. shows the language diversity of collected preferences.
ROSETTA's 's ability to handle unconstrained language comes from Preference Grounding and Staging, which interpret the language. In Fig. 4 RoSErTA-nn-grounding suffers on alignment for context- dependent cases where environment code is insufficient. RoSETTA-no-staging suffers in general.
5.2.3 ROSETTA CAN HANDLE DYNAMIC, UNSEEN, COMPOUNDING PREFERENCES.
Besides diverse language features, human preferences also contain diverse content as in Fig. 5 ROSETTA maintains performance across content features based on Fig. 7 Policies reach specific positions ("right edge of left bin"") and orientations (" ("give it quarter-tumn") and adapt to behavior preferences ("go slower, that's stress- ful to watch");; more examples in Sec. B.2 We again see ROSETTA'S superior ability to handle changing tasks:
Table 1: Breakdown of optimizability and alignment across iterations. Metrics are as described in Sec. 5.1
Iter.
Succ. Rate
& Suce. >50
More Sat.
Sat. Score
95.2
Tab. ssooss that ROSETTA outperforms ablations goal- related, behavioral, and preferential content. ROSETTA- no-followup matches performance only in corrective cases where verifications are distracting. Tab. demonsttatte that ROSETTA'S 's performance shows only limited degradation, staying above 75% all metrics.
2
89.4
90.0
80.0
75.6
3
94.4
100.0
80.0
76.9
4
78.1
75.0
89.5
78.9
The strength comes from both o1 -i-minn't 's coding and reasoning capabilities and ROSETTA's 's domain knowledge. There is orientation-speccfic prompting, but o1 -mi ní reason about the quater-
8

---

Published as a conference paper at ICLR 2026
Original task: Stack the cuocs in purple, grean, grey order
you pinsee grean?
New pursle bottom grey in midile, and gleen
Pace them Ir groen gry.
grean
Figure 6: Example real robot results. The first element in each color column represents a preference given the policy trained on the original task. The second element represents another iteration of preference given on its parent. We see that simulation-train policies transfer to real effectively.
nions to reward "move the ball in an arc". However, spatial reasoning-heavy preferences such as "corner of the bin" require domain knowledge about how to offset target positions by object dimen- sions; behavioral preferences require knowledge how apply unstaged penalties. Comparison to RoSETTA-no-followup shows that in most cases, review using knowledge is beneficial.
Fig. demonstrates ROSETTA's 's content failure modes. These naturally include preferences that are physically difficult ("throw the ball against the wall and have bounce back to the bin") or impossible ("pick up the ball instead of pushing" when the ball is too big for the gripper). It also includes preferences that are impossible to represent in ManiSkill's 's default reward formulation, which does not include temporal info, such release: preference annotators often ask explicitly for ball to be dropped into a bin from a height. Without any temporal info, this requires an "ungrasp" reward that has no option but to be conditioned on the ball being above the bin. The moment the gripper ungrasps the ball, it stops being "above" the bin and the agent loses reward. See Sec. B.3
5.2.4 EFFECTIVENESS OF THE EVALUATION FRAMEWORK (Q4)
The proposed evaluation framework investigates performance more deeply than standard metrics alone. ROSETTA is performant on all metrics as shown in Tab. 2 but overlap is not given: 73% of human-selected policies have 250% success rate, and only 66% have the highest success rate across all options. Optimizability metrics retain practical value in assessing whether ROSETTA is technically reliable: high success rates indicate behavior that expected from the reward function.
Low-alignment high-optimizabiliity rewards appear when the reward code effectively induces its own success condition, but the success condition itself does not match the preference: a semantic hallucination. A common example: ROSETTA is given preference with nonstandard phrasing and mostly ignores it, outputting a barely-edited version of the previous reward function. The result is good success rate but behavior that does not reflect the input. This is part of the reason the DOSETTA-no-grounde
ablation, in which Staging and Coding receive more nonstandard phrasing, has high optimizability but significantly lower alignment than ROSETTA. It does match ROSETTA'S alignment on corrective preferences, where not changing the success condition is ideal.
High-alignment low-optimizability rewards demonstrate deeper challenges. We find that without aligmment, we wouldn't 't know the annotator's 's true opinion; but without optimizability we don't know the reasons for failure. For example, given preference "push the ball on the table next the bin". the human-sclected rollout shows gentle, ineffectual push toward the bin with a success rate of O: the high-success rate policies pick and place instead of pushing. Key misalignments: for the annotator, the push is more important than the position, but this is unclear from the language. Second, ROSETTA set specific target point because this is typically more optimizable than a target region, but here that point was difficult to push to. Third, the reward function did encourage closed gripper, but the success boost outweighed the opening penalty.
5.2.5 REAL ROBOT EXPERIMENTS
We take sim-to-real approach for real robot experiments. We test on four two-iteration preference chains in the ThreeCubes long-horizon aggostii environment for which policies have been trained in simulation. We track positions of the cubes from single RGB image using FoundationPoss (Wen jet al. (2024)), then populate the state dictionary required for our long-horizon action primitive setting. This results in observation equivalent to ManiSkill's 's standard state state_dict dict. The average first.

---

Published as a conference paper at ICLR 2026
iteration success rate is 72.5%; the average second iteration success rate is 80.0%. Examples shown in Fig. [6] We observe no sim-to-real gap from the policy.
LIMITATIONS AND BROADER IMPACTS
One key limitation of our work transfer to other simulators. ROSETTA task-agnostic, but it would help to know the effort required to transfer domains. Another limitation is the expert evaluation for Semantic Match metrics. Analytic/lcarned metrics may be cheaper and more objective.
This paper enables adaptation of RL-based embodied agents to human language preferences, and uses foundation models to do so. Certain human preferences may be less represented a foundation model''s 's domain and result in less aligned performant rewards. Furthermore, preventing harmful generation is an unresolved ethical and technical question. This paper is designed to an asset that empowers users: frameworks like this one can and should be fitted with further domain-specifie safety measures as they advance. We design ROSETTA to he easily extendable to more domain knowledge.
7 CONCLUSION
We aimed to investigate how reward signals can be extracted from unconstrained, diverse, and dynamic human language to train embodied Al agents. To this end, present ROSETTA. ROSETTA takes in such language, extracts desired task content, and generates useful reward functions even when that content changes and contradicts prior content. Given the inherent challenge of evaluation under unknown, subjective ground truth, we present an interactive task structure and metrics of both alignment and objective efficacy. ROSETTA is demonstrated be semantically matched with preference, optimizable, and aligned with human preference.
ACKNOWLEDGMENTS
This work is in part supported by the Stanford Institute for Human-Centerea AI (HAI), AFOSR YIP FA9550-23-1-0127,, ONR N00014-23-1-2333 ONR MURI N00014-22-1-27440, and ONR MURI N00014-24-112748.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu. Mengyuan Yan, and Andy Zeng. Do as can, not as say: Grounding language in robotic affordances, 2022. URL fittps://arat
o abs 2204 01691
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress, 2020. URL pattss://arxtttt org/ /taabs/1100 06877
Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie A. Shah, and Anca D. Dragan. Aligning human and robot representations. In HRI, pp. 42-54 2024. URL
1145/
0610977.3634993
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Enmon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman. Neel Guha. Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar
10

---

Published as a conference paper at ICLR 2026
Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zancle Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2022. URL
08 07258
Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning geweralizaaeeeeeeeeeeee robotic reward functions from "in-the-wild" human videos, 2021.
16817
Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas A. Roy, and Chuchu Fan. Autotamp: Autore- gressive task and motion planning with ilms as translators and checkers. ArXiv, abs/2306.06531, 2023. URLltttss: htt.a/apppaaaaaaaaaaaaa.mmaaaaaa semant icscholar wwtp:oooo/oooooo/05333
Yung-Sung Chuang, Linlu Qiu, Cheng- Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1419-1411 14399,146..001 Mi- ami, Florida, USA. November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.emnlp-main.66 URL htt http:pps///aaaaaaaoooo.y..o.o///
84/
Jaden Clark, Joey Hejna, and Dorsa Sadigh. Efficiently expprrattnneeppeeessseeeeuuuudid behaiors via language-gun guided preference learning, 2025. URL httLLp/ps///exw.oorr/////// 2ttp://ww.....rba///220000...0
Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf. Emanuel Tewolde, and William S. Zwicker. Position: Social choice should guide AI aligmment in dealing with diverse human feedback. In Forty-first International Conference on Machine Leurning, 2024. URL /ttp:s:/ppeeeeveeeeeeee ew net forum? dow
Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE International Conference Human-Robot Interaction, HRT
'23, ACM, March 2023. doi: 10. 10.1145/5668122:5582222 URL lattp://cx/ doi org.
1145/3568163 3578623
Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. The participatory turn in ai design: Theoretical foundations and the current state of practice, 2023. URL httpps://axxiv
erg/abs// 20.00 221123011..000777
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision- language models as success detectors, 2023. URL https
//arxiv. wwwp://.g//aaaaa//ooooo.ooobb 2btp:///aabb333...//
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018a. URL https?
//openreveww.o htt:/ Htt.:enttttttoommiiii-...
Tuomas Haarnjja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 18X2::1812.2. 2.09055 2018b.
Sami Haddadin, Sven Parusel, Lars Johannsmeier, Saskia Golz, Simon Gabl, Florian Walch, Mo- hamadreza Sabaghian, Christoph Jähne, Lukas Hausperger, and Simon Haddadin. The franka emika robot: A reference platform for robotics research and education. JEEE Robotics & Automation Magazine, 29(2):46 64, 2022, doi: 10. 200//RRA.00113133333
11

---

Published as a conference paper at ICLR 2026
Rishi Hazra, Alkis Sygkounas. Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. Revolve: Reward evolution with large language models using human feedback, 2024. URL httts://arxiv.. org/ haarr//abs//e 2406. 01309
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint urXiv2200.007007 2022a.
Wenlong Huang, F. Xia, Ted Xiao, Harris Chan, Jacky Liang. Peter R. Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning. 2022b. URL
https://api.. swmant icschhoac.cch......
250451569
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023. URL https
//arxiv. org. abs 221.2327.0005
Minyoung Hwang, Luca Weihs, Chanwoo Park, Kimin Lee, Aniruddha Kembhavi, and Kiana Ehsani. Promptable behaviors: Personalizing multi-objectio rewards from human preferences, 2023. URL
wttp////g/abbaaao.ooo 0.09337
Michael Hyman and Jeremy Sierra. Open- versus close-ended survey questions. NMSU Business Outlook, 14, 02 2016.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.07871, 2018. URL
http:// http://aaxivv..o org wwtp:rggg.bbb.roeeee 077871
Boyi Li, Philipp Wu, Pieter Abbeel, and Jitendra Malik. Interactive task planning with language models, 2023. URL httlt: arxiv. org/ abs 02310.1666
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang. Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAV LORAV-OR-II 1k: A benchmark for embodied Al with 1,000 everyday activities and realistic simulation. In 6th Anmual Conference on Robot Learning, 2022. URL TtR.hhtts::/// /ttp::////eeeeeeeeeeeeeee ew wwt/
d= 1==__dd__eee__eeeeee T
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang. Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a.
net www.moorimiio. oot.:://ooruummo11111111l
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking lims for embodied decision making. In NeurlPS 2024, 2024b.
Zhaoyi Li. Kelin Yu, Shuo Cheng, and Danfei Xu. LEAGUE++: Empowering continual robot learning through guided skill acquisition with large language models. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024c.
Jacky Liang Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753 2022.
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 17028 17047, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18.13 http://.222222m/m/... 1aap..maii.99 URL https: //acl: antho wtw.oog...ooo..........oo.o...o. orgp00244/22224/ emn p-ma ma i 9477
12

---

Published as a conference paper at ICLR 2026
Huihan Liu, Alice Chen, Yuke Zhu, Adith Swaminathan, Andrey Kolobov, and Ching-An Cheng. Interactive robot learning from verbal correction, 2023. URL http:::/aaaxxi..oooo org/ abs/
2355.
Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruy Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Oshert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia. Vision language models are in-context value learners, 2024a. URL
04549
Yecheng Jason Ma, William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, net 2024b. URL httttpss://ppereeeee ietaerrrreeeeettt.fuooreer.aeaa Forummidd-EduRuOSS.
Soroush Nasiriany, Huihan Liu, and Yuke Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7477-7484. IEEE, 2022a.
Soroush Nasiriany, Huihan Liu, and Yuke Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In IEEE International Conference on Robotics and Automation (ICRA), 2022b.
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Camey,, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino. Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky. Ben Wang. Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng. Brad Lightcap. Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg. Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shem, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czameckii, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely. David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang. Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Fugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang. Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon. Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O' Connell, Ian O' D'Cnnell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, lya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz. Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lec, Jessica Shich, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu. Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng,
13

---

Published as a conference paper at ICLR 2026
Lindsay McCallum, Lindsey Held, Long Ouyang. Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz I Litwin, Matthew Zeng. Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang. Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang. Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman. Sam Schoenholz. Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver. Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin. Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojeiech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card, 2024a. abs 2410 21276
URL
OpenAl, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong. Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang. Felipe Petroski Such, Filippo Raso, Florencia Lconi, Foivos Tsimpourlas. Francis Song. Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang. Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo. Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavloy, Miles Wang,
14

---

Published as a conference paper at ICLR 2026
Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz. Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen Hc, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024b. URL httpthtttt/aaaatt wwtp:/iivoorvooooooooo abs/ / 2412.22222 16720
Liangming Pan. Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484-506 2024. doi: 10.1162/tacl a a006600 URL htt.ps://aclarcholooy.oooo wtg.o//..or//222 tacl-1] 277
Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Melis Erkan, Yahya Kayani, Satya Deepika Chavatapalli, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing, 2024. URL httphttpt:/aaaee /ttsss//aaxvv../ org/ wtt./or/////aa/eaeeeeeeeeeeee//e http///b///22229///// 09394
Vishnu Sarukkai, Brennan Shacklett, Zander Majercik, Kush Bhatia, Christopher Ré, and Kayvon Fatahalian. Automated rewards via Ilm-generated progress functions, 2024. URL https
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL
Lucy Xiaoyang Shi, Zheyuan Hu, Tony Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint 2rXiv:2403.12911 2024.
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical lssion-laguuage-action models. arXiv preprint 29411 2025.
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday houschold activities in virtual, interactive, and ecological environments, 2021. URL https::////astt
orgg abs 200810001111 03332
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arny Gurha, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai, 2024. httpts:
www.xii.rrgabs///oo 0042
Gemini Team. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical report, Google DeepMind, 2025.
Upwork. Upwork: The world's work marketplace, n.d. URL https: www. upwork. com Accessed: 2025-01-30.
Yufei Wang. Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RI-vIm-f: Reinforcement learning from vision language foundation model feedback, 2024. URL
httpps://arxiv.orr/ab... /202. 03681
15

---

Published as a conference paper at ICLR 2026
Zihao Wang. Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2332.01560, 2023.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects, 2024. URL U 3 08344
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song. Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots,
2023a.
Zhenyu Wu, Ziwei Wang. Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. ArXiv, abs/2307.01 2023b. URL httppttts://pp..eeemee
ID: http:/Corpus:::2555500:
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and
Tao Yu. Tex12reward: Reward shaping with language models for reinforcement learning, 2024a.
URL https://azxi>: wwtpxxvvvrrrr..ooo org/ http/////9./ 11489
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024b. URL https
//openreveew.oo net.
Chao Yu, Qixin Tan, Hong Lu, Jiaxuan Gao, Xinting Yang, Yu Wang, Yi Wu, and Eugene Vinitsky. lcpl: Few-shot in-context preference learning via Ilms, 2025. URL https://axxv..or// 00s/2410.17233
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. Conference of Robot Learning 2023, 2023.
Lihan Zha, Yuchen Cui, Li-Heng Lin, Minae Kwon, Montse Gonzalez Arenas, Andy Zeng, Fei Xia, and Dorsa Sadigh. Distilling and retrieving generalizable knowledge for robot manipulation via lan- guage corrections. ArXiv, abs/2311.10678. 2023. URL https: api. semant icscholar Btc//ooopus1D:225221222
16

---

Published as a conference paper at ICLR 2026
Optimizabliity
Aligument
Preferences
Groundling
Staging
Codlin
Caocading
Success
(%)
Satisfied (R)
Satisfaction
Short Hovison
50
95.7
87.3
84.6
78.
89.2
90.1
84-8
76.4
Long Horiscn
36
80.0
90.0
82.9
78.6
82.2
88.8
$8.8
92.4
Table 2: Experimental results evaluating semantic alignment, optimization potential, and user satisfaction. Semantic match metrics: expert Likert evaluation grounding quality, staging quality, coding quality, cascading (product of all three). Optimizability metrics: Success rate: average success rate of best policy for single preference, Pet Success >50%: percent runs with success rate above 50%. Alignment: Satisfaction: % preferences where annotator was satisfied, Satisfaction Score: preference annotator Likert evaluation how much more/less satisfied (50 is neutral). Note: Cascading Match excludes cases where expert evaluators found the feedback difficult to interpret.
A ADDITIONAL RESULTS
We see in Fig. and Fig. 8 that ROSETTA performs consistently on various types of language and content, as referenced in the main text. This maintained in challenging cases like context-dependent language and behavioral feedbacks where reward strategies are nonobvious. It also demonstrates that alignment and optimizability are not the same, but do relate.
Language
Content
1.0
0.8
Values
0.6
0.4
delta satisfaction
0.2
ses____cceessssss once
0.0
ambiguous
context- context-dependent detedeependnt
colloquial
distraction
mulli-part part-
verbose
asking for change
affirming
behavioral
corrective
inadvertently long-hortzon
goal-related
physically difficult
preforential
specific orientation
speaific position
Category
Figure 7: ROSETTA performance on various categories on short-horizon enviromments. For legibility reasons, we exclude some language categories that foundation models are able handle without additional domain knowledge, such as 3rd-person POV, 2nd-person POV, no POV, curious tone, directive tone, and suggestive We also exclude conditional and physically impossible feedbacks, which are rare.
B EXAMPLES
In this section, present examples illustrating how our pipeline processes human feedback, along with its limitations. Additionally, we provide examples demonstrating the necessity a multi-faceted evaluation framework.
17

---

Published as a conference paper at ICLR 2026
Language
Content
1.0
0.8
Values
0.6
0.4
0.2
delta satisfaction best cesst_sccesssong _nnee
0.0
ambiguous
context contert-dependen dependent
colloquial
verbose
affirmin
asking for change
behavioral
corrective
goal-reian goal-related
preferential
specific order
specific position
Category
Figure 8: ROSETTA performance on various categories on long-horizon environments. For legibility reasons, we exclude some language categories that foundation models are able handle without additional domain knowledge, such as 3rd-person POV, 2nd-person POV, no POV, curious tone, directive tone, and suggestive tone. We also exclude conditional, inadvertently long-horizon, physi- cally difficult, distraction, multi-part, and physically impossible feedbacks, which didn't 't occur in long-horizon experiments.
B. SUCCESSFUL CASES IN SHORT-HORIZON
Here, we present sequence of interactions between our pipeline and the annotator, where ROSETTA successfully adapts the policy to the human's 's feedback and makes continuous progress.
Initial Demonstration: Robot pushes the ball to the goal.
Step Human Feedback: I like the way the robot uses its claw to push force on the sphere
to move it. want to the robot to push the sphere rightward on the table, but as far as robot is able to.
Step 1 Generated Grounded Feedback: Push the ball as far to the right on the table as possible.
Step Generated Goal: Push the ball as far to the right on the table possible.
Step Demonstration:
The robot pushes the ball all the way to the right
Satisfaction Score: 87.5
Step 2 Human Feedback: I can see the "effort" of the robot trying to push the ball right. I want to see the robot push the ball in the best spinning motion can
Step 2 Generated Grounded Feedback: Push the ball using spinning motion.
18

---

Published as a conference paper at ICLR 2026
Step 2 Generated Goal: Push ball goal position using spinning motion.
Step 2 Demonstration: The robot spins the ball sideways, pushes the ball to the right.
Satisfaction Score: 75.0
Step 3 Human Feedback: like the effort given to perform the task. Although was not complete, I enjoyed it. It was hard task. It kind of rolled the ball in big circle maybe it could try using its gripper to just rotate the ball?
Step Generated Grounded Feedback: Use the gripper to rotate the ball directly towards the goal instead of moving it in circular path.
Step Generated Goal: Push ball to goal position by rotating directly towards the goal using the gripper.
Step 3 Demonstration: The robot pushes the ball towards the right of the target while
rotating it sideways.
Satisfaction Score: 50
Step Human Feedback: do not want the robot to try to push the ball the goal position anymore. The robot performed good prior. I want it to try to make the ball rotate in place, maybe by twisting its gripper on top. want to see the ball spin.
Step 4 Generated Grounded Feedback: Stop trying to push the ball the goal position.
Instead, make the ball rotate in place by twisting the gripper on top of it to see the ball spin.
Step Generated Goal: Make the ball rotate in place by twisting the gripper on top of it to see the ball spin.
Step Demonstration: The robot slowly rotates the ball by twisting the gripper.
Satisfaction Score: 100
B.2 SUCCESSFUL CASES IN LONG-HORIZON
Here, we showcase instances where our pipeline successfully handles feedbacks to long-horizon environments.
Example 1: Success Rate: 0.9, Satisfaction Score: 100
Human Feedback: Place the light colored items in one bin and the dark objects in the other.
Feedback Categories: no pov, context-dependent, preferential, goal-related, specific position, directive, asking for change.
19

---

Published as a conference paper at ICLR 2026
Generated Grounded Feedback: Place the apple and orange in one bin, and the baseball and tennis ball in the other bin.
Previous Goal: Place the apple, orange, baseball, and tennis ball all into same bin.
Generated New Goal: Place the apple and orange in one bin, and the baseball and tennis ball in the other bin.
Previous Demonstration: The robot picks up the objects and puts them into the white bin one by one.
New Demonstration: The robot placed the baseball and tennis ball in the white bin, and apple and orange in the blue bin.
B.2.1 EXAMPLE 2: SUCCESS RATE: 1.0, SATISFACTION SCORE: 100
Human Feedback: Can the robot pick up the other items on the floor? It doesn't 't appear complete with its task leaving the other items there.
Feedback Categories: 3rd-person, verbose, suggestive, preferential, context-dependent, corrective, asking for change.
Generated Grounded Feedback: Pick the red soup can and place it in the drawer.
Previous Goal: Put apple in drawer.
Generated New Goal: Put Apple and Red Soup Can In Drawer
Previous Demonstration The robot picks up the apple and places it in the drawer.
New Demonstration The robot placed the apple in the drawer, then the soup in the drawer.
B.2.2 EXAMPLE 3: SUCCESS RATE: 0.6, SATISFACTION: 100
Human Feedback: am glad the robot was able to completely stack the three cubes in the push the green cube next the pink cube. After, want the robot to pick the red cube and drop on top of the two cubes.
correct order. would like the robot
Feedback Categories: 3rd Srd-person, preferential, behavioral, specific order, specific position, goal-related, asking for change.
20

---

Published as a conference paper at ICLR 2026
Generated Grounded Feedback: Move the green cube next the purple cube. Then, pick up the red cube and place on top the green and purple cubes.
Previous Goal: Stack the green cube top of the red cube, then stack the purple cube on top of the green cube.
Generated New Goal: Move the green cube next to the purple cube. Then, pick up the red cube and place it on top of the green and purple cubes.
Previous Demonstration: The robot stacks three cubes together.
New Demonstration: The robot pushes the green cube next to the purple cube and places the red cube top of them.
B.3 EXAMPLES OF LIMITATIONS
B.3.1 LIMITATION 1: MISTAKE ACCUMULATION
specific failure case we observed occurs when the language model propagates errors from the current reward function to the new one. In this case, the LLM mistakenly added a +0. offset to the z-coordinate of the target position. In the previous step, due to a correctly defined success condition and some luck, the robot was able to successfully learn the task. However, this error was inherited in the generated reward function, leading to unsuccessful learning in the subsequent step.
Previous Reward Function Snippet:
def' (gge_d0_eward()) ():
# Target position computation # Action: pick up the red apple
# Target position is slightly above the apple to approack from above apple_pos _pss_pps__ prev_ine info ['apple_ppe target_pos__ apple_pos. coos. ()
[x, y, z]
target_pos. [2] += 0.1 # Raise by 0.1 meters above the apple
Generated Next Step Reward Function Snippet:
def Saree__rewardde ():
# Target position computation
# Action: pick up the red apple # Target position
slightly above the apple approack from above
apple_pos = prev
target_pos. prev_info[ a'ppll'ppppppp "appe_pos''' # [x, z]
np. ((
apple_pos [0].
apple_pos [1]
apple_pos [2] 0.1 # 0.1 meters above the apple
D)
21

---

Published as a conference paper at ICLR 2026
Previous Demonstration: The robot puts the two objects in the drawer and closes it.
New Demonstration: The robot repeatedly tries to pick up the apple but doesn't 't lower the gripper enough.
The flawed reward function ended with 0 success rate and 0 satisfaction score.
B.3.2 LIMITATION 2: CONTENT CONSTRAINT
Inadvertent long-horizon scenario.
Step Feedback: I want the robot to release the sphere after placing it.
Step Demonstration: The robot lifts the cube above the bin but fails release it.
Step 2 Feedback: I want the robot to put the ball in the corner of the bin
Step Demonstration: Still, the robot lifts the cube above the bin but fails to release
Step 3 Feedback: I want it to not try to first bring the ball above the corner then release. It seems that's 's not going to work. It should put it straight down into the bin comer.
Step 3 Demonstration: The robot picks up the ball and moves it the comner of the bin. Though this is listed as an example of fundamental limitation, this final adaptation despite contaminated history shows the power of the modular framework recover and adapt to helpful human input.
22

---

Published as a conference paper at ICLR 2026
B.4 CONSTRAINTS
While ROSETTA accepts unconstrained language, we established certain content constraints for annotators, though we accepted some violations show the system's limitations. Constraints:
No temporal dependency. Since the state history is not given to the reward function except step back in the action primitive case, we disallowed preferences referencing history. An example: "pause your gripper 2cm above the sphere, then continue down." This cannot be implemented with history-agnostic masking. The most successful agents paused 2cm above the sphere and never progressed.
No long-horizon tasks in the continuous control setting - described above. This was violated many times as it can be nonobivous to annotator.
To some degree, please accept jitter. Stabilization is possible, but it's not the default behavior. If you want stabilization, please say so explicitly.
Avoid asking for multiple goals at once (multi-objective reward). This was rare regardless, as annotators quickly recognized that asking for two things at once was unrealistic.
Avoid asking for ungrasp in continuous control: this turns out to be difficult learn since grasping is usually the right move. The algorithm did learn it in many cases, but this was more due to its own sampling leading success boost than the dense ungrasping reward.
Action primitive setting only: can only give feedback on positions, cannot ask for pulling action.
PROMPTS
C.I
SHORT HORIZON REWARD GENERATION PROMPTS
To generate the reward function for a short- environment, we follow the steps outlined below. The entire pipeline takes the following inputs:
Current Environment Code: The full ManiSkill environment code, including the current
reward function.
Grounded Feedback: Grounded human feedback.
Demo Summary: A generated language description of the demonstration.
New Task Goal: The updated task goal based on the feedback.
The generation process is carried out through multi-round conversation with GPT-4o, following these steps:
1. Staging: In this step, the LLM is tasked with planning the reward function based on the feedback. (Prompt
2. Coding: The LLM is tasked with generating reward function code snippets according to its plan. (Prompt[2
3. Error Correction 1: We execute the generated code and ask LLM to regenerate the reward function with the error trace if the code fails.
4. Geometry Review: The LLM reviews the generated code and corrects any geometry-related
mistakes. (Prompt 3)
5. Target Position Reward Review: The LLM reviews and corrects target position setting mistakes in the generated code. (Prompt
6. Reward Design Review: The LLM reviews and corrects reward function design mistakes in the generated code. Ptrompt5aaamm (Prompt(6)
7. Error Correction 2: We execute the generated code and ask LLM to regenerate the reward function with the error trace if the code fails.
Prompt used:
23

---

Published as a conference paper at ICLR 2026
Short Horizon Staging Prompt
Instructi in
You are a reward engineer that is an expert at designing reward functions to solve reinforcement learning tasks. You will output outlines for two functions, "evaluate" and srrpute_densereeaaee natural language. They should concrete and ready for implementation.
You will be given:
1. Code for task environment class that reinforcement learning- based robot has already been trained in. This includes the reward function the robot was trained on, namely the existing version of compute_dense
2. Feedback given to the ropot by human who watched it after it had been trained on the existing versien compute__enss__eeaat
You should:
1. Generate high level plan for "compute, dense reward', and evaluate" needed. "compute dense reward" is the reward function, and "evaluate" is a helper that analyzes the current environment state and compiles information that is given to computo_dense_rowet
2. Design staged reward:
Split the task into stages and give the agent encouraging it to complete each stage. The reward should therefore accumulas late, NOT be all-or-nothing at the end.
gradually,
Consider interdependencies and tradeoffs between different task stages different aspects Ensure your reward design isn't counterproductive.
the feedback and overall goal.
3. If there are aspects of the feedback that it's impossible to incorporate without modifying other methods, say "I cannet do < aspect>* and ONLY incorporate the other parts of the feedback, if there any.
4. there are aspects of the feedback that are physically impossible, say "I cannot do <aspect>" and ONLY incorporate the other parts of the feedback, if there are any. 5. Explain your reasoning each step.
Rach stage must be +*single outcome**. It must be a change in an envi ronment state, not particular action. It must obey one of the following templates:
<desired position>. Example: " reward the robot for traveling to the point just left of "cubeA", Reward the robot for getting <desired object> to <desired position>, Example: "reward the robot for getting the bettle to be inside of the drawer."
Reward the robot for traveling
Reward the robot for <other single cutcome>, Example: "reward the robot for moving more smoothly."
Reward the robot for <other single descriptor> <other object>, Example: "reward the robot for getting the top cube's edges aligned with the bottom cube's edges."
Tips:
Notice how the stage templates are highly atomic - one single outcome.
if the human feedback has multiple parts, your stages must ST atomic. Just because the person lists multiple parts, this doesn't mean each part is a single stage. One part may still require multiple atomic stages, and one atomic stage may contribute to multiple feedback parts. BE SMART, DON'T JUST LIST OUT THE FEEDBACK AS YOUR STAGES.
IMPORTANT:
24

---

Published as a conference paper at ICLR 2026
Note how the stage templates deal with changes the eavirenment (including the robot) rather than the robot actions.
#P Details for "evaluate" ""yyhhon
def evaluate (selÉ: BaseEnv) -> Dict [str, torch. Tensor]
Return dict is the dict mapping strings of reward-relevant questions to their boolean-valuca answers for the batch. Sc, the values of this diet are torch. Tensers first dimension is a batch of episodes and the second is the boolean answer to the string question for that individual episode.
bool dtype, where the
Should create useful set of information. "evaluate" will be called on both the previous state (before the agent took an action) and the current state (after the agent took an action) to calculate reward.
### Details for "compute_dense_rexaa" "'python
def compute seruute_dense_eeee (sel: BaseEnv, obs: Any, action: terch. Tensor, info: Dict(str, torch.Tenser]. torch. Tensor ***
Encodes reward for each possible action based on "evaluate" output di dictionary and other current instance attributes).
info (envirenment
Incorporates human feedback as given.
Obeys the following structure:
1. Stage reward
Task is split into stages and reward is given to the agent gradually, encouraging it to complcte each stage. The reward accum:lates.
Interdapendenciie and tradecffs between different task stages and different aspects feedback and overall goal are considered. Reward design is not overall counterproductive to meet short-term goals.
2. Extra
bump successful episodes
3. Return reward value
Original code:
"python
(ENVIRONMENT CODE.)
Current task description: (NEM TASK GOAL) Robot execution description: (DEMO SUMMARY) Human's feedback: (GOUNDED FREDBACK)
## Instructions
Think step-by-step to make high-level plan for rewriting
evaluate" and ccmppuee dense cannse"ewwrrr.... The plan should be a series of stages.
Don't code yet, just plan the stages.
Consider dependencies and conflicts between task stages,
create plan that doesn't do anything counterproductive. Carefully consider the physical aspects of the task.
- Plan realistic paths
25

---

Published as a conference paper at ICLR 2026
Consider the physical state the robot after each stage. write the next stage to build directly on it.
RULES
FOLLOW:
1. Incorporate human feedback as given.
2. Do not add anything that contradicts the feedback.
3. Do not invent your
changes or speculate about what the
feedback is going for.
4. Do not add anything counterproduetive
overall geal.
5. Make every stage atomic.
6. Make the plan complete think about all the atomic stages the
robot needs to achieve every part of the feedback, not just the
final ones.
7. NEVER ask the robot to release an object from the gripper.
8. NEVER ask the robot to first move above the desired location,
below desired location, then release lower down or move up. Just have it go straight to the desired location.
Short Horizon Coding Prompt
# instructions
1. Code the reward function based on the stages.
Use the Code Block below wherever possible.
b. Follow the Checklist below.
2. Only output methods you are eaiting.
3. It you are editing method, output the whole edit methed, not just your edits.
4. Don't introduce new methods, not even helper methods. Just edit and "evaluate'.
5. Comment your code as needed.
6. If there are aspects of the feedback that it's impossible to incorporate without modifying '_load scene" _iintiialie_eepiseeee episcde <aspect>" and ONLY incorporate the other parts of the feedback, if there are any.
say "I cannot
7. Explain your implementation.
8. Explain checklist adherence. Be concise.
1. Code the reward function on the stages. Write short chain of reasoning before your code, to explain your reasoning.
Use the Code Block below wherever possible. Follow the checklist belox.
c. Comment your code as needed.
2. Only output methods you are editing.
3. If you are editing method, output the whole edit methed, not just your edits.
4. Don't introduce new methods,
helper methods. Just edit poopuue__ennse reward" and "evaluate'.
5. If there are aspects of the feedback that it's impossible to incorporate without modifying "_load_scene" or "_initialiee_episdn say "I cannot do <aspect>" and ONLY incorporate the other parts of the teeaback, there are any.
Code Blocks for ccomptte_dense_reeaal
### Make Robot Reach Target Positions
dense caanse_reward'.. **Minimize the distance** between the object's current pesition and the target position in the reward function.
In "compute_dense
26

---

Published as a conference paper at ICLR 2026
**Normalize and smooth distances** using functions tanh":
like "terch.
""python
transport_reward 1.0 torch.tanh (torch, lwn.llinll....m norm (target_pos max 1.0, decreases with distance
ob].pos()))
**Don't just give a boost when the target position has already been reached. ** The robot will get no guidance during mevement and be unable to begin.
Function signatures
## Details for "evaluate"; "'ythhn
def evaluate (sedi: BaseEnv) Dict Istr, torch. Tensor]
Return dict is the dict mapping strings of reward-relevant questions to their boolean-valued answers for the batch. Se, the values this dict are torch. Tensors with bool dtype, where the first dimension is a batch of episodes and the second is the boolean answer to the string question for that individual episode.
Should create useful called on both the previous state (before the agent took an action) and the current state (after the agent took an action) tc calculate reward.
of information. "evaluate" will be
Always contains key called "success" that maps success condition
Details for "compure_dense_rexardd"" ""ythonn def ceattc__dens_rrwwrrdeee Tensor, info: Dict (str, Tenscr))))))) -> torch. Tensor
BascEnv, obs: Any, actien: terch.
***
Encodes reward for each possible action based on 'evaluate" output dictionary and other current environment info (eavircnment instance attributes).
Incorporates human feedback as given.
Obeys the following structure:
1. Stage reward
Task is split into stages and reward is given the agent gradually, encouraging it to complete each stage. The reward accumulates. Each stage's reward is dense wherever possible. Interdependencies and tradcoffs between different task stages and different aspects of feedback and overall goal are considered. Reward design not overall counterproductive to meet short-term goals.
2. Extra success bump for successful episodes
3. Return reward value ***
Reward Function Checklist
|| Coding Best Practices
**Use ".clone () ** when calculating and mutating "torch. Tensor's to avoid unintended reward changes
27

---

Published as a conference paper at ICLR 2026
### Selecting Target Positions
When defining target positions, **specify all coordinates (8, Yr
*** Do not leave any unrestricted. "<element>.pos returns the a.eeter positiona of the element. Adjust with offsets if targeting the top, bottom, or sides. **Understand geonetry and dimensicns** by reading the envirenment code.
**Example:** If placing an object inside box, acccunt for wall thickness.
- **Example:** I aligning an object with the edge of target, consider the target's 's shape.
### Staged Reward Masking
++Use torch masking** to activate rewards for stage **only after** its prerequisites are met.
**kxamplexs; Two stagesāAî()) grasp 'bottlea", (2) move "bottleA" to "boxA"
"python
# Assume 'is bottled on boxA' and 'is bottlel grasped' are in info and the target position for "bottleA" is defined # Stage 2 reward moving "bottleA" to "bexA" move_reward torch. (torch. linalg.nnnmmm.........om.oooooaa (targetttt_ bettleA. pos()))
# Reward moving "bottleA" to its target pesition "target_pos" only after it's grasped
infol 'is _ottteee___gaappe move Let "ssboottee__gaspeed"
**Mask based on completed stages**, not the current one. Example of bad masking:
"'pyhhnn
# Incorrect: Masking on the current stage's completion
reward r boxA']] += [ine
is js_botttoo_on_bexal1
# This fails because it docsn't reward reducing the distance between 'bottlcA' to "boxa" until the distancc is zero. robot therefore doesn't get dense process reward.
**Don 't reward conflicting stages simultaneouslye: **Maintain non-decreasing rewards** to encourage progression. **Exampl:*** I= the robot must grasp "abject&", place it, ungrasp, then grasp "objectB":
Continue rewarding the grasp of "objectA" even when it's being placed.
Mask the reward grasping 'objectB' based "objecta" being the target location, not whether "objectA" is grasped.
### Reward Component Weighting
++Equalize the maximum values and shapes of each stage' reward components.**
**Prevent reward imbalance for different, stages+* or the robot may fail to progreas the task because the additional reward is relatively small.
### Success Boost
28

---

Published as a conference paper at ICLR 2026
**Add reward boost** when the task is successfully completed to emphasize successful spisodes.
Knsure "info[" suucesss is boolean indicating success.
Example code:
""python
success sasss__oot______e 5.0 Adjust to 1/4 total possible reward so far
reward (succcsssssssssssssa auccess') + success boost__vvl
(MANISKILL ENVIRONMENT DOCUMENTATION)
Short Horizon Geometry Review Prompt
Verify that for each object, this code handles the fact that : pos ()" returns the center location. Cemmon pitfalls:
Not applying the right offsets when placing two objects relative to each other. E.g. half-width, radius, etc.
Verify that this code considers the physical attributes of each object and how they might offset target positions. Common pitfalls:
Thickknss box walls
Verify that this code uses x as the front-back axis and y as the left-right axis. z is still the up-down axis.
Verify that when applying penalties, this code does not OVER- penalize. For example, if slowness is required, bound for speed, but not penalize entirely or rebet will be stalled.
low upper
1. Write out your verification step-by-step.
2. Edit the code needed according your explanation. Comment your changes.
3. Only output methods you are eaiting.
4. If are editing method, output the whole method, not just
your edits.
5. Don't introduce new methods, not even helper methods. Just edit and "evaluate".
Short Horizon Target Position Review Prompt
Verify that each of the three coordinates is considered for every position and offset in the code. Common pitfalls:
Forgetting the z-component
Ignoring one or two coordinates when setting target position.
If the robot is not given constraints in some axis, it cculd do anything, leading Verify that specific target positicns are always set. Cemmon pitfalls:
crash into scmething else.
Simply incentivizing the robot to go in a certain direction (e. g. "down" "below an object"). It should always be pushed toward SPECIFIC POINT IN SPACE, it won't stop and will fly wildly.
1. Write out your verification step-by-step.
2. Edit the code as needed accorcing to your explanation. Comment your changes.
3. Only output methods you are editing.
4. you are editing method, output the whole method, not just your edits.
5. Don't introduce new methods, not even helper methods. Just edit cempuut_dense_rcviiie and "evaluate".
29

---

Published as a conference paper at ICLR 2026
Short Horizon Reward Design Review Prompt
Verify fy that reward *dense* when, and ONLY when, you want the robot gradually approach a certain state. "Dense" means a continuous function that gives more and more reward as the robot approaches the right position, rather than a step function enly reward it once it's reached.
Distance, angular difference, velocity all continucus values that can have dense, continuous rewards
- Traveling to a location gradually and oponing/closing a gripper to certain point gradually ALMAYS DENSE.
Verify that all spenalties are NOT dense. Examples:
To slow the robot down, there should a penalty on speed that is NOT dense. You want it to just stay below a specific speed, not get more reward the slower it is.
TO HAVE THE ROBOT SIOW, THE CODE SHOULD SIMPLY SET A CONSTANT UPPER BOUND SPEED.
HELP HERE. - To have the robot to keep its gripper closed, the penalty on gripper opening should not be dense - you want it to stay below certain opening, not get more reward the more closed it is.
REWARD
Verify that the code actually requires the robot to reach its target position. Common pitfalls:
- DDfining a "near" throshold that is larger than the "at" threshold for a target position, then not requiring the agent to nove once it's "near" even if it's not "at".
1. Write out your verification by
2. Edit the code as needed according to your explanation. Comment your changes.
3. Only output methods you eaiting.
4. you are editing method, output the whole method, not just your edits.
5. Don't introduce new methods, not even helper methods. Just edit ccomutte_dense_rewaal and "evaluate'.
Short Horizon Masking Review Prompt
Verify that the reward *eveer decreases** during the progression of the task, that the robot isn't disincentivized from making progress.
The code should only mask on genuine prerequisites
The reward should be enabled even when the stage is cemplete
For example, is the robot needs to grasp oaject to carry it to location and then 1et it go, give the grasp reward net only when the gripper is ready to grasp, but also after the object has been carried to the location and ungrasped.
Verify that stage reward compenents Example: reward component guides the robot to put "objA" next "bbj"". Masking this with iinfo["is_ojjA_nn next_ ______et___too___ 11 only get reward after it succeeds, so it will never get started, - Example: reward component guides the robot to travel to point target_pos pos" Masking this with
masked prematurely.
means it'
means it '11 only get reward after succeeds, so it will never get started.
1. Write out your verification step-by-step.
2. Edit the code needed according your explanation. Comment
your changes.
3. Only output methods you are editing.
4. If you are editing method, output the whole method, not just
your edits.
30

---

Published as a conference paper at ICLR 2026
5. Don't introduce new methods, not even heiper methods. Just edit Semmpute_dnse_rewaeee and "evaluate".
C.2 LONG HORIZON REWARD GENERATION PROMPTS
To generate the reward function for long-horizon environment, follow the steps outlined below. The entire pipeline takes the following inputs:
Current Environment Code: The full ManiSkill environment code, including the current
reward function.
Grounded Preference: Grounded human preference.
Demo Summary: A generated language description of the demonstration.
Current Task Goal: The task goal that the policy is currently attempting to achieve.
New Task Goal: The updated task goal based on the preference.
Environment Information Key: Dictionary Keys of the observation space, with corre- sponding descriptions.
Environment Setup Description: A description of the environment setup.
Error Correction: We execute the generated code and ask LLM to regenerate the reward function with the error trace if the code fails.
The generation process is carried out through multi-round conversation with GPT-4o, following these steps:
1. Staging: In this step, the LLM is tasked with planning the reward function based on the preference. (Prompt
2. Coding:
The LLM is tasked with generating reward function code snippets according its plan. (Prompt8)
3. Geometry Review: The LLM reviews the generated code and corrects any geometry-related mistakes. (Prompt 9
4. Reward Normalization Review: The LLM reviews and corrects normalization-relac
mistakes the generated code. (Prompt TO
5. Code Cleanup: The LLM reviews and resolves any coding or formatting errors in the generated code. (Prompt
6. Construct Reward Function: The final reward function constructed by inserting the generated code snippets into Template 12
Prompt used:
Long Horizon Staging Prompt
Instructions
You are reward engineer that is an expert at designing reward functions solve reinforcement learning tasks. We have trained reinforcement learning-based robot in task environment, We then demonstate the policy to a human who gave feedback on the rebot's performance. Now, we want to design a reward function that incorporates the human's feedback You will output an outline for the reward function compute_dense_etee reward', in natural language. should be concrete and ready for implementation.
You will be given:
31

---

Published as a conference paper at ICLR 2026
1. "Original Code": Code for task envircnment class, including the reward function robot has already been trained in, namely compute_dense_rewat
2. "Original Task Goal"; The original task goal of the rebot before the feedback was given.
3. "Simulation Environment Setup": Descriptien the simulation environment.
4. "Demonstration Summary": Summary of the rebot's performance during the demonstration.
5. "Human Feeaback's : Feedback given the rebot by human who watched it after it had been trained on the existing versicn of secuute_enne__eveeee
6. "New Task Goal": The new task goal for the robot after the feedback has been incorporated.
You should:
1. Generate high level plan for "compute_ dense reward".
2. Design staged reward:
Split the task into stages and give agent reward gradually, encouraging it to complete each stage. The reward should therefore accumulate, NOT be all-or-nothing at the end.
- Consider interdependencies and tradeoffs between different task stages and different aspects of the feedback and overall geal. Ensure your reward design isn't counterproductivve.
3. If there are aspects of the feedback that it's impossible to "I cannct do <
incorporate without modifying other methods, aspect>* and ONLY incorporate the other parts of the feedback, if there are any.
4. there are aspects of the feedback that are physically impossible, say "I cannot do <aspect>" and ONLY incorporate the other parts of the feedback, if there are any.
5. Explain your reasoning each step.
Design details # ccoppue__dessorowarrd"
There three stage templates you use. You CANNOT anything else. The robot is ONLY capable these three stages.
#### "pick up"
Template:
Action: reward the robot for picking up <desi red object>.
Outcome: <desired object> is in the robot 's gripper.
Example:
Action: reward the robot for picking up "red cube""
Outcome: "red_ cube" is in the robot gripper.
#### "place"
Templato:
Action: reward the robot for placing <desired object> <desired positions,
Outcome: <desired object> is <desired position>.
Example:
Action: reward the robot for placing 'red_cube" next to "sphereB",
Outcome: "rea_ cube" is next to "sphereB'.
#### "push"
Template:
Action: reward the robot for pushing <desired object> <desired position>.
Outcome: <desired object> is <desired position>.
Example:
32

---

Published as a conference paper at ICLR 2026
reward the robot for pushing 'ree_cube" just left of the center of the target.
Action:
Outcome: rre__cube" is just left of the center of the target.
Output format
Your output should be:
An ordered list of stage plans. Format:
""'mardownn
### stage <stage number>; <language description of stage number> Stage template: "<pick up or place>"
**Action:** <language description of action>
***uttcom:*** <language description of outcome>
**Dependencies:: <reasoning about any dependencies with previous stages>
**Other reasening:** <whatever you think is important note!>
Examples:
""markdown
I! Stage 1: Pick up "objl"
Stage template: "pick up"
**Action:** Reward the robot for picking "objl'. **Outcome:** "objl"
in the robet's gripper.
**Dependenciesi** The robot can do this casily, so there aren't prior dependencies.
++Other reasoning:*:: lopen-enced, you will not be provided with example]
mmarkdown
### Stage 4: Put "obj2" next to "objl'. Stage template: "place"
**Action:** Reward the robot placing "obj2" next to "objl'. *ouucomme:*4 "obj2" is next to 'objl'.
++Dependencies:** Requires that the robot is grasping 'ebj2'. ++Other reascning:*4; [open-ended, you will not be previded with example]
Notice that stage index starts from 0.
Original Code:
python
(ENVIRONMEN CODE)
## Original Task Goal: [CURRENT TASK GOAL]
# simulation Environment Setup: (ENVIRONMENT SETUP DESCRIPTION)
Demonstration Summary: (DEMO SUMMARY)
# Human Feedback: (GROUNDED FEEDBACK)
## New Task Goal: (NEW TASK GOAL)
Think step-by-step to make a high-level plan for writing seapute_ddeseerewarrre
The plan should be a series of stages. Don't code yet, just plan the stages.
Consider dependencies and conflicts between task stages, and create plan that doesn't 't do anything counterproductive.
RULES YOU MUST FOLLOW:
33

---

Published as a conference paper at ICLR 2026
1. Use ONLY the stage templates you'v 'v been given.
2. Incorporate human feedback as given.
3. Do not add anything that contradicts the feedback.
changes speculate about what the feedback is going for.
4. Do not invent your
5. Try accomplish the task goal with minimal stages. Long plans are harder to train on.
6. Make the plan complete think about all the atomic stages the robot needs to achieve every part of the feedback, not just the final ones.
Long Horizon Coding Prompt
Excel lent. Now write the relevant code for each stage.
Instructions:
1. Don't add your own helper methods. Only edit what I've told you
to edit.
2. Go through the coding tips belcw.
3. Explain your implementation.
4. there are aspects of the feedback that are physically impossible, say "I cannot do <aspect>" and ONLY incorporate the other parts of the feedback, if there are any.
5. Number of stages should match the number of stages in the plan. be different from the number of stages in the original code. How to calculate target positions and position rewards
## Coding best practices
Pay attention to the "Original Code" and the "Simulation Environment Setup' given you earlier.
All positions are numpy arrays of shape (3,). Make sure you don't modify the content in info dicticnaries. Don't worry about missing arguments
### How to set a target position for the robot to pick at/place at. Object coordinates are x, Y, and 2 coordinates. =Explicitly reason about every coordinate when setting the target position*s, Note that those objects.
x, y, 2 coordinates of the *center* of
So when setting target positions or getting object positions, if you want the top, battom, or sides of an object, add the right offsets.
Otherwise, make
you' reascning about the center.
Consider common sense physical issues. +*Read the envirenment code to understand geometry and dimension values.** Examples:
- A box has walls, so if you want to put something inside the box, consider its wall width.
you want put something at the edge the target, consider its shape. Remember that direction matters, not just distance. Let's say your intended target position is two block side-lengths away from a block center. You can't take the block center, then tell the robot to go two block side-lengths away from that. It could ge two block side-lengths in any dectiion! You have to cal caleulate your intended position, unambi guously.
target might be circle or square
### Fow write reward component that gets the robot choose the right location
Add a reward component that minimizes the difference between the position the robot is currently planning to to, and the target position you set
34

---

Published as a conference paper at ICLR 2026
Normalize and smooth these differences with function like "np. tanh".
- For example, if the robot has selected position current_selected__
and you set target position 'target_pos", you can take the hyperbolic tangent of the norm of the difference between these two.
- You can also increase the coefficient on the norm within the np.tanh" to encourage "target_paa pos" more aggressively, you can subtract from 1.0 that the overall reward ranges from to 1 and increases as distance decreases, and/or you can use different normalizing function based on the reward landscape you want.
**Design based on what we need from this reward component. DON'T give step function-like boest that only activates when the object has reached its target position. If you do this, the agent won't be able to get started.
Documentation of function inputs Functions take in two arguments from the environmen:: "prev_info" and "cur_info'. These are dictionaries with the following keys:
(ENVIRONMENT INFORMATION KEYS)
'prev_infa" iino' represent the environment state BEFORE "action" taken "cur_info" represent the state AFTER "action" was taken.
## simulation Environment Setup: (ENVIRONMENT SETUP DESCRIPTICNI
Demonstration Summary: {DEMO SUMMARY)
## Euman Feedback: (GROUNDED HUMAN FEEDBACK]
## New Task Goal: (NEW TASK GOAL)
Your task:
Fill out the TODOs in this markdown to get reward. Fill cut new copy of the markdown for every stage. Write short chain of reasoning before each code block to explain your reasoning.
""markdown
stage target action: TODO: select l'pick", "place", "push" Select EXACTLY the one given you in the plan NOT make your own judgement call.
pyython
def compute_targee tww.ttarrrpssiiiootttt Cerssiiinsstaaeeeaeee (self, prev_info, cur :
***
Defines the target position for the robot's action the location it needs to pick at/placc at/push to. Has return. The function ends in the definition of target_pos_l" and "target_pos_2".
Arguments:
'self (RaseRnv)") : gi ves access to envi envi ronment attributes and methed calls.
purev_inão /str, Anyl)': state representation of the environment state BEFORE "action" was taken.
"cur_info (dict [str, Anyl)'; same as 'prev_info', except from the state AFTER "action" was taken.
=**
TODO: implement target position for stageN's target action based on environment.
35

---

Published as a conference paper at ICLR 2026
the action is "pick" or "place", target_pess pes__ must be a 3D position indicating the target position for the action, and _arget_pos__ must be None.
If the action is "push", target_pos__ must be 3D positien indicating the target starting position for the push acticn, and target_pos must be a 3D position indicating the target ending position for the push action.
target_pos__ target_pos_2
"python
def computee_target_ps___aae _tt___oo___rrr__ttaeelll (self, prev_info, cur_info, perrent_seleccte__ppo current_seleen surrent__elecced__po___ target__os__, target_pos 2):
Sets a reward to encourage the robet to take the acticn at target_pos__" and "target_pos_2". This reward should be dense, setting a continuous-valued penalty for any difference between and "current_sell paarent_selected_oo_2 the robot IS CURRENTLY PLANNING to pick at/place at/push tc) and target_pon _os_l"pp__"" and "target_pos_? (the locatien the robot SHOULD pick at/place at/push to). Has no return. The function ends the definition of "reward".
(the location
Arguments:
"self (RaseRnv)'s : gives access to environ attributes and method calls.
'prev_info (dict [str, Anyl)': state representation of the environment state BEFORE "action" was taken.
"cur info (dict /str, Anyl) same taken. and
except
from the state AFTER "action" ccurrent_selected_pos
numpy. sodyynddaray) the pos the robot IS CURRENTLY PLANNING TO take the action at
"narray)) and 'target_pos_2 (numpy. ndarray) : 'target_pos" defined in mamputtetarrgt_positteee the pos the robot SHOULD take the action at
"target_pos_l
Notice that current_selle seerent_seeecte__ca_ 2 and target_pos are None if the action is "pick" or "place".
not then in this This function will only be called selected action equals target action. In other words, current_selected__pe target_pos_2 will be both None or both numpy arrays.
and
reward TODO implement *dense and normalized. reward for guiding the robot to do "target_acticn at "target_pos s, i.e. align coureett_eelecte__pot
and target_pos_1" and "target_pos_2". Use the "target_pos_1" and target_pos_2" param, don't calculate your
with
- Feel free use normalization functions like 'np.tanh", Your reward MUST be normalized to between -1 and 0.
# Ensure your reward is sdensex. Do not give a sudden boost for reaching the target position. Instead, make sure your reward Lmpentataio gradually and continuously guides the robot to target positions. Otherwise, the rebot won't be aple to get started.
>* python
def stageN_ssccees success (info):
36

---

Published as a conference paper at ICLR 2026
Return true is the robot has successfully completed stageN, else return false.
Arguments:
"info (dict [str, Any])": state representation the current environment state.
Returns:
'bool': Truc if the robot has successfully completed stageN, clse Falsc. ***
TODO
Don't write "compute Seempute_dens___eeeee beense_rewaks as a whole. Only write the functions I asked čor.
Long Horizon Reward Function Geometry Review Prompt
Review the setup description again:
ENNIRONMENT SETUP DESCRIPTION)
For every target position, did you specify all three coordinates? Did you target positions +exactly where you desire the robot go+*? Common pitfalls:
Adding a small error threshold te the target position. Always allow a bit of error when calculating #boolean check in stageN_success', but the target positiens themselves should have no error.åa
Did you reason about the dimensiens of each object and how they might impact target positions?
Picking an object up requires positicning the gripper arcund the center of the object. Did you consider this? in other words, did you set the target position to the center of the object?
Verify. If you need to make any edits, do so.
1. Only output functions are editing.
2. If you are editing function, output the whole thing. Only do so once, with all corrections made.
Long Horizon Reward Normalization Review Prompt
you consider the range your nermalization functicns correctly? Common pitfalls: make sure the range of the reward is -1 to 0.
Did you consider sensitivity thoughtfully? Common pitfalls: not in the reward and success
having tolerance for small calculation.
Verify. If you need to make any edits, do so.
1. Only output functions you editing.
2. If you are editing function, output the whole thing. Only so once, with all correct ions made.
37

---

Published as a conference paper at ICLR 2026
Long Horizon Reward Function Cleanup Prompt
you make sure not to incl ude return statements all compute_ target_ potpeet_oosiiio_stttee stagss" and compute target_ pos porsrrewwrdder reward_ssagell target_ pos" and "reward", respectively?
and instead just define
Did you any new instance variables, i.e. did you create any new "self.<vare" <vllnavarraem>'? You should not you should just set target_pos" and "reward" as regular variables. will handle the rest.
Did you the right datatypes? All positions should be numpy arrays shape (3,), rewards should floats, and "stageN_success" returns boolean.
Verify. If you need to make any edits, do so.
1. Only output functions you editing.
2. you are editing function, cutput the whole thing. Only so once, with all corrections made.
Long Horizon Reward Function Template
def evaluate (sclf): info = sclf._get_obb__inee ()
def <GENERATED STAGE SUCCESS CONDITION>
:info)
def stagel_succee success (info):
<GENERATED STAGE N SUCCESS CONDITION>
info (sstage_ssuceess" stage0_success (info)
infol "sstageN_successe" stageN_succest (info)
"sscces"" torch.tensor (False)
if curtccuursttteall
info("succesa) = Norch.tenser (info ("staace_ssccccssss"
info
def Sompute_dense_rewarllsss prev_infa, cur_info, acticn, ** kwargs): coaardd_cmmeenttss diet (kk, 0.0) for in self.
reward soward_compeeettt.
ourrontteelecctddaaciioo = np. np.argaax (action l:l (self.
aurrent_selectedd_
tack_sskil_indicc.kkkss1)))
if current_selected_ettio in [0, 1]:
qurzent_selectee_ppo___ action [len sel.f.taskskkklll ()) +3]
keys jeeeeteell....ak.llllii......... key. ())
curzent__cclectee__pe___ = None
else:
curzent__eelecteed.pps.
action oet
keys ( hit.eneeellttcac.....nnneeeeeees
current_seleected_pps. - action[len (tt.t..tatttttlll_.i.
+3:1++ (sel http:eae...caa.kinnnnnnncceeeeee
def stage reward
<GENERATED TARGET POSITION AND ACTION DEFINITION>
if curront-electtoeaction targot_action:
38

---

Published as a conference paper at ICLR 2026
<GENERATED DISTANCE DEFINITION>
reward_components (Mafford"] - (1 reward) * 5.0
if cur_innfo"staaoo__ccceee
reward dwward_componentee "success"] 10.0
return reward_ coward_componetss
if wwwf.cur_sttaeeeeee
reward Stage_0_reward, ()
if welf.cur. sott.eee
reward stage reward
if (self.cur_stage 0) and
self.cur_stage
if (self.cur_stage = N-1) and cur_info sttage successs
self.cur_stte cur_stage N
return reward
C.3 PREFERENCE GROUNDING PROMPT
Feedback grounding consists of two stages: (1) translating demonstration video and its trajectory into a language-bar based description, and (2) leveraging the language description and human feedback to generate grounded feedback and update task goals.
C.3.1 DEMONSTRATION TO LANGUAGE DESCRIPTION
To address the Vision-L.anuuen Action (VLA) model's limitations in spatial and motion understand- ing, we process each demonstration by decomposing it into frames and augmenting them with state information, including object 3D positions and gripper states.
Short-Horizon Descriptions
We process short-horizon descriptions in the following steps:
1. Sample frames at every 5-frame interval.
2. For each sampled frame, extract object positions, states, and the corresponding video frame, then prompt GPT-4o with Prompt 13 generate a state description.
3. For the transitions between sampled frames, input the consecutive state descriptions into GPT-4o with Prompt 15 and request a description of the motion occurring between them.
4. Compile all state and motion descriptions into an interleaved sequence of states and actions.
Long-Horizon Descriptions For long-horizon descriptions, we segment the demonstration into sequence of primitive actions and process them follows:
1. Generate a separate language description for each action's 's start and end state by providing GPT with the corresponding video frame and state information using Prompt [4]
2. Provide the start state description, end state description, primitive action type, and action parameters as input to GPT-4o with Prompt T6 requesting language-based description of the action.
3. Compile all state and motion descriptions into an interleaved sequence of states and actions.
Prompts used:
39

---

Published as a conference paper at ICLR 2026
Short Horizon State Description Prompt
Given image from robotic simulation, a description of the setup, and state information, write a caption describing the scene, ## Your response should be similar to the following examples: EXAMPLE 1: robot gripper is slightly above the sphere. The sphere is the table, next to the bin.
EXAMPLE 2: The robot is holding a blue cube above the orange cube. The orange cube is on stacked on top of the pink cubc. ## Input
Setup Description: (ENVIRONMENT SETUP DESCRIPTION) Here is the state information: (STATE)
Objects do not float. an object is elevated, is either held by the robot, stacked on top of ancther object, or inside a drawer or bin.
<ENCODED PRAMES
Long Horizon State Description Prompt
Given the image from robotic simulation, descriptien of the setup, and state information, write caption describing the Your response should be similar to the following examples: EXAMPLE 1: robot gripper hovers above a wooden table, helding green cube, while red cube and a purple ball rest on the table surface.
EXAMPLE 2: The robot is holding green ball above the table. There are two cubes stacked together on the table. The red cube is on top of the purple cubc.
## Input Setup Description: (ENVIRONMENT SETUP DESCRIPTION] Here is the state information. (STATE)
Objects do not float. If an object is elevated, it is either held by the rebot, stacked top of ancther object, inside drawer bin.
<ENCODED FRAMES
Short Horizon Action Deseription Prompt
Your job is to generate language description of robot's action based on
(1) START STATE: a dictionary containing the state and pesition of objects before the action,
(2) START STATE DESCRIPTION: the language description of the start
state,
(3) END STATE: a dictionary containing the state and positien of objects after the action,
(4) END STATE DESCRIPTION: the language description of the end state, and Environment description: (ENVIRONMENT SETUP DESCRIPTION! EXAMPLE 1: The robot released the blue cube on top of the green cube,
EXAMPLE 2: The robot picks up the blue cube.
Here are the inputs START STATE: (START STATE)
START STATE DESCRIPTION: (START STATE LANGUAGE DESCRIPTION) END STATE: (END STATE)
40

---

Published as a conference paper at ICLR 2026
END STATE DESCRIPTION: IEND STATE LANGUAGE DESCRIPTION: Now generate a description of the robot's 's action. Pocus on the coordinates. State description might be wrong,
Long Horizon Action Description Prompt
Your job is to generate a language description of a robot 's action based on
(1) START STATE: dictionary containing the state and position objects before the action,
(2) START STATE DESCRIPTION: the language description of the start state,
(3) END STATE: dictionary containing the state and positicn of
objects after the action,
(4) END STATE DESCRIPTION: the language description the end
state, and
(5) ACTION DESCRIPTION: the parameter of the action command to the
robot.
Environment description: (ENVIRONMENT SETUP DESCRIPTION!
Your response should be one or two sentences long and should be similar to the following examples:
EXAMPLE 1:
The robot successfully picks up the green cube. EXAMPLE 2: The robot trys to place down the red cube on the green cube. However, the red cube ended on the table.
Here are the inputs START STATE: (START STATE)
START STATE DESCRIPTION: (START STATE LANGUAGE DESCRIPTION) END STATE: (END STATE)
END STATE DESCRIPTION: [END STATE LANGUAGE DESCRIPTION) ACTION DESCRIPTION: (ACTION)
Now generate a description of the robot 's action.
C.3.2 GROUNDED PREFERENCE AND TASK GOAL GENERATION
With the generated language description of the demonstration, employ Prompt to generate grounded preference and language summary of the demonstration and Prompt 18 to update the task goals.
Prompts used:
Grounded Preference Prompt
You are helpful assistant. You are skilled at taking things people say off the cuff and without context, and specifying exactly what they mean in unambiguous, clear language.
showed a person video of a robot doing a task and asked them to give feedback to/about the robot. You will be given:
1. Description of the task
2. Frame-by-fram description of the video
3. The person's Seedback could be anbiguous, confusing, overly long, overly short, full of errors, etc.
41

---

Published as a conference paper at ICLR 2026
You need to provide:
1. summary of the video descriptive, complete, concise.
2. A rewrite of the feedback that is unambiguous, contextualizes
the person's feedback in the video/task description, uses standard language, directs the robot to de exactly what the persen meant to ask for.
Feedback grounding examples UNGROUNDED FEEDBACK 1: Not that onc! GROUNDED FEEDBACK should not it should move Red cube to the right.
robot moved Blue Cube to the right. It
EXPLANATION 1: The person says "nct that one" and there are two cubes in the scene, meaning they appear to have liked the action, but not the specific cube direct references to the Blue Cube and the Red Cupe. Since was the Blue Cube that was moved to the right and the person clearly wanted the other option, I said not to do that and to instead move the Red Cube to the right.
being done on. So, added
UNGROUNDED FEEDBACK 2: Hmmm, I like that the it move arm like that and put banana smoothly? but can it put banana in another one? GROUNDED FEEDBACK 2: Arm movement is good, but put Banana in Shelf 2, Shelf or Shelf
EXPLANATION 2: The person likes the arm movement. They want the banana to be put in another "one" since it was previously put on shelf, this means they want it put in a different shelf. Previously it was shelf 1, They also liked the way the robot put the Ranana now they're asking for it to be put in a different one, sc even though they liked it previously, I don't ask for that now.
now ask for Shelves 2, 3, or 4. Shelf but
go further GROUNDED FEEDBACK Move Cube further to the right. LXPLANATION 3: The person referenced a "block". There's nothing in the doscription called "block", but there is a "cube", which is a synonym. They want it moved further but don't specify where, which suggests they want it moved further in the original direction. That was to the right, so preserve that here.
UNGROUNDED FEEDBACK 3: the block
Task description (CURRENT TASK GOAL]
video description (LANGUAGE DESCRIPTION OF THE DEMO)
Feedback
[RAM HUMAN FEEDBACK)"
Your task
1. Think step step bricfly summarize the demonstraticn, then
go through the checklist, then give me your final versien of the feedback, grounded to the description.
2. Make sure your grounded feedback has a directive tone.
3. Use standard, unambiguous language over using the person's language.
4. Do not have ANY ambiguous objects. ALNAYS use the official term for an object is it has one.
CHECKLIST:
feedback may not be ordered, even if it contains multiple parts. Did you assume order inappropriately?
The person might compliment something about the past. Don't conflate that with what they're re asking for *now.. Did you make sure
42

---

Published as a conference paper at ICLR 2026
to isolate only the current instructiens, and remove things they like but don't want to keep?
The person will NEVER reference something not in the videc. If they use word you don't think is in the video, YOU ARE WRONG. Did you figure out what they were talking about? Did you replace any unexpected terms with official terms from the descripticn?
Output JSON format:
summary"; "<SUMMARY>
"explanae amplanatioo": "<EXPIANATION"",
"feedback"; "<GROUNDED FREDBACK>",
Task Goal Update Prompt
You are a helpful assistant that is excellent at interpreting what humans are asking for. You will be shown three phrases:
1. task description that was attempted by robot
2. description of the robot's attempted demonstration
3. Original feedback the person gave after watching the rebet make the attempt.
4. Grounded feedback that has been rewritten to be clear and unambiguous.
Your job:
If the person's Feedback changes the task itself such that the original task description longer applies, rewrite the task description.
If the person's feedback adds to the task, then add the new information to the task descripticn. However, if the person's feedback deesn't change the task and only comments the robot's exccution, then output empty string.
Give these in a json output.
Current task description:
(task_dess desscketccrppion
Demo description: (demo_ded descripticni Original Feeaback: prriginal_feccaace Grounded Feeaback: Igrounded Igrounded_feedbaack
Your output should be JSON object with the following format:
"task_deecriptiion: <TASK DESCRIPTIONS,
C.4 ENVIRONMENT SPECIFIC PROMPT
C.4.1 PUSHIBALL
PushBall Setup Description
There should be robot gripper and ball on the table. the 3D coordinate of the projects zx, x represents left and right positive is left, negative is right, represents forward and backward, positive is forward, negative is backward, and represents height.
represents the table surface. The radius
43

---

Published as a conference paper at ICLR 2026
of the sphere 0.06. Position is measured each object center. Expect errors in the measurement.
C.4.2 PLACESPHER::BIN
PlaceSphere2Bin Wide Wide Setup Description
There should be robot gripper, sphere, and two bins en the table. the 3D coordinate of the projects [x,y, represents forward and backward, positive is forward, y represents left and right positive and 2 represents height. 2 0 represents the table surface. The radius of the sphere is 0.02 and each bin is 0.16 wide and 0.01 deep. Fosition is measured at each object center. Expect errors the measurement.
left, negative is right, negative
backward,
C.4.3 STACK3CUBE
Stack3Cube Setup Description
There are a robot gripper, a rec cube, a green cube, and purple cube on table. The 3D coordinates of the projects 1x, are defined from the viewer's perspective: the x-axis represents forward and backward, with positive values being closer negative values farther away from the viewer; the y-axis denotes horizontal direction, with negative values to the left and positive to the right of the viewer. The z-axis measures height, where 2=0 corresponds to the table surface. Each cube is 0.04 by 0.04 by 0.04. Position measured at each object's center. Expect errers the measurement.
Stack3Cube Information Keys
"red__ : 3D coordinate of red_cube
"green_cube_pos" 30 coordinate of green_cube
"porple_cube_pos" coordinate of purple_cube se__re__cube_rraappe ris_green__om
whether red_cube is grasped by the robet "areee__uube_rrppp""_""_e" whether green_cute is grasped by the robot
purpleeeuuueee _aabb__gaaped' whether purple_cube is grasped by the robot
"is
gripper_pos"" 3D coordinate of the robot 's gripper
C.4.4 PUTORIRCTINDRAWER
PurObjectlnDrawer Setup Description
There should be a robot gripper, a red apple, a red soup can, and a drawer on the floor. Objects are roughly 0.05 in height and width. Object positions are measured at the center of the object. The 30 coordinates of the projects [x,y,z] perspective: the x-axis represents forward and backward, with positive values being closer and negative values farther away from the viewer; the y-axis denotes horizental direction, with negative values to the left and positive to the right of the viewer. The
defined from the viewer's
44

---

Published as a conference paper at ICLR 2026
axis measures height, where z=0 corresponds to the floor. The bottom drawer is facing the right (ty direction). it's open when the scene starts. The drawer is 0.36 wide, 0.22 deep, and 0.16 high. In other words, drawer, x [rawer_pos[0] 0.18 the right edge of the drawer, y drawer [ra + 0.11 is the drawer front, 2 drawer_pes(2] + 0.16 is the top of the drawer, and z arawer_poss[2 is the bottom of the drawer. The robot is placed en the floor and cannot reach the inside of the drawer. put object in the drawer it must be dropped from above.
drawer_pos[O]
0.18 is the left edge of the
PutObjectlnDrawwrr Information Keys
"apple_pap pos': 3D coordinate of apple "soup_cee can pos': 3D coordinate of copp_can apwll_grapsseeeeeeeppppppppppppa
whether apple is grasped by the robot "is_soup__aa__gaaped"" whether soup_can is grasped by the rebet "gripper_pos"; 3D coordinate of the robet's gripper "drawer_handle__p : 3D coordinate of the bottom drawer handle "drawer_pos"; 3D coordinate of the center bottom of the drawer "drawerr_open_offee how much the drawer is open, measured as the distance between the drawer's position now and when it is fully closed
C.4.5 OBJECTTOBIN
ObjectToBin Setup Description
There should be a robot gripper, an apple, an orange, a baseball, a tennis ball, and two bins on the table. One bin is light blue and is white. The 3D coordinates of the projects [x,y,2] are defined from the viewer's perspective: the x-axis represents forward and backward, with positive values being closer and negative values Farther away from the viewer; the y-axis denotes horizontal di rectiion, with negative values to the left and positive to the right the viewer. The z-axis measures height, where z-0 corresponds to the table surface. Each object is about 0,05 diameter. Each bins are 0.20 by 0.20 by 0.02 in size. Position is measured at each object 's center. Expect errors in the measurement.
ObjectToBin Information Keys
"apple_pos": 3D coordinate of apple
*orange_pos": 3D coordinate of orange
"baseball_pess pos": 3D coordinate baseball
"tennis "tennis_ball_pee ball__aall___________ll______eee pos'' 3D coordinate tennis ball
"blue bin pos": 3D coordinate of blue biue_bin
bin__os'::
"white_bin_po'', 3D coordinate of white_bin
"is_apple_grasped": whether apple is grasped by the robot iis_orange_graaped: whether orange is grasped by the robot fiss_baeeblll_ge "raebaall_rraaped:: whether baseball is grasped by the robet "is_tennia stennis_ballll grraslggrra_d''_ whether tennis_ball is grasped by the
robot
gripper_pos" 3D coordinate of the robet's gripper
45

---

Published as a conference paper at ICLR 2026
C.5 BASELINES PROMPTS
Eureka Prompt Template
We are going to use a Franka Panca robet to complete given tasks. The action space of the robot is a normalized "Box 1, (num_en env ,7), float32) where num_env means the number of environments in parallel, many attributes in the environment are batched, the first dimension num_env,
Environment Code: (enviromment_code)
Maniskill Doc: gewardd_
Previous empetttntti:
Previous reward function:
"'python
aprevious_rewardy
Your reward function code has been analyzed, the feedback is as follows:
Human Feedback: (human_feedback)
Objective Feedback: [eedbaci
Please carefully analyze policy feedback and provide new, improved reward function that can better solve the task. Scme helpful tips for analyzing the policy feedback:
(1) If the success rates are always near zero, then must rewrite the entire reward function
Please gencrate a reward function that follows all guidelines and addresses the human feedback. The code output should be formatted python code string: """pyyhon ... .....
"compute_dense_reerrr" containing the complete reward function
code
The functions MUST have these EXACT signatures:
def compute dense (seaf: BaseEnv, obs: Any, action: torch. Tensor, info: Dict[str, torch.Tensor]) -> torch. Tensor VEncodes reward for each possible action based on "evaluate" output dictionary and other current environment info (envircnment instance attributes).
#Incorporates human feedback as given.
#Obeys the following structure:
#1. Stage reward
Task is split into stages and reward given to the agent gradually, encouraging it to complete each stage. The reward accumulates.
Interdependencies and tradecifs between different task stages and different aspects feedback and overall goal are considered. Reward design is not overall counterproductive to meet short-term goals.
#2. Return reward value
46

---

Published as a conference paper at ICLR 2026
Your implementation here pass
The function should:
1. Match the exact function signatures shown above
2. Bandle batched operations correctly (num_ env dimensicn)
3. Include comprehensive comments
4. Follow the reward funstion best practices
5. Be consistent with the interface of the previous implomentations while incorporating the requested changes
The code output should be formatted python code string: python ....
Text2Reward Prompt Template
We are going to use a Franka Panda robot to complete given tasks. The action space of the robot is normalized "Box (-1, 1, (num_en) env ,7), float32)', where num means the number of environments in parallel, many attributes dimension is num_env,
envirenment are batched, the first
Sask-Specific Environment:
(envi canvirroment__dscrrppii..
Available Objects and Their Properties: (env saa_cclass_deee descl
Maniskill (reward_guidellness)
Previous Implementations:
Previous reward function:
""pythonn [previous_reward]
Human Feedback: saamar_feedback)
Please generate reward function that follows all guidelines and addresses the human feedback. The code output should be formatted as python code string: """"pyhoon ....
"compute_dense_rowarr containing the complete reward function
code
The functions MUST have these EXACT signatures:
def compute compute_dense__ee aecessereward(seee BaseEnv, obs: Any, action: terch. Tensor, info: Diet Istr, torch. -> torch. Tensor #Kncodes reward for each possible action based on "evaluate" output dictionss and other current instance attributes).
info (envi renment
IIcorporatess human feedback as given.
#0beys the following strcture: #1. Stage reward
47

---

Published as a conference paper at ICLR 2026
Task is spiit into stages and reward given to the agent gradually, encouraging it to complete each stage. The reward accumulates.
Interdeppeneencess and tradecffs between different task stages and different aspects of feedback and overall goal are considered. Reward design not overall counterproductive to meet short-term goals.
#2. Return reward value
Your implementation here pass
The function should:
1.
Match the exact function signatures shown above
2. Handle batched operations correctly (num_env dimension)
3. Include comprehensive comments
4. Follow the reward function best practices
5. consistent with the interface of the previous implementations
while incorporating the requested changes
The code output should be formatted as python code string: pythor
Text2Reward Environment Class Discription
#class FancaRobot:
self. www.pcccp.ppppppooompmooo pr torch. Tensor (Inum_eee env, 3]) # Batched by the number of environment, indicate the 3D position of the robet's gripper
pose. Tensor ( ([num_ent) 4]) I Batched by the number of environment, indicate the quaternion of the robot's 's gripper
self.robet. qpos: torch. Tensor ([num_en) env, 9]) Batched by the number of environment, indicate the joint positions (last 2 for gripper) of the robot at this key frame rob.t....ttrg...
torch. Tensor ( ([num_env, 9]) f Batched by the number of environment, indicate the joint velocities (last 2 for gripper) of the robot this key frame
def (se1f,, object: Actor, min_force= max_angle -5) -> torch. Tensor [num_env, I, bool) # Hatched by the number of environment, check if the robot is grasping object
class RigiaObject:
self.pose, pose.p: torch, Tensor ([num_env, 31) I Batched by the number of environment, indicate the 3D position of the simple rigid object in each environment
self.pose.q: torch. Tensor env, 41) Batched by the number of envment indicate the quaternion the simple rigid object in each environment.
def wwlaannulrr_vlleiiee (self,) -> Tensor ([num_env, 31) # Batched by the number of environment, indicate the angular veloeity of the simple rigid object
def get_linear_vellcine Hatched by the number of environment indicate the linear velocity of the simple rigid object
(s) -> torch, Tensor ([num_env, 31)
self.get wtt.etiirrrssssiiieeeeeeeeeessee collision mesh() .bounding box. bounds: np. ndarray (2, 3)1 & non-batched, indicate the bounding box of the simple rigid object
class TargetObject:
48

---

Published as a conference paper at ICLR 2026
self. pas..ppsepp.. torch. Tensor (Inum_eay env, 311 Batched by the number of envi ronment, indicate the 30 position of the Vargetobject in each ennment
self. self.pose.ce q: corch.Tenson ([num_env, 41) I Batched by the number of environment, indicate the quaternien the Targetobject each environment
class LinkObject:
self. posc.po torch. Tensor ([num_cnv, 31) # Batched by the number of environment, indicate the pesition the link object each environment
self. self.pose.q: torch.Tensor ([num_env, 4]) # Batched by the number of envi ronment, indicate the quaternion of the link object in each ronment
def get vet__anularr wwwaaaaagaeeeeieaaeaaa vel.city (self,) -> torch. Tensor (Inum_es env, 311 Batched by the number of environment, indicate the angular velocity of the 1ink object
def get_linear_veloite (self,) >> tersor ([num_env, 31) Batched by the number environment, indicate the linear velocity of the link object
self.qpos torch. Tensor [[num_ent) couumeev,,I] I, float) # Batched by the number of environment, indicate the position of the link object joint
self.qvel torch. Tensor ([num_env, float) Batched by the number of environment, indicate the velocity of the 1ink object joint
class Articulate@bjentt
self. self.pose.pp torch, Tensor ( ([num_env, 31) # Batched by the number of environment, indicate the 3D position of the articulate object in each environment
self. silf.pose.q: torch. Tensor ([num env, # Batched by the number of environment, indicate the quaternion the articulate object in each environment
self. spos torch. Tensor ([num_env, 9]) # Batched by the number of envmentmen, indicate the joint positions of the articulate object this key frame
self.qvel torch. Tensor ([num_eng _nv, 9]) # Batched by the number of environment, indicate the joint velocities of the articulate object at this key frame self.get_fte ttsst_cstscclssoon mesh .bounding box.pounds np adarrary [(2, 3)1 & non-batched, indicate the bounding box of the articulate object
Documentation For Eureka and Text2Reward
Documentation of classes, methods, and properties you
when implementing Sampute_dense_revie
This is parallel training environment with many episedes. Each pose, boolean description of the scene, etc. is therefore "torch. Tensor" where the #irst dimension is the +batch dimensiont. Each episode is independent, so you must calculate reward separately for each one. specifically
"reward" is Corrhh.ensoor of scalar reward values for the batch of episodes. The first dimension is batch of episcdes, and the second dimension is the scalar reward value for that individual episode.
For example, if you want to give reward boost of 5.0 to exactly the episodes where the object is grasped, and you have
49

---

Published as a conference paper at ICLR 2026
already computed the _is_obj_graspes tenser, you can say "reward is_obi_grasped") +- 5.0".
## "Actor" class
Non-robot objects scene are subclasses "Actor".
### Properties
"static", "dynamic")" indicating physics behavior - static (inmovable), dynamic (physics- affected), kinematic (not physics-affected)
ppx_body__tt bode_type": "kiteral["kinmmatii
The robot cannot move "static" and "kinematic" objects. feedback asks you to change their locations, reject that part. "name" "str' acting as the unique identi for the actor "merged": "bool" indicating if the actor is composite of multiple other actors
"pose": "Pose" object representing the current state
"Pose" object: dataclass
"pose.p" is position
"pose.q" is quaternion indicating orientation "pose.
is concatenation "p" then "q" "Fose.create_frromp911177
q=q)" returns a new 'Pose"
object at that "p" and "q"
### Instance methods (called with "<actor pecttoriinstanee.. c <method_ name>> params>)")
"gge_sstte";
feelorractr_iiie wnw.acttoossttccco..eooo..
() [;, 7:10]": contains linear
velocity
haaffffaccttriisttntnte betttaacccegett_ttttteee [:, 10:13]* contains angular
velocity
"is_static (iinatnnnnnttintnete ie ln boolean check i
actor is static based on velocity thresholds
Liinshhrehh's linear velocity thresheld -ang thresh': angular velooity threshold
H Agent class "Basoagent' class for the robot. The robot is a 7-DoF arm-and-two finger gripper. Degrees of freedom: arm position, arm orientation, and gripper open/close.
Note that you will see "tep' throughout the code, with its specific position being queriea. "tep' stands for Tool Center Point, the center between the two fingers. "tcp.pose" attributes tells where the gripper is positionea. ### Properties
"self.agent. controller"; :self.agett.conrrolleer.. currently activated controller "self.agent wtt.faaget..ccion__ee..... action_space"; controller has been sent to in the latest action, concatenated for the two controllers (arm and end-effector) ### Methods
that
returns dicticnary with the follewing
keys:
"robot_root_poa="; root pose
"robet_root__ee1"; root velocity
"robot_root_qvel1, root angular velocity
"robot_qpos": joint position
"robot_qvel"; joint velocity
"controller"; output of "controller.ce get_statee]) which contains
target positions and orientations of the currently activated controller
"self.agent. irrgent.i__graspingggoo (object:: Union[Actor, None] None)';
boolean check 1f agent is grasping object
(threshllld: float)": boolean check robot is static within given threshold) in terms q velocity
"self.agent. sabbot.ge__llmmits () [0, -1, 11 2": gripper width, i.e max possible opening
50

---

Published as a conference paper at ICLR 2026
# Other
"tep" stands for "tool center point" it is the point between the robot's grippers and is TODO whole robot
the values of "info', quantities are tensors over the batch. For example, "pose" a tensor where the first dimension is the batch of multiple episodes, and the rest describe the pose of that episode.
Like "reward"
IMFORTANT: here, is the front-and-back axis, is the left-and- right axis, and 2 flipped check your work!
up-and-down axis. Typically, x and y are
LMPC Prompt Template
## High-Level Description:
You are an expert robot reward function programmer. Your goal to write improved reward functions for a Franka Panda robot arm with gripper to fulfill tasks based on feedback and previous Implementations.
You will analyze feedback and create reward functions that better guide the robot to accomplish its tasks.
## Robot Environment Details:
We are using Franka Panda robot to complete given tasks. The action space of the robot Eloat32)", where "num_env" means the number of environments in parailel. Many attributes in the environment are patched, with the first dimension being "num_evv".
normalized (=xx 1, (num_env, 7),
The robot a 7-DoF arm with two-finger gripper. Degrees freedom include arm position, arm orientation, and gripper open/ close. The Tool Center Point (tcp) is the center between the two fingers.
## Coordinate System:
The coordinate system is right- and three-dimensicnal:
x-axis: front-and-back
y-axis: left-and-right
z-axis: up-and-down (aligned with gravity)
Note that x and y are typically flipped in standard notation check your work carefully!
## Previous Implementation and Feedback: For each task, you will be provided with:
1.
A previous reward function implementation ("compute_dense_revaa
)
2. Fuman feedback the previous implementation ('human_feedback
## Your Task:
You must carefully analyze the feedback and provide new, improved reward function that better solves the task. Your functicn should:
1. Match the exact function signature:
def seaerreddeneeeerrrdeeee BaseEnv, abs: Any, action: terch. Tensor, info: Dictlstr, torch.Tenscr() torch. Tensor torch.Tensor
# Available APIS:
51

---

Published as a conference paper at ICLR 2026
### Actor Class (Non-robot objects in scene)
#### Properties.
p>__body_tppe' Physics behavior type ("kinematic", "static", dynamic*)
"name": Unique identifier
"merged": Boolean indicating the actor is composite
"pose"
Position and orientation
"pose.p": Position
"pose.q's Quaternion orientation
rawspos'._posc'. Concatenation p and q
#### Methods:
"get_state() [:, 7:101"; Linear velocity
"get_state [:, 10:13]*: Angular velocity
-ss sis_staaicelt thre ang threshheeeee"""" Boolean check if
actor is static
UP Agent Class (Robot)
INV Properties:
"self.agent. controler.ctttoler'. Currently activated controller "sse.faaeet.acttion cecciionnspcce"" Position//rrennaaiion/eet of controllers
#### Methoos:
agent get_sttteeettttee_e Returns robot state dictionary (object)": Boolean check if agent is
grasping object
Hoolean check robot is static "self.agent robot geeet.geegggiiiie [0, -1, 1] 2": Gripper width
## Important Notes:
"tep" is the Tool Center Point (center between robot grippers)
quantities are tensors over the batch dimension
The robot cannot move "static" and "kinematic" objects
## Output Format:
Your output should be formatted Python code string: """"python
The code should contain the complete "compute_dense_eee function
Robot Code Writing Hints:
Do not use any functions or object names besides the cnes mentioned above.
Chat Turn Example:
## Environment code:
python
class CubcândTargetEnv (BascEnv):
***
**aak Description:
simple task where the objective is to pull 1 cube ente
target.
*#nndomizationss:
the cube's ' xy position is randomized on top of table in the region [0.1, 0.1] [-0.1, -0.11.
the target goal region marked by The position of the target is fixed to be the cube's s xy pesition [0.1 + goal_radius, 0]
white target.
52

---

Published as a conference paper at ICLR 2026
** Conditions:**
the cube 's xy position is within goal_radius (default 0. 0.1) of the target's s position by euclidean distance.
sample video link huttpp///gthhub..co//assalllla/
main/ hatnr//iiuureeeeevronnnettttttt www.oonnnnddddo//uuttee wtt.cs//uulluub-rrlllrr...
SEFFORTED_ROBOTI = ("panda", "fetch"]
agent: Union (Panda, Fetch]
goal_radius 0.1
oube_half size 0.02
def ii____nit_______ *args, vobet_uids="ppanee robot_init_cpos_na_ se 02, **kwargs):
self. init spos noise rebot init apopt_psit__ooss_.o_oooo spos _nise intt__poitttttttt____eo_pp super init (*args, robot wids=robot uids, **kwargs)
eproperty
def _edeffllt_eeeeeeeeee _eefauut_eessrrconfigggg (self):
pose look_at (eye= 0, 0,6], target=1-0.1, 0, 0.1])
return [CameraConfig ("base_camera", pose, 128, 128, np.pi /
2, 0.01, 100)
@property
def Betrault_huaa_eeee
(self))
pose = 1ook_at(10.6 6, 0.7, 0.6), 10.0, 0.0, 0.35]) return CameraConfig ("rendder ("render_camere pose, 512, 512, 1,
0.01, 100)
def _ooad agent (self, options: dict): super () Load agent (options, sapien. Pose (p=1-0.615, 0, 011)
def _load_scened (self, options: dict):
wolf.table_scoet = TableSceneBuilder, env=self, robot
robot robot_inite joot___itt_cpos_niise
salf.table_scene.bitt
create cube
self.obj actors cettorsbbill__cuk
self.acene, half_siza--slf.aub__hhaa
color-np.array ([12, 42, 160, 255]) / 255,
name="cube",
body body_=sppe="dyammie"
initial pow.ssppppsssspien....m (0=10, 0, self. caff.cube_haale size])
create target self. self.goal_reginn
self.scene, radius-sell.goallrree
thickness=le-5, "aan=="goal_region"
add cad.colliiioonnallee.
body_type=""inematicc"
def Lnitiaaliz__epssdd(Ses env torch.
torch.Tensor. Tensor, options:
dict):
with corch.devise (sel..
53

---

Published as a conference paper at ICLR 2026
b 1en (env_iio idx)
sel f.
ine (env_idx)
xy² = torch. zeros (b, 3))
xyz :2] torch. (b, 2)) 0.2 0.1
self www.sssw........ee.e.. aaf.cube_half_sizz
q [1, 0, 0, 0]
obi_pose = Pose. www.eaaeeeroo_epppppeeeeee q=q) self.obj.set_ppeeee (obj_ target_region_XX xyz - torch.tensor ((0.1 self.
goal_radi us, 0, 0])
2] le-3
sel2. www.erregnnnnnnoeeeeeeee (
Pose. crw.te cesseccreee__rommp.oo Bere__rro________________
xyz, greuler2quat (0, np.pi 2, 0),
self. selffobject_liee = ("cube": self.obj,
"geal": self. self.goal_regiine
def evaluate (self):
ie_obi_placed
torch.Linalg.nome
self.obj. wos...b.ppo............ :2] self. gelf.goal_regien. pose.p
I... :21, axis-1
sel f self.goal_radiin
return "success"; plaob__
def _get_obs_exttee _xtraeetttaaaeeeeeeeeeeeeee (self, info: Dict):
obs dict (
goal www._rregc.n..........
if self. mode in ""state", "ict": obs. update) obi_pose=self.oo objj. .ow.eose...pppppp.ooooooooo..e.. raw_pose,
return obs
def Sempute_dense_rewareee
obs: Any, action: Array, info:
Dict):
grippers should close and pull from behind the cube, not
grip it
distance to backside of cube (+ 2+0.005) sufficiently encourages this
tcp_pull_pos -
+ torch.tensor tensor 2 0.005, 0, 0], device=se...
device
to top_to_pull_poee = tcp pull pas - self aglt..ggnt...p.pocee.. tep_to_pull_pose__leee torch. linalg. norm tomtttcp_tt_pulllpoe
top_to_
axis-1)
reaching_reward torch. (5 teq_to_pull_pse_ddiss
reward reaching reaching_rewars reward
reached tep_to_puile pose dist < 0.01
54

---

Published as a conference paper at ICLR 2026
ob goal dist torch. pos......pp.......... :2] self.goa lar.lggal_ere___aeee on pos..............
torch.Lina.g. www.ig..nnnn.......
:2], axis-1
place_reward torch. tanh obj_ro_goal_distt)
reward place_reward reached
("Nacc(in = return reward
def compute_nooe Array, info: Dict):
obs: Any, action:
max_reward 3.0
sel compurre__enseerewwrrrdd (obs-obs action-action, info=info) max reward
def get fitness fitness_score (self) score get the fitness score of the current episode
returns tensore of shape (batch_size,
currently, the fitness score is the distance between the cube and the goal
+ fitness score always the higher the better return the negative distance
return -torch. Annalg.norm(
self.obj wet...obp.pos.... wwwpppp.................p....... :2] self, geal.goll_ree p
:2), axis=1
## Previous Implementations:
Previous reward Function:
"python
def peaputte_dense_revrlleeee
obs, action, info)
reward www.h..erooss.i.....oo (info (into("task_complettd"] dtype-torch.
float32)
# Simple reward for moving toward object
target_pos self.obbectt.ose.
gripper_pos self acett.t
(wwipppgegrrrrpepp pos target_pos, dim=-1) reward - distance 0.1
distance =
return reward
Your reward function code has been analyzed, the feedback is as follows:
## Euman Feedback:
The robot moves toward the object but doesn's 't attempt to grasp it.
# Output:
"python
def deaetre__deneeeewwrrddee Tensor, info: Dict [str, Tensor]) -> torch.Tensor:
BaseEnv, abs: Any, action: terch.
Initialize reward tensor for batch processing
reward torch, like (into) "task "task__completed"] atype=torch. float32)
target_pos
gripper_pos self agant.
distance torch. (gripper_pos parget_pos, dim=-1) reward distance 0.2 Increased weight for faster approach
55

---

Published as a conference paper at ICLR 2026
is graarign self.agente grs_ggggiinnggeeeee (self. opjject
rexard[is_graspingg +- 1.0 Reward for successful grasp
if "grasped_duration" in info: reward info ("grasped_durattoon"" 0.05 for maintaining grasp
Small reward
return reward
New Chat Session
Knvi ronment code:
pyython sunviromment_code
Previous Implementations:
Previous reward function:
""yython Betpaatte_enseeeeeeeee
Your reward function code has been analyzed, the feedback is as follows: ## Human Feedback:
(human_feedback)
Your output should be formatted as Python code string: "pythhn
The code should contain the complete ccompute_denae dense reward" function
# Output:
C.6 ERROR CORRECTION PROMPTS
Short Horizon Error Correction Prompt
You are being given code that is training environment for reinforcement learning-based robot. Code details: cestiaaizeeepissoee
1. "_load__ceen scene", by them, set up the scene.
and all the methods called
2. "evaluate" calculates information about the current envirenment state in dicticnary, Its output dicticnary is given to compute_dense deaeeddeeeeeedddd""eeeeeeeeee reward" in the parameter "infe".
3. Sempute_dense__evee
outputs a final reward.
The code is as follows:
python [generated_env_code)
This code has error in it that you need to fix. The stack trace is follows: "'python
lerror_trace,
56

---

Published as a conference paper at ICLR 2026
The line that threw the error (error_]inge
Please fix it.
1. You cannot take debugging steps, you have fix it without more info.
2. You should only edit "evaluate" and "compute denddeeneeeeeeeeeeeeeeeee reward' and any of their helper methods. Enclose then in markdown "" python < your code>
delimiters.
"pyhhn ... delimiters anywhere clse in your response, You can use <eode> - if needed.
3, Don't
3. If you would absolutely have to edit other methods to make the code work, you should instead say "this requires edits to restricted parts of the code. can't continue."
4. Follow the documentation below. Do not violate
5. OUTFUT THE WHOLE REVISED FUNCTION, NOT JUST A SNIPPET.
|documentation]
Long Horizon Error Correction Prompt
You are being given code that is training environment for reinforcement learning-based robot. Code details: 1. "_load_scene', "_initiallize_epssodd and all the methods called by them, set up the scene.
2. "evaluate" calculates information about the current environment state in a dictionary. Its output dictionary is given to skill_reward in the parameters "prev_inio" info" and cur_info" each represents "evaluate" called on different ronmmnt state. 3. "prev_info" and 'cur_info" go through a conversion that turns their ttrrhh.Tensor values with batch dimensions into "numpy. ndarray' values WITHOUT batch dimension.
3. "skill reward" outputs final reward.
The code is as follows: "'python (generateed_nn_ccdde)
This code has an error in it that you need to fix. The stack trace is as follows: "'python lerror_trace,
The line that threw the error is (error_line).
Please fix it.
1. You cannot take debugging steps, you have to fix it without more info.
2. You should only edit "evaluate" and rsward and any of their helper methods.
3. If you would absolutely have to edit other methods to make the code work, you should instead say "this requires edits to restricted parta of the code. can't continue."
4. OUTPUT THE WHOLE REVISED FUNCTION, NOT JUST A SNIPPET.
57

---

Published as a conference paper at ICLR 2026
D HUMAN SURVEYS
D.1 HUMAN ANNOTATORS
We recruited total of five preference/evaluation annotators through Upwork Upwork (n.d.). Each chain of preference was generated by one annotator: they would watch an initial video and give preference, evaluate the result and issue new preference, and so on until chain termination. Videos were 2-6 seconds in length, evaluation and preference forms took 1-3 minutes to fill out. Annotators were asked to respond within 6 business hours and given maximum of 24 hours, so one four-step chain would take 2-4 days. Annotators were compensated with $20 each time they provided eight new pieces of data, whether evaluations or preferences.
Expert evaluators were recruited for voluntary effort.
D.2
SURVEY CONTENT
D.2.1 PREFERENCE/ALIGNMENT EVALUATION SURVEYS
We conduct three types of surveys:
1. Robot Video Preference, as shown in Survey [ This survey collects human preference on robot trajectory videos.
2. Video Quality Evaluation, as shown in Survey 2 This survey evaluates how well newly trained robot behaviors align with preference from previous videos.
3. Preference Evaluation, shown in Survey 3 This survey collects human preferences among videos generated by different methods.
Robot Video Preference
Watch robot task at the video sent you, then give feedback. Indicates required question
Required Questions:
1. What 's your name? Make sure to enter your name the same way every single time.
2. Which video are you giving feedback for? Copy-paste the .p4 mp4" put "videc_name"
file. if the file is named here.
3. Describe the video
4. Give feedback. Again, be natural! Speak the way you'd 'd want to speak to your personal robot assistant. Even if the task isn't inherently interesting, feel free to be creative ask to change the order, the location, and more.
Task Overview:
You' re going to watch robot do a task. Your job is to give the robot feedback.
Purpose:
The robot 's execution might have problems
lven if technically fine, it might not meet your preferences
want feedback what you do and don't like
Goal is make the robot adapt you, not vice
CRUCIAL GUIDELINES:
1. Procedure:
Watch the entire video
58

---

Published as a conference paper at ICLR 2026
Describe the video in the first answer box Enter feedback in the second answer box
2. Focus Outcomes:
Don 't guess how robot works Don't suggest technical mechanisms
Focus on preferences and corrections Describe what you want/don't want
3. Feodback Limits:
Maximum one feedback on overall behaviors
Maximum one Seedback on object orientation
No feedback speed (technical limitation)
4. Communication Style:
Speak naturally
Use everyday language
Be specific
Context-dépendent statements are acceptable
EXAMPLE SCENARIO:
Robot sweeping crumbs task
High Quality Feedback Examples:
"The robot didn't finish sweeping, There are crumbs all over the place. The crumbs near the couch and TV should have gotten swept up
like that the robot brought 'hat's a good idea."
crumbs close the trash can.
"Ew, hate that the robot just left the crumbs near the trash. That whole corner of the kitchen is so gross."
"I like that the crumbs in the kitchen, out now it should put the broom away."
Low Quality Feedback Examples:
"The robot should work getting its arm to reach further so it can sweep up crumbs." " "This is dumb. It should be calculating more precise trajectories
Challenging but Acceptable:
"Not that one, the other one." (If context makes neaning clear)
Note: Your feedback can be longer than examples and include multiple ideas. Be thorough and specific.
This content is neither created nor endorsed by Google. Forms
Video Quality Evaluation
Earlier, you watched video of rebot doing task and gave feedback. tried to incorporate your feedback, and sent you one folder.
or more o in your
Instruction ions:
1. Choose the video you think is best from that folder.
2. For that video only, fill out this form that asks about your satisfaction with the update.
Indicates required question
59

---

The robot did something you Published as a conference paper at ICLR 2026
Basic information:
1. Copy-paste the name of the video you liked best here. Remove " mp4", you do in the other form.
Feedback Implementation Assessment:
2. In the most recent video, did the robot incorporate your feedback that **wasn't met in the previous video**? oval.*
*Mark only
the robot didn't incorporate any part my feedback o Some the robot incorporated some parts of my feedback, but not all of them.
Yes the robot incorporated all parts of my feedback.
3. In the most recent video, what degree did the robot incorporate your feedback that **wasn't met in the previcus video *+?
*Mark only one oval.*
None the robot didn't incorporate my feedback
o Medium - the robot incorporated my feedback to moderate degree but not fully.
High the robot incorporated my feedback fully.
most recent video, the rebot get worse with respect to any of your feedback **oomarred to the previous video**? +Mark only one oval.*
4.
No it did not get worse with respect any part of my feedback.
Some it got worse with respect to some parts of my feedback, not others.
Yes - it got worse with respect to all parts of my feedback.
5. most recent video, did robot change its perfermance in ways you didn't ask for *ompaare to the previous videc**? Specifically, change even if not directly/purely negating your feedback?
For example, if you asked the robot be neater and smocther but didn't say anything about speed, and the robot became noticeably faster in this video compared to the last one, you would say yes. Else, no.
*Mark only oval.* *skip to question
o No *skip to question 11x
Progress Assessment:
6. How much * you feel the robot made the goal you wanted?
If it got hal toward the goal your feedback was asking for, that would be 5. *Mark only 6 7 oval.*
Did the goal fully o 0000000 didn't ask for
Unasked Changes:
7. Which of these statements best describes your feelings about the change (s) you didn't ask for?
*Mark only one oval.*
I strongly dislike them.
somewhat dislike them.
I feel neutrally about them.
I somewhat like them.
I strongly like them.
60

---

Published as a conference paper at ICLR 2026
8, Was the robot doing these same things you didn't ask for the previous video? *ark only one oval.
9. In the most recent video, does the robot's performance directly contradict your feedback? *arr only one oval.* o Yes No
10. Describe the changes you that you felt you didn't ask for, but weren't cirectly/purely against your feedback.
Satisfaction Assessment:
11. Are you more satisfied with the robot in the most recent video than you were
previous viceo?
*Mark only one oval.* Yes +Skip to question 12.
No *Skip to question 13*
Satisfaction Level:
12. How much more satisfied are you?
+Mark only one oval.*
1 2 2 3 4 5
Not at all 0o00 Completely satisfied
13. How much more dissatisfied are you? *
*Mark only one oval.*
2
Not at all 00000 Completely dissatisfied
Overall satisfaction:
14. How satisfied are you with the task overall, regardless of your feedback? *Mark only one oval.*
2 3 4
0 Very satisfied
This content is neither created endorsed by Google. Forms
Preference Evaluation
1. Folder Name
Please copy and paste the name of the folder. (e.g., Sabeett=laeeshtrreBBiiidddeed
2. Please rank the videos from 1 to N based your preference, with 1 being the best.
For example, it there are six viceos, rank of 3,4,2, indicates you lave 4.mp4 the best.
2
5
6
7
8
9
video1
Video2
Video3
0
0
0
Video4
Video5
Video6
61

---

Published as a conference paper at ICLR 2026
Video7
0
0
video8
0
0
0
Video9
*Mrr only one oval per
D.2.2 SEMANTIC MATCH EVALUATION SURVEY CONTENT
Expert evaluation
Thank you for doing this expert evaluation for us! Our project aims to use LLMS to take natural language feedback and generate dense reward functions. our goal with this evaluation to measure semantic match between feedback and reward function does the generation make sense given the feedback?
Context:
We train Franka arms in several ManiSkill environments see video.
report.me .nd has four separate sections: "Original feedback" what the annotater said. "Grounded fecdback" attempts to ground it in environment prior performance. "Staging plan" similar to task plan but with scme differences. "Code" comptte_dense_reeae
generated version of the feedback that
generated plan for writing a staged reward,
generated functions, "evaluate" and the actual reward function, "evaluate" is a helper function that packages scene info and defines success. You can mainly read "compute dense "annse_eeward" and just refer back to "evaluate" definition).
needed (e.g. for success
we will ask you to evaluate these purely on whether they make sense given the feedback common sense, logic, semantics. You do not need to worry about trying to predict whether the reward will actually work not.
Start by watching the video to get some context!
NOTE: the bin task, the bins cannet be moved.
push ball
task, the target cannot be moved. The robot s starting position cannot be changed in either case.
If feedback asks for that, it's ckay if it's in the stage plan, but it should not be in the code.
To what degree does the grounded feedback correctly interpret the original feedback?
[Likert scale 1-7]
Note: please evaluate the staging as relates to the grounded feedback, not the original feedback. Re want to separate evaluation of each step.
To what degree does the staged reward plan like accurate and precise attempt at incentivizing the desired pehavior? [likert scale 1-7]
Are there missing steps? Yes/No
Are there unnecessary steps?
Yes/No
62

---

Published as a conference paper at ICLR 2026
Is the strategy bad overall? Yes/No
If you'd like, describe the problems in sentence twc.
Note: please evaluate the coding as though the staging were reasonable, whether or not it actually is again, trying to evaluate each part separately.
To what degree does the reward code seem like accurate and complete attempt at incentivizing the desired behavior? [Likert scale 1-7]
Does the reward function miss part or all of the stage plan? Yes/No
Is the reward function so sparse that it's not even semantically aligned? Note that somewhat sparse rewards can be semantically aligned, if they won't be effective for training. This is asking for reward sparse that it's not, or only trivially, semantically aligned. Yes/No
At semantic level, there logic errors? Contradicticns, cireular dependeneies, etc. Yes/No
Are there common sense errors? Yes/No
If you'd 'd like, describe the problems in a sentence or two.
Even if all the generations technically seem correct given the feedback, there's s obviously nuance in language that may missed. Would you have interpreted and approached this similarly? Yes/No
If you answered anything less than 7 for any the overall quality questions, to what degree do you think this was due to difficult human feedback? [Likert scale 1-7]
D.3 TAXONOMY
Taxonomy definitions
3rd-person vs. 2nd-person vs. no-pov: how the preference addresses the embodied agent
verbose: concepts explained in lots of words, more than may be necessary
colloquial: concepts explained colloquially, casually, with imprecise (but not ambiguous) terms
context-dependent:
requires info from the demonstration or previous preferences to be understood; environment code is not sufficient
ambiguous: even context is not enough, and the preference is up to interpretation (rare)
multi-part: multiple concepts within one preference
directive vs. suggestive vs. curious: preference tone. Directive is command-like. suggestive is phrased question but clearly instructional, and curious is genuine question about the agent's 's abilities or potential
distraction: contains distracting language that may be incorrectly incorporated into the reward function
63

---

Published as a conference paper at ICLR 2026
contradiction: preference that directly contradicts prior preference
preferential: adds to changes task requirements
corrective: doesn't add to or change the task requirements, instead asks for improvement on current requirements
preferential: adds to or changes the task requirements
goal-related: affects the definition of success
behavioral: affects other parts of the reward landscape
inadvertently long-horizon: creates an impossible masking problem in the short-horizon continuous control setting
affirming: positive about some aspect the task that should be propagated
asking for change: asking for some aspect to be changed
physically difficult: difficult for the test agents, e.g. throwing ball
physically impossible: impossible for the test agents, e.g. picking up a ball that is much wider than the gripper.
multi-objective:
preference containing multiple complex goals that need to be achieved simultaneously. Restricted (and rare)
specific position: complex or specific natural language description of position
specific orientation: complex or specific natural language description an orientation
illegal: requires edits to code outside of
reward and eva uate.
E
DETAILED METRICS
Alignment
Percent more satisfied (MS): given binary question asking whether they more satisfied with the current video than with the previous video given their preference, for what percentage of preferences did the annotator say more satisfied?
Satisfaction score (SS): annotators were given a five-point Likert scale to indicate their degree of (dis)satisfaction, based their answer the binary satisfied-yes-no question. In both cases, was the smallest amount of (dis)satisfaction) and 5 was the highest amount. Satisfaction score is calculated by shifting both from 1-5 0-4 means neutrality, negating the Likert score value in dissatisfaction cases, and rescaling from 10.
Percent chosen (PC): when directly comparing policy rollouts single preference, frequency the given method's 's rollout being selected over ROSETTA'S
Optimizability
Success rate (SR): success rate of best policy across all variants for given preference Percent success >50: percent preferences where best policy has success rate higher than 50%
Semantic match
Grounding: normalized 7-point Likert scale score on semantic match between original and grounded preference
Staging: normalized 7-point Likert scale score on semantic match between grounded preference and staging plan
-Coding: normalized 7-point Likert scale score on semantic match between staging plan and reward code
-
Cascading: product Grounding, Staging, and Coding. Generally evaluated only for preferences expert evaluators consider reasonable, given quickly compounding errors.
64

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Learning rate
3e-4
Discount factor (?)
0.8
Hyperparameter
Value
GAE parameter (A)
0.9
Learning rate
3e-3
Number of minibatches
32
Discount factor (?)
0.2
PPO epochs
8
Update Coefficient (7)
0.5
Value function coefficient
0.5
of Gradient Step
Entropy coefficient
0.0
Batch Size
1024
Gradient norm clipping
0.5
Policy Model Size
512*2
Target KI. divergence
0.1
Actor Size
256*2
Table 4: SAC Hyperparameters
Critic Size
256*2
Table 3: PPO Hyperparameters
Environment Name
Total Timesteps
Episode Length
Description
Pick I Cube
10M
50
Cube picking
Publ
10M
50
Cube pulling
PushBall
20M
50
Ball pushing
PlaceSphere2 Wide
100M
200
Sphere placement in bins
Table 5: Short-horizon Environment Setups
REINFORCEMENT LEARNING DETAILS
F.1 SHORT-HORIZON SETUP
We implement our approach using PPO for training agents four short-horizon environments. The implementation is based on the ManiSkill environment, which provides a realistic physics-based simulation platform for robotic manipulation. The policy is trained using parallel environments (num_envs = 1024) to improve sampling efficiency and training stability. We use state-based observation and 7-dimensional pd_joint_des delta action.
The training hyperparameters are given in Table given T
The training steps of different environments are
F.2 LONG-HORIZON TRAINING SETUP
RL Problem Setup We model the robot manipulation task as Markov decision process denoted by the set (S, .A, P, R, y). representing the state space, the action space, the transition function, the reward function, and the discount factor. A policy a mapping from the observation state space to probability distribution over the robot action space A.
The observable state space S consists of the position of the object in the scene, the gripper's position. the interaction between the robot and the object (e.g.. whether the robot is picking up an object), and the current stage of task progression. The observation space for each long-horizon environment provided in 6
For the action space A, we use a subset of parameterized primitive skills from MAPLE (Nasiriany et al. (2022a)), specifically: Pick, Place, and Push. The descriptions of these skills are provided in Table The action space A is 9-dimensional, where the first three values represent the probability distribution the available skills, and the remaining six values serve parameters for the selected skill. Each primitive skill execution typically completes within 100 low-level actions. If execution exceeds 250 low-level actions, the robot stops and retracts to its home position.
65

---

Published as a conference paper at ICLR 2026
Environment
Observation Space
ThreeCubes
3D positions of the red, green, and purple cubes; grasp status of each cube; robot's gripper position; and the current stage of the task.
Dbwwatr
Positions of the drawer, drawer handle, apple, and soup can; grasp status of the apple and soup can; robot's gripper position; and the current stage of the task.
ObjectsAndBins
3D positions and grasp status of the tennis ball, baseball, orange, and
apple: positions of two bins; robot's gripper position; and the current stage of the task.
SpdereAndBins
3D positions and grasp status of the sphere; positions of two bins; robot's gripper position.
BallAndTarget
3D positions and grasp status of the ball; positions of the target, robot's gripper position.
Table Observation Space for Long Horizon Environments
Skill
Description
Pick
Moves the end-effector to a specified (x, y, z) location and closes the gripper grasp an object.
Place
Moves the end-effector to a specified (x, y, z) location and opens the gripper to release an object.
Push
Moves the end-effector to specified (x, z) location, then applies a displacement (ōx, öy. 82) to push an object.
Table 7: Parameterized Primitive Skills
The task horizon is inferred from the generated reward function and varies depending on human feedback. The remaining RL setup is consistent with the short-horizon case.
Training Setup We train a hierarchical policy (Nasiriany et al. (022aa) with two 512-dimensional layers using Soft Actor-Critic (SAC) (Haarnoja al. (2018b)). Hyper-params are given in Table 4
Training terminates early when evaluation accuracy reaches 0.9 or higher. If this threshold is not met, training continues until 200 million lower-level steps have been completed. The same training configuration is applied across all three long-horizon environments for every generated reward function, with hyperparameters optimized minimize training time.
To further improve sample efficiency and reduce training time, we apply a balanced rollout buffer that ensures rollouts are evenly sampled across all task stages, leading to faster learning, especially for the robot to learn later stages.
G COST ANALYSIS
Here, we compare cost of ROSETTA to Eurcka and Text2Reward. We note that Eureka and Text2Reward both use all rounds of reward iteration optimize performance whereas ROSETTA adds more overhead in a single round so that it can adapt in one step to a new task. The strategies that require these extra tokens, particularly staging and full code rewrite at each step, allow ROSETTA to outperform on task-changing preferences.
single task,
Terms:
Terms:
66

---

Published as a conference paper at ICLR 2026
c_code: average cost of writing reward function with ol-monn c_lang: average cost of grounding and staging with gpt 40 (significantly less than c_code)
c_train: robot training cost
num nun_mitte wwl.iiteaiinnnt number of iterations that the entire method is run for one task (ROSETTA: 1, Texr2Reward: 3, Eureka: 5)
c_reflection: average cost of doing one reward reflection
Expressions:
ROSETTA: 7* (c lang + c_ c_code) + c o_train 7 variants generated due to verification questions, variants trained and shown human
Text2Reward: 3*c 3+c_code + +3cc_train Text2Reward experiments use 3 iterations
per task
Eureka: rec_rrfflcctio ioteecioon400--- +00 -os_ 400+00000ttrii Eureka experiments use 5 iterations per task, each containing an evolutionary search with 5 evolutions and 16 branches per evolution.
We note that Eureka and Texi2Reward experiments less powerful LLMs than ROSETTA exper- iments do; reality, it is likely that all three methods will be closer in cost than they seem here, because powerful LLMs raise c_ costs but reduce hyperparameters like number of evolutions and evolution branch batch size. However, we expect the ranking to stay as-is.
For training costs, short-horizon tasks take 20-90 minutes tasks take 8-12 hours on the same, though they can be sped up on higher-end GPUs.
etitan rtx rtx 2080; long-horizon
67


Reviews:
[{"review_id": "NT3o4q922m", "reviewer": "Reviewer_BmrL", "summary": "This paper proposes a multi-stage LLM framework that is used to design reward functions that match indicated preferences at each time-step in open-ended behavior generation tasks. The paper constructs detailed prompts to help with this design process and appears to show improvement in human satisfaction above the baselines.", "questions": "- I don't understand the amount of human annotation used in these. The paper points to section F.2 but the relevant content is not there? Where are the details on your experimental protocol for the annotators? Is it one human watching 80 videos? 80 different people? How long are the experiments for them?\n- I'm not sure what is meant by Rosetta's \"domain knowledge\", a frequent claim throughout the paper. Where is the domain knowledge contained in Rosetta as opposed to in the LLMs that comprise Rosetta? More particularly, in what way does Rosetta have domain knowledge that your baselines does not (in particular, I'm responding to line 153)? Do you just mean the prompts in section C? If so, are similar prompts provided to your baselines?\n- What LLMs are used for the Eureka and Text2Reward baselines? Are they also using 4o and o1-mini?\n- Can you provide more information on the protocol used by the roboticists to evaluate your reward function? The text points to section D in the appendix but section D appears to be about something else. \n- In 372 to 377, what are these scores that are being given? Is it the average satisfaction score across several categories? Given that the performance degrades quite a bit when you paraphrase the prompts, why do the authors claim that \"the specific prompt matters less than the conceptual structure\"?\n- Why does the Sat. Score fall off so much after the first round?\n- In the real robot experiments, are the \"two-iteration preference chains\" generated by human annotators? What is the state dictionary that must be populated there?\n- How much did this all cost, particularly the annotations? This is relevant information for those who might choose to build on this paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- I really appreciate figure 5, the authors have a very thorough categorization of the different ways that users can express their preferences.\n- The paper appears to improve over the results of its baselines. \n- The paper provides an interesting, somewhat new goal of generating arbitrary behaviors in each iteration as selected by a human annotator.", "weaknesses": "Most of the \"weaknesses\" in this work are placed in the questions category below. I am uncertain if similar levels of tuning was done on the baselines relative to how much work was done in designing the prompts for Rosetta and would like to understand that better before being able to make a final judgement on the work.\n\n## Very minor (does not affect my score)\n- This paper puts me in mind of two papers, \"Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning\" by Clark et al. and \"In-Context Preference Learning\", but Yu et. al. which also have a similar iterative loop that uses human preferences to select amongst generated candidates. You may want to look at these works as neither are cited but seem highly related.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AfNW72yKjz", "reviewer": "Reviewer_Tpkx", "summary": "The paper tackles an important problem of adapting the behavior of embodied systems to natural language feedback. It adopts the approach of using foundational models to understand diverse and free form language commands and feedback to generate code that steers the actions of the agent towards the desired objectives. Further, the authors present additional evaluation metrics to analyse the behavior of the method beyond success rate, in terms of human satisfaction, feasibility and semantic similarity of the adapted behavior. The authors demonstrate that they out-perform the baselines across multiple tasks and users, with improvements in human satisfaction.", "questions": "- Do we expect Eureka to perform similarly if you can provide it with the correct fitness function at each step?  Is it an upper bound on performance? A small explanation of how the baselines were implemented under the interactive setting would be helpful to understand the differences from prior work.\n- What instructions are provided to the annotators? Because since the tasks in simulation have a single goal, how does the diverse preferences arise under the same objective for the relatively simpler tasks? Could the authors include some information about the number of participants, and other details about the experiments?\n- For the real-world tasks, it would be helpful for reproducibility to understand the implementation details.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The papers focus on the setting of unconstrained human preferences where the task is unknown initially and the agent has to adapt on the fly. It is a very important skill that is required for AI systems to be deployed in the real-world.\n- The paper is very well written with very descriptive figures, with the motivation, method and experiments clearly laid out.\n- The authors transfer their method across both simulation and real-world tasks which shows the strong applicability of the proposed solution.\n- The paper includes extensive qualitative examples from their experiments to show the different success and failure cases, while also thoroughly discussing the limitations of the paper.", "weaknesses": "- The method and the different components are explained well, but is it difficult to understand the individual contributions of each module to the final method? It would be interesting to see other additional baselines that run an LLM end2end for the whole tasks (with prompts to reason or use CoT to perform staging, ground and code generation by the same model)? This could make the contribution of having this modular stack as compared to making a single call to the LLM.\n- The authors include average reward values across different models in their method, but what is the average reward achieved by the different baselines? \n- In this method, it requires humans in the loop to provide the fitness function, which seems to be a very expensive process. It would be very interesting to talk about the sample efficiency and human effort required in the overall method?\n- Overall, the paper seems very solid, with some missing details in the presentation. I would be happy to raise my score as my questions are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8TqKwbAW5C", "reviewer": "Reviewer_zcpx", "summary": "This paper proposes Rosetta, a framework that leverages foundation models to ground and disambiguate natural language preferences, construct multi-stage reward functions, and implement them via code generation. The core contributions include the Rosetta framework itself, an evaluation framework measuring alignment, semantic match, and optimizability, and extensive experimental validation on 116 human preferences across five manipulation tasks. Key results report an 87% average success rate, 86% human satisfaction, and outperformance over state-of-the-art baselines like Eureka and Text2Reward, particularly for non-corrective preference types.", "questions": "1.\tHow does ROSETTA perform with long sequences of preferences in terms of performance degradation and computational efficiency? Does the foundation model context window become a bottleneck? (Sec. 5.2.3; Table 1)\n2.\tWhat is the impact of alternative error correction strategies (beyond iterative ol-mini) on success rate and human satisfaction? Could reinforcement learning for code correction be beneficial? (Sec. 4.3; Sec. 5.2.1)\n3.\tHow generalizable is ROSETTA to non-manipulation embodied tasks? Are there module design limitations requiring modification? (Sec. 1; Sec. 5.1)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Well-Structured Framework**: The three-module design (Preference Grounding, Staging, Coding) effectively handles unconstrained language by disambiguating references and structuring rewards, as illustrated in Fig. 2 and described in Sec. 4. Integration of domain knowledge through verification questions in the Coding module (Sec. 4.3) enhances reward reliability, with 95.38% of generated functions running after error correction. The framework supports online adaptation to dynamic preferences, maintaining performance over multiple iterations (Tab. 1), which is a significant advancement over prior fixed-goal methods.\n\n**Comprehensive Evaluation**: The proposed evaluation framework (Sec. 3.1) with alignment, semantic match, and optimizability metrics provides a holistic assessment beyond standard success rates, addressing the subjectivity of human preferences. Extensive human evaluation on 116 preferences across five environments (Sec. 5.1) demonstrates robustness, with results broken down by preference type (Fig. 3, Fig. 4) and language/content diversity (Fig. 5, Fig. 7). High performance metrics (86% human satisfaction, 87% success rate) across short-horizon and long-horizon tasks indicate practical utility for real-world applications.", "weaknesses": "**Insufficient Details and Reproducibility**: Hyperparameters for policy training are not detailed in Sec. 4.4 or Sec. 5.1, making replication difficult. Computational costs and resource requirements are omitted, despite the use of multiple LLM calls per preference in Sec. 4. The policy selection process in Sec. 4.4 relies on human annotators choosing from variants, but no inter-annotator agreement or selection criteria are reported, potentially introducing bias.\n\n**Limited Generalization**: The framework is task-agnostic but tested only in ManiSkill 3 environments; transfer to other simulators or domains is not evaluated, as acknowledged in Sec. 6 (Limitations). Semantic match evaluation in Sec. 3.1 relies on expert assessment, which may not scale and lacks objectivity compared to automated metrics; no inter-rater reliability is reported. The approach depends on specific foundation models (gpt-4o, ol-mini), and while open-source model tests in Sec. 5.2.1 show promise, full ablation on model choices is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZDk9ViC54K", "reviewer": "Reviewer_9E3U", "summary": "This paper introduces the ROSETTA framework, which uses LLMs to directly translate human natural language preferences into executable reward code. It addresses a problem in embodied intelligence: enabling agents to adapt to human preferences in real-time and online. ROSETTA can generate new reward functions and train policies, demonstrating performance significantly superior to existing baselines.", "questions": "1.Has the author verified the entire process in a real robot scenario?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses a critical problem by directly modeling human language, tackling the challenge of aligning with unconstrained feedback.\n\n- The capability for online adaptation is a major strength. The agent can adapt to a completely new human preference within a single interaction, which enables dynamic and continuous interactive alignment.\n\n- The evaluation methodology is comprehensive. The authors propose three-dimensional evaluation metrics that assess the policy from multiple perspectives, establishing a strong benchmark for this problem.\n\n- The framework was evaluated on a large-scale set of human preferences, achieving very high success rates and human satisfaction.", "weaknesses": "- Portability is questionable: The method relies heavily on domain knowledge. Migrating it to a new robot or a new environment would likely require significant manual setup.\n\n- The policy learning process is simplified: While the paper highlights \"single-step adaptation,\" it glosses over the policy learning process. The overall training loop is complex: after the reward function is generated, a policy must be fully trained in the simulator before the user can watch it and provide new feedback. The authors should provide details on the time and resources required for this training phase, as well as the iteration time for the LLM to generate the reward function.\n\n- The necessity of the human selection step is unclear: The system relies on training multiple reward code variants and then requires a human evaluator to watch videos and select the best one. It is unclear if this human-in-the-loop selection step is essential and how performance would be affected without it.\n\n- Experiments are simulation-only: Despite the large number of preferences and comprehensive experiments, the evaluations are conducted exclusively in simulation. Unlike a paper proposing a purely novel algorithm, this paper's primary contribution is a feasible framework. Therefore, it needs to demonstrate the feasibility of the full real-sim-real loop, perhaps with a simple case study. Simulation-only experiments overlook numerous potential errors that can occur in this cascaded process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)