ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics
Authors: Letian Chen
Nina Marie Moorman
Matthew Craig Gombolay
Venue: ICML 2025 poster (2025)
Keywords: Learning from Demonstration
Vision-Language Models
Inverse Reinforcement Learning
Primary area: applications->robotics
TL;DR: ELEMENTAL enables robots to learn user-aligned reward functions by combining user language instructions and demonstration through interactive self-reflection using vision-language models.

Abstract: Reinforcement learning (RL) has demonstrated compelling performance in robotic tasks, but its success often hinges on the design of complex, ad hoc reward functions. Researchers have explored how Large Language Models (LLMs) could enable non-expert users to specify reward functions more easily. However, LLMs struggle to balance the importance of different features, generalize poorly to out-of-distribution robotic tasks, and cannot represent the problem properly with only text-based descriptions. To address these challenges, we propose ELEMENTAL (intEractive LEarning froM dEmoNstraTion And Language), a novel framework that combines natural language guidance with visual user demonstrations to align robot behavior with user intentions better. By incorporating visual inputs, ELEMENTAL overcomes the limitations of text-only task specifications, while leveraging inverse reinforcement learning (IRL) to balance feature weights and match the demonstrated behaviors optimally. ELEMENTAL also introduces an iterative feedback-loop through self-reflection to improve feature, reward, and policy learning. Our experiment results demonstrate that ELEMENTAL outperforms prior work by 42.3% on task success, and achieves 41.3% better generalization in out-of-distribution tasks, highlighting its robustness in LfD.

ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics
Letian Chen
Nina Moorman
Matthew Gombolay
Abstract
Zhu et al., 2017). However, the effectiveness of RL hinges on the availability of carefully designed reward function that accurately encapsulates the desired behavior. Without sophisticated reward functions, RL agents often struggle to learn competent policies (Matignon et al., 2006); worse yet, poorly designed reward functions can lead RL agents un- desirable outcomes (Gupta et al., 2024). Booth al. (2023) shows reward specification is non-trivial even for experts, and designing reward functions that align with end-users' expectations is particularly challenging due to the varied and latent user preferences (Abouelazm et al., 2024).
Reinforcement learning (RL) has demonstrated compelling performance in robotic tasks, but its success often hinges on the design of complex, ad hoc reward functions. Researchers have explored how Large Language Models (LLMss) could en- able non-expert users specify reward functions more easily. However, LLMs struggle to balance the importance of different features, generalize poorly to out-of-distribution robotic tasks, and cannot represent the problem properly with only text-based descriptions. To address these chal- lenges, we propose ELEMENTAL (intEractive LEarning froM NEtoNsttaiion And Language), novel framework that combines natural language guidance with visual user demonstrations to align robot behavior with user intentions better. By incorporating visual inputs, ELEMENTAL over- comes the limitations of text-only task specifi- cations, while leveraging inverse reinforcement learning (IRL) to balance feature weights and match the demonstrated behaviors optimally. EL- EMENTAL also introduces an iterative feedback- loop through self-reflection to improve feature, reward, and policy learning. Our experiment re- sults demonstrate that ELEMENTAL outperforms prior work by 42.3% on task success, and achieves 41.3% better generalization in out-of-distribution tasks, highlighting its robustness in LfD.
Considering recent advancements in language models (e.g.. large language models (LLMs) and vision-language models (VLMs)) on text understanding (Bommasaní et al., 2021; Touvron et al., 2023) and emergent abilities (Kojima et al., 2022; Wei et al., 2022), researchers have explored utiliz- ing LLMs for reward engineering. For instance, the EU- REKA framework takes as input a text description of the task, queries language models for a draft of the reward function, and trains policy with the reward function (Ma et al., 2023b). This paradigm, while promising, presents several limitations. First, describing complex robotic tasks purely through language is imprecise: humans often have latent, unspoken preferences that are difficult to articulate fully, leading to incomplete or ambiguous descriptions of their objectives (Nisbett & Wilson, 1977; Ericsson & Simon, 1980; Hoffman et al., 1995; Feldon, 2007). Second, even all objective function components are accurately conveyed, determining the relative importance weights of these com- ponents poses another significant challenge: assigning these weights involves subtle mathematical trade-offs, something that I.LMs are not particularly equipped to handle. As a result of these limitations, methods like BUREKA struggle to generalize well to out- distrrbbutt tasks.
1. Introduction
Reinforcement Learning (RL) has been shown to be power- ful tool for enabling robots perform complex tasks across wide range of domains, from manipulation (Kober al., 2013; Levine et al., 2016) to navigation (Tai et al., 2017;
Given these limitations, more natural and effective ap- proach is for users to provide demonstrations of the desired behavior to supplement general task description. Demon- strations offer rich, illustrative information that can capture not only the task objectives but also the nuanced, latent preferences that may be difficult to express verbally. Learn- ing from Demonstration (LfD) approaches seek to leverage human-provided demonstrations to reverse-engineer the un-
'School of Interactive Computing, Georgia Institute of Tech- nology, Atlanta, United States. Correspondence to: Letian Chen <letian.chend gatech.edu>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ELEMENTAL
derlying objective and optimize a policy accordingly (Chen et al., 2020; 2022; Suay et al., 2016; Ravichandar et al., 2020). However, key challenge in LíD is the ambiguity in interpreting demonstrations - there can be an infinite number of possible reward functions that could explain the same set of demonstrations, a problem commonly referred to as the reward ambiguity problem in Inverse Reinforce- ment Learning (IRL) (Abbeel & Ng, 2004). Prior methods have sought to address this ambiguity by pre-designing fea- tures to constrain the space of possible reward functions, but this often limits the flexibility and generalizability of the learned rewards and policies (Zhu & Hu, 2018; Arora & Doshi, 2021). Our key insight is that language models are well-suited to contextualize demonstrations and infer task features, narrowing down the possible interpretations and enabling robots learn more robustly.
which facilitates more accurate behavior understanding.
2. We evaluate ELEMENTAL on a set of challenging, stan- dard robotic benchmarks in IsaacGym, demonstrating its superior performance over previous state-of-the-art (SOTA) reward design and LfD methods by 42.3%, show- casing its effectiveness.
3. We further assess EI ELEMENTAL'S generalization capa- bilities by designing novel variants of the standard bench- marks. Our results show that ELEMENTAL achieves 41.3% better generalization than existing methods, under- scoring the importance of combining VLMs with LfD.
2. Related Work
Learning from Demonstration (LID) - LiD approaches, such as Behavior Cloning (BC) and IRL, have long been used to enable robots to learn from human- suman-provided demon- strations. BC (Ross et al., 201 1), a supervised learning ap- proach, is effective for relatively simple tasks, but is prone to compounding errors (known as covariate shift). IRL (Ng & Russell, 2000; Abbeel & Ng. 2004; Ziebart et al., 2008; Ziebart, 2010) seeks to infer the underlying reward function that explains the demonstrated behavior. However, reward ambiguity poses a significant challenge - an infinite number of reward functions could explain the same behavior. This challenge becomes exacerbated in complex domains with limited, hetergeneous demonstrations (Chen et al., 2020; Peng et al., 2024a). ELEMENTAL addresses this key lim- itation by integrating VLMs to inject emergent reasoning capabilities into the learning process. VLMs reduce ambi- guity by providing semantic context that allows robots to better understand relevant task features.
We propose to integrate the strengths of language mod- els and LiD methods, leveraging (1) the emergent rea- soning capabilities of language models to identify robust and relevant objective function components, and (2) the capabilities of Lff to determine the optimal weighing of these components. Crucially, we incorporate visual demonstrations into Vision-Language Models (VLMs), facilitating a more comprehensive under- standing of human objectives. Additionally, we introduce a self-reflection mechanism that enables VLMs and LiD to iteratively improve both feature extraction and reward & policy learning. This fusion of LfD and VLMs offers a novel pathway for more effective robotic learning from demon- strations. This paradigm also mirrors how humans naturally learn from others. When observing demonstration, hu- mans typically (1) identify the key aspects of the task, (2) formulate a policy to match the demonstration based on those salient features, and (3) reflect on the discrepancies between their own behavior and the demonstration to refine their understanding and execution (Locke, 1987; Di Stefano et al., 2014). By iterating through these steps, humans pro- gressively improve their performance. This iterative cycle of observation, reflection, and refinement is not only fundamen- tal to human learning but also serves as an idcal framework for robotic learning (Chernova & Thomaz, 2014).
cimnonnttaaiinmmatcinn
Language Models as Reward Engineers- Recent works, such as EUREKA (Ma et al., 2023b) and L2R (Yu et al., 2023), leverage LLMs to convert language descriptions into reward functions, offering a promising alternative to man- ual reward enginecring. However, these methods are lim- ited by their reliance solely on concise task descriptions, restricting their ability to capture the full complexity of robotic tasks and the subtle preferences of users. Addi- tionally, determining the appropriate weighting of different objective function components is particularly challenging, as the reward design process is disconnected from policy training, resulting in poor -iitr---iitiitttion generalization. ELEMENTAL addresses these limitations by integrating IRL with VLMs and supplementing task descriptions with demonstrations. In ELEMENTAL, the responsibility of assigning reward component weights is shifted from the VI.M to IRI., which matches the reward components to the demonstrated behaviors. This allows the VLM to fo- cus on its strength-semantie semantic understanding and task fea- ture identification. ELEMENTAL differs from prior work that uses language as a semantic prior or task modeling
To develop this novel integration between VLM and LfD, we introduce ELEMENTAL (intEractive LEarning froM dEmoNstraTion And Language). ELEMENTAL enables robots identify key task features from human demonstra- tions, learn rewards and policies that align on these features, and iteratively reflect on their performance to improve over time. Our key contributions are three-folds:
1. We propose novel, general framework that integrates VLMs and LID and introduces an iterative self-reflecting mechanism for autonomous performance improvement. ELEMENTAL is the first to incorporate visual demon- stration inputs into language models to accomplish LfD,
2

---

ELEMENTAL
Phase 1: Initial Prompt
Phase 2: Learning
Phase 3: Reflecting
Text MDP Environment Code
Feature Function
Rollout
(s): R
GM pleylt,, dler (ecirtt soo (om
legett, und
Task Description
mate the ast rat ferverd as fest postiate
VLM
www.s/eeee.eeee...... (tome Heigt 4-20 262 402
IRL
Visual Demo
Feature Counts
$ B--r-m [o())
2 B--r- (+)
Reward & Policy
(s) = (T)(s) My argmax
Self- Refleceelott for next-iteration feature function
Figure 1. This figure illustrates the overall pipeline of ELEMENTAL. The process begins with an initial prompt to the VLM, which generates a draft of the feature function based on both textual descriptions and visual demonstrations. In the learning phase, ELEMENTAL infers the reward and policy from the drafted feature function and the demonstration. In the final phase, ELEMENTAL performs self-reflection by comparing the feature counts from the generated trajectory and the demonstration, again utilizing the drafted feature function. This self-reflcction loop updates the feature function by feeding the results back to the VL.M for iterative refinement.
aid (Lin et al., 2023; Ma et al., 2023a; Fan et al., 2022; Nair et al., 2022: Myers et al., 2024; Adeniji et al., 2023). These methods often employ language to pretrain representa- tions (Nair et al., 2022), construct modular task plans (Myers et al., 2024), or inject commonsense priors into embodied agents (Fan et al., 2022; Lin et al., 2023). In contrast, ELE- MENTAL leverages VLMMs to directly generate structured, executable feature functions from multimodal input, which are then used for downstream IRL-based reward inference. This unique pipeline grounds reward learning in both lan- guage and demonstrations and supports iterative refinement through feedback.
textual descriptions alone are insufficient. In addition, to account for the complex domains, we introduce an upgraded MaxEnt-IRL approach, detailed in Sec. 4.2. By incorporat- ing visual modality, using general prompt across domains, and developing enhanced IRL, ELEMENTAL provides a more scalable solution, as demonstrated in our results on standard robotic benchmarks without requiring any prior knowledge of task features.
LM-A Robot Learning- Several recent works have sought to leverage LMs to assist in robotic learning, such as RI-VILM-F (Wang et al., 2024), RoboCIII (Son- takke et al., 2024), and (Du et al., 2023). While promising, these methods depend on surrogate reward derived from the LM's understanding of the task-state alignment, which can introduce inaccuracies. Furthermore, these approaches lack interactivity with users, a key component shown to be helpful in gaining human trust (Chi & Malle, 2024). In contrast, ELEMENTAL potentially allows engineers and users to interactively refine the robot's behavior, ensuring that the robot is user-aligned.
A closely related line of research is that by Peng et al. (2024b), which also integrates feature design using LLMS with IRL. While this prior work is promising, it operates under several limiting assumptions. First, assumes that demonstrations can be provided as input exclusively text- based state-vector sequences. Due to this constraint, their approach is only applied to simpler tasks (e.g., problem hori- zons of fewer than five steps) with all but one or two features of the ground-trutt features already specified. Additionally, as seen in other works (Yu et al., 2023; Kwon et al., 2023), this method relies on specially designed prompts tailored to each experimental domain. These limitations hinder its scal- ability to more complex, high -igh-iiiension tasks typically encountered in real-world robotic applications. In contrast, ELEMENTAL leverages visual modality for demonstration inputs, making it well-suited for more complex tasks where
Other works, such as Wang al. (2023) and Mahadevan et al. (2024), directly query LMs to output robot actions primitives. These approaches rely heavily on the LLM''s ability to plan and optimize actions, but LLMs are not in- herently designed for mathematical optimization required for robot control. ELEMENTAL addresses these limitations by using VLMs to understand task features and deferring
3

---

ELEMENTAL
the demonstration-tooppiee alignment to IRL algorithms, which are better suited to optimize behavior.
involves constructing an initial feature function through a VLM based on visual human demonstrations and environ- ment specifications (Sec. 4.1). In the second phase, this fea- ture function is integrated with IRL to learn a reward func- tion and policy that best align with the demonstrated hehav- iors (Sec. 4.2). The final phase introduces a self-reflection mechanism that automatically compares the learned behav- ior with the demonstrations, enabling iterative refinement of the feature function, reward function, and policy (Sec. 4.3).
3. Preliminary
In this section, we introduce Markov Decision Processes, Inverse RL, and maximum-Entrooy IRL.
Markov Decision Process: We formulate the robot learning problem Markov Decision Process (MDP), defined by the tull (S, A, r) S is the set of states the agent can occupy. A is the set of actions the agent can take. T S x A + [o, 1] represents the transition probability function, where T(s, a, s gives the probability of transitioning from state 8 to state after taking action a. R S -> R is the reward function that assigns scalar reward to each state. Y E [0,1) is the temporal discount factor. The goal of Reinforcement Learning (RL) is to learn policy π : S + A that maximizes the expected cumulative reward, given by: (TT = Er Ei 7 [(ss)].
4.1. Phase 1:
Initial Prompt
The first phase begins with an initial prompt of three inputs: (1) the MDP environment code specifying the environment's state space S, (2) text description of the task, and (3) a visual human demonstration of the desired task. The form of the visual demonstration depends on the task domain (Figure 2): for tasks where superimposition is meaningful (e.g., locomotion tasks such as the moving-forwan ant or humanoid), we compose a superimposed image that displays the sequences of demonstrated motion (shading indicates temporal progression). For manipulation tasks, where super- imposition can result in cluttered robot poses (e.g., Franka- Panda Cabinet), four keyframes picked by robotic experts are provided to the VLM to illustrate stages of the task exe- cution. By incorporating visual demonstrations alongside language-based based task descriptions, ELEMENTAL allows the VLM generate feature functions that more accurately reflect the user's latent goals.
Inverse Reinforcement Learning: In IRL, instead of ex- plicitly programming a robot's behavior, we aim to learn the underlying objective or reward function, R(s), which explains the behavior demonstrated by human expert. A set of demonstrations are given, D = (T where each trajectory T; = (s), ai, $2.02.... .). consists of a se- quence of states and actions. We assume that the true re- ward function, R(s), is a linear combination of features: R(s) = 02 d(s), where dss) E Rdd feature vector representing the task-relevant or ussr-preference-eelevaa properties of the state, and f Rd is weight vector that specifies the relative importance of each feature. The goal of IRL is to recover P based on the provided demon- strations, D. MaxEnt-IL (Ziebart et al., 2008) models the likelihood of a trajectory under the assumption that the expert's behavior is stochastically optimal, as shown in Eq. 1. In this equation, Z()) is the partition function, z(0)= E, exp (Ea,er R(se)).
These inputs are then processed by the VLM, which lever- ages its emergent capabilities to infer task-relevant features. The goal is not just to describe the state but also to cap- ture important factors that align with the user's intentions for the task. The output of this phase is a feature function & S + R", where (ss) (ci (s), Φ2(5),.... (s)) is n-dimensional vector of features describing the key as- pects of the task. Given VLM coding capabilities (Piccolo et al., 2023), we ask the VLM to output feature functions as Python code, providing a structured representation. If the returned code is not executable (e.g.. wrong function signature), we re-prompt the VLM up to three times with the trackback information. A successfully executable feature function serves as the foundation for reward learning and policy optimization in the subsequent phase.
0( = Z(0) e Enter R(si)
e E.ner d(e( Z(0)
MaxEnt IRI. seeks to infer the reward weights / that maxi- mize the likelihood of the expert demonstrations (Eq. 2).
i arg MAX
Σ
log P(
(2)
TED
4.2. Phase 2: Learning
In ELEMENTAL, we upgrade MaxEnt-IRL for high- dimensional robotic tasks, allowing us to link features from VLMs to demonstrations via the weight vector 0.
After the feature function has been drafted in Phase 1, we next optimize the reward function, [s))) 0 (s)) to match the demonstrations using the feature function, d(s). As in Sec. 3, we modify the MaxEnt-IRL. algorithm to ac- complish reward and policy learning in more complex tasks. We name Approximate MaxEnt- (Algorithm 1).
4. Method
In this section, we present ELEMENTAL, which consists of three interconnected phases (Figure 1). The first phase
Directly computing the partition function Z(0) is intractable
4

---

ELEMENTAL
(b) start the trajectory.
(c) reaching the cabinet handle.
(a) Superimposing ant demonstration.
(d) opening the cabinet.
(e) the cabinet is opened.
Figure This figure illustrates the visual demonstrations for both locomotion manipulation tasks. (a) shows an example the locomotion task, where superimposed images used. For manipulation tasks, superimposed images can result in cluttered robot poses, so we use key frames as visual demonstration inputs instead. (b-e) present four key frames from the FrankaCabinet manipulation task.
optimizes the estimated reward, Ro(s) = 0 ((8). (), in Eq. 4.
Algorithm 1: Approximate MaxEnt-IRL.
Input :feature function S - R" demonstration D, number of IRI.
Vo = ETWD Σ {(s)) LEET
Σ LOET
EYNTy
E
((s))
(4)
iterations m, learning rate a, policy learning steps
In particular, optimize the policy Tे interleaved with Rg. following a paradigm similar to that of AIRL (Fu et al., 2018), as shown in Algorithm lines 3-4 and line 6. In- tuitively, by applying this gradient, we refine that the reward function Rø(s) more accurately reflects the features emphasized the demonstration and guide Th to align the policy's behavior closer to the demonstrated behav- ior. The learned policy, Te. is also a natural byproduct of this process, serving as the ultimate goal of LID and laying the foundation for the reflection phase in Phase
Initilize reward function feature weights {1/n] and policy weights s). for <- to do
ensure
for j to & do
Optimize ne. based on J (TT) with
Ro(s) = 0T ((s) via PPO
Obtain Vo by Eq. and normalize to get V by Eq.
Update A by +- 8 + +aV Normalize by Eq. 6
7
As the feature's 's numerical magnitude varies, we apply a normalization procedure the reward gradient ensure stable learning. First, normalize the Ll-norm of the gradient vector in Eq. 5.
Output:RA n
due to the summation over all possible trajectories. We circumvent the need for explicit computation of (()) with the key insight (Zicbart et al., 2008) that the gradient of the MaxEnt IRL objective is given by the feature expectation difference between the demonstrated trajectories and the stochastically optimal trajectory under 0, shown in Eq. 3.
V
V
(5)
Velli
Next, we apply gradient ascent on the reward weight vector, 6, using a learning rate a: 0 < 0 V. After each update, we normalize (Eq. 6) to stabilize the training of the policy, Ty. the semantics of a reward function do not change by scaling (Fu al., 2018).
Vg E,<D Σ d(s) - ET-P(T)0)
Ess)
(3)
LEET
SET
0
We approximate P(T(0) by parameterized policy, Np. that
|
(6)
5

---

ELEMENTAL
4.3. Phase 3: Self-Reftecting on Features
due to its high-dimensional state and action spaces. The performance of ELEMENTAL in these tasks demonstrates its robustness and scalability, making it a suitable framework for real-world robotics problems using IRL.
The third phase introduces a self-reflection mechanism, de- signed to close the loop and iteratively refine the feature function drafted in Phase 1. After Phase 2, the learned pol- icy, (s), is executed in the environment, and its behavior is compared to the demonstration, D. based on the drafted feature function (()). Specifically, we calculate the ex- pected feature counts of the generated trajectories under the current policy and for the demonstration trajectories (Eq. 7).
Baselines
We compare ELEMENTAL against key LfD
and LLW-poeered reward engineering approaches:
1. LID methods - We include standard LfD techniques, i.c. Behavior Cloning (BC) and IRL. These baselines learn from demonstration but do not incorporate the VL.M feature-extraction or visual input.
nn = EETTT
Σ ((s)
DD = EEnD
Σ (s))
(7)
2. EUREKA- This is the previous SOTA method for reward design with RL in IsaacGym. EUREKA relies on LLMs to infer task features from textual inputs but does not utilize demonstrations, visual inputs, or IRL.
Discrepancies between the two feature counts indicate that the current feature function may not fully capture the rele- vant aspects of the task as demonstrated.
The two feature count vectors are then fed back to the VLM, which uses the feature count differences revise the feature function d(s). By accounting for previously overlooked or misinterpreted features, the VLM's understanding of the task becomes progressively more aligned with the demon- strated behavior. This process of self-reflection continues iteratively, alternating between Phase 2 (reward function and policy optimization) and Phase 3 (feature refinement), allowing the robot to improve its behavior over time.
3. Random Policy- A random policy establishes an approx- imate lower bound for competent task performance.
4. Ground- Truth (GT) Reward - The performance of a pol- icy trained with GT reward predefined in IsaacGym pro- vides an upper bound of the task performance. Note that although we call it "upper bound", it is not necessarily the maximum performance one can achieve, as the RL is given the same budget of environment steps train, and the GT reward may not be the best.
The reflecting phase is fully automatic, leveraging the policy, (s), and the feature function, d(s), generated in previ- ous phases. As both the policy and the feature fiunction are available, ELEMENTAL can continuously refine its under- standing of the task without additional human input. How- ever, should the user wish to intervene and provide further feedback or corrections, ELEMENTAL could accommodate interaction in future work through prompting.
5. Ablations of ELEMENTAL - To better understand the contribution of each component in ELEMENTAL, we conduct ablation studies across six variants: 1) With- out (w/o) self-Reflection This variant removes the self- reflection mechanism introduced in Phase 3, keeping only the VLM-guided feature inference and policy learning phases; 2) w/o Gradient Normalization: This ablation removes the gradient normalization in Eq. 5; 3) w/o Weight Normalization: This ablation removes the weight normalization step in Eq. 6; 4) w/o Visual Input: This variant removes the visual demonstration input to the VLM, leaving only language-based task descriptions for feature inference; 5) w/ Text Demo: Instead of visual demonstrations, this ablation provides textual input by listing observation vectors from a subsampled sequence of ten states from the demonstration, same as (Peng et al., 2024b); 6) w/ Random Visual Demo: To test whether the VLM effectively extracts task-relevan information, replace the high-quality visual demonstration input with random visual demonstrations (e.g., falling Cartpole or Humanoid). The IRI. phase, however, still utilizes the original high-quality demonstrations.
5. Results
In this section, we evaluate the performance of ELEMEN- TAL on challenging robotic tasks from IsaacGym (Makoviy- chuk et al., 2021) against SOTA baselines (Sec. 5.1) and show generalization task variants (Sec. 5.2).
Environments and Tasks - In benchmark experiments, we test ELEMENTAL and baseline algorithms on nine chal- lenging IsaacGym Robotics tasks, using GPU-accelerated training to enable efficient experiments. These tasks span various domains, including locomotion and manipulation, and are recognized for their complexity in the robot learning community (Makoviychuk et al., 2021). For all methods that utilize LLM/VLM, we use the OpenAl GPT-4o model unless otherwise noted. We use five demonstrations for each task collected with RL-trained policies.
Performance is evaluated using an average task success rate of the fin steps during training, which is a more reliable metric versus only using the max success during training reported by Eureka, In all experiments, we report
To the best of our knowledge, this is the first successful ap- plication of IRL to IsaacGym- with or without large models
6

---

ELEMENTAL
Table This table shows benchmarking results IsaacGym tasks. Bold denotes the best performance except the GT Reward.
IsaacGym Environments
Method
Ball Balance
Franka Cabinet
Allegro Hand
Shadow
Cartpolc
Quadcopter
Ant
Humanoid
Anymal
Hand
Random (LB)
25.42
87.39
-1.63
0.00
0.00
-0.04
-2.45
0.00
0.02
149.85
344.55
-1.19
0.01
-0.05
-0.43
-2.14
0.04
0.03
IRI.
28.15
162.06
-1.87
0.00
0.88
2.13
-2.22
0.01
EUREKA
215.91
454.18
-0.22
0.21
6.88
3.78
-424
11.12
0.00
ELEMENTAL (Ours)
233.92
464,40
-0.30
0.36
8.49
4.70
-0.83
22.97
2.71
w/o Self-Reflection
114.66
153.52
-0.93
0.00
5.05
3.65
-1.71
0.02
0.03
w/o Gradient Normalization
186.52
423.78
-1.20
0.02
7.29
2.73
-0.55
13.39
2.32
w/o Weight Normalization
192.51
459.33
-0.54
0.02
3.23
4.87
-1.15
0.04
1.57
w/o Visual Input
178.68
304.58
-1.01
0.00
8.16
4.49
-1.41
18.52
0.03
w/ Text Demo-
207.51
412.17
-0.92
0.00
7.43
4.60
-0,88
7.07
0.04
w/ Random Visual Demo.
269.46
352.53
-1.07
0.02
7.13
3.93
-1.07
20.75
2.33
GT Reward (UB)
260.14
461.90
-0.27
0.40
7.00
5.07
-0.03
23.70
0.15
We tried running with six seeds, but Eureka with GPT-4o failed to generate executable reward function for ShadowHand. In the calculation of percentage improvements Eureka, we treat the improvements ShadowHand to be 100%.
As the original implementation (Peng et al., 2024b) not available, we implement demonstration text form with ELEMENTAL.
the best performance across three random seeds to account for variable responses from GPT-4o. Hyperparameters and prompts are provided in supplementary.
code execution rate approximately 80% in the first it- eration, compared to less than 50% EUREKA. While both methods improve with successive iterations, ELEMEN- TAL consistently generates more executable code across all tasks. A two-way repeated measures ANOVA across nine paired tasks reveals a significant main effect of algo- rithm (F(1,8) 7.00, = .030) and round (F(2, 16) 10.03, p = .002). indicating that ELEMENTAL achieves statistically significantly higher execution rates than EU- REKA. The interaction between algorithm and round is not significant (F(2, 16) 2.21, p .144). These results support the interpretation that GPT-4o is more effective at generating executable feature extraction code than complete reward functions, thereby validating ELEMENTAL'S design choice to delegate reward weighting to IRI..
5.1. Benchmarking Results
In the first set of experiments, we benchmark ELEMEN- TAL and the baselines on the IsaacGym tasks, with the results presented in Table 1. BC performs adequately simpler tasks such as Cartpole and BallBalance, but fails on more complex tasks due to covariate shift. IRI. without VLM-based feature extraction struggles to learn effectively in most tasks, highlighting the challenges posed by Isaac- Gym's high-dimensional state spaces, While EUREKA able learn capable policies, ELEMENTAL achieves on-average 42.3% higher performance and outperforms BU- REKA on eight out of nine tasks, demonstrating the effec- tiveness of integrating IRL with VIM-derived features and visual demonstration information. The correlation between learned reward and ground-truth reward shown in Figure 2 also illustrates ELEMENTAL's strong ability to learn a well- aligned reward function by matching with demonstrations. Expanded results with statistical significance tests and addi- tional baselines (e.g., VLM+BC and ablations with different VLMs) are shown in Supplementary C.
The ablation results Table I demonstrate the impor- tance of ELEMENTAL's key components. Removing self- reflection significantly reduces performance, confirming its role in refining both the feature function via VLM and the policy via IRL. Notably, even without self-reflection, ELE- MENTAL outperforms standard IRL, indicating that the ini- tial MTM--genratted feature function already provides ben- efits. The normalization steps are also removiggg them leads to unstable reward learning and inconsistent scaling, negatively impacting policy performance.
To further evaluate whether VLMs are more suitable for feature extraction or for drafting full reward functions, we compare the code execution rates of ELEMENTAL and EUREKA across three algorithm iterations. A higher code execution rate indicates fewer coding errors, suggesting better compatibility with language model capabilities. As shown in Figure 3, ELEMENTAL achieves successful
Performance degrades more sharply when substituting vi- sual demonstrations with textual inputs, as in the setup from Peng et al. (2024b), suggesting that VLMs leverage visual semantics more effectively than structured state vec- tors posspaaticularly on tasks that are difficult to describe via language alone. For example, FrankaCabinet. replacing

---

ELEMENTAL
Table 2. This table shows reward correlation of inferred reward functions for EUREKA and HLEMENTAL (ours) on IsaacGym tasks.
IsaacGym Environments
Mothod
Ball Balance
Franka Cabinet
Allegro Hand
Shadow
Cartpole
Quadcopter
Ant
Humanoid
Anymal
Hand
EUREKA ELEMENTAL (Ours)
0.77
-0.53 0.85
0.96
0.93
1.00
0.59
-0.86
0.34
0.00
0.99
0.89
0.98
1.00
0.98
0.97
0.58
0.31
Table This table compares the generalization performance ELEMENTAL and EUREKA on Ant-variant environments. Results are maximized over three seeds. Bold denotes the best performance.
Method
Ant Original
w/ Reversed Obs
w/ Lighter Gravity
Ant Running Backward
EUREKA ELEMENTAL
6.88
5.96
4.39
5.62
8.49
8.47
5.89
9.30
w/o Visual Input
8.16
7.63
3.14
7.46
visual demonstrations with text inputs reduces performance from 0.36 to 0.00, underscoring how VLMs extract more actionable features from visual inputs than from tokenized state vectors (Peng et al., 2024b).
2. Ant with Reversed Observations: The order of the state vector is reversed, testing the algorithms' ability to adapt to changes in the structure of the input data.
3. Ant with Lighter Gravity: The gravity coefficient is re- duced from 9.81 to 5.00, requiring the features and policy to adjust for different dynamics. A performance drop is expected; the ant moves with less friction.
Additionally, using random visual demonstrations leads to significant drop in performance, confirming that VLMs extract meaningful task information when provided with high-quality demonstrations. These results highlight the necessity of self-reflection, normalization, and multimodal demonstrations in achieving ELEMENTAL's stronger per- formance.
4. Ant Running Backward: The task is modified to require the Ant to move backward rather than forward, assessing how well the approaches generalize to a different task objective in the same environment.
Under the same computational resource setup, EUREKA av- eraged 68.21 minutes of runtime across the nine tasks, while ELEMENTAL averaged 168.36 minutes. This increase is primarily due to the environment rollouts ELEMENTAL uses to estimate the reward gradient. However, this trade-off enables ELEMENTAL to learn a better-aligned reward func- tion, leading to improved task success and generalization. We discuss future work to reduce runtime in Sec.
We show the comparison between ELEMENTAL and BU- REKA in Table 3. In out- -iitt-i-iittiittionn tasks that language models have not seen in the training set, EUREKA' A's per- formance declines, suggesting GPT-4o might have mem- orized the IsaacGym task rewards, which not helpful when the state vector reversed, when the environment dynamics change, when the task objective is altered. In contrast, ELEMENTAL queries the VLM only for feature functions-informaaiic not available in the IsaacGym pub- IRL process
5.2. Generalization Results
lic data-and uses the inmoonstration-matche to determine the reward weights. ELEMENTAL accom- plishes an average performance improvement of 41.3%, highlighting its robustness in adapting changes in both the environment's physical properties and the task's nature.
In the second set of experiments, we ask the question "are the LLMs/VLMs just remembering the reward function, as the knowledge cut-off date is Oct 2023, after IsaaeGym is public?" To answer this question, test the generaliza- tion capability of ELEMENTAL by applying it to modified versions of the IsaacGym environments, particularly vari- ants of the Ant task. For these modified environments, we change certain properties, such as state vector order, physics property (e.g., gravity coefficient), and the task, to evalu- ate whether ELEMENTAL'' VL.--riveen feature inference combined with IRL can adapt new, unseen environments better than EUREKA. We test four Ant variants:
5.3. World User Study: Teaching Salad Mixing
To evaluate ELEMENTAL's real-world effectiveness, we conducted within-subject user study where twelve partici- pants taught a Kinova JACO arm to perform salad mixing task. ELEMENTAL was able to operate interactively in real time: each learning round completed within 4 minutes using a remote server with an NVIDIA A40 GPU. Demonstrations were captured using a ZED stereo camera and automatically converted into ten equally spaced visual frames per skill.
1. Ant Original: The standard Ant task without modifica- tions, serving as the baseline environment.
8

---

ELEMENTAL
1.0
tracted from ZED cameras, and ELEMENTAL operated entirely from these visual inputs and user feedback, without manual annotation. We provide an illustration of the user study setup in the supplementary material (Figure 4), with additional examples and insights in Appendix A.
0.8
Execution Rate
0.6
6. Limitation and Future Work
0.4
While ELEMENTAL demonstrates strong improvements in LfD with VLMs, it has several limitations that open av- enues for future research, One key limitation is its reliance on environment rollouts for IRL updates, which increases wall-clock running time. This is a trade-off that allows EL- EMENTAL to infer reward functions that better align with demonstrations. Future work could explore more stable and efficient RL/IRL methods (Henderson al., 2018; Hussenot et al., 2021; Adkins et al., 2024), such as Adversarial IRL (AIRL) or model-based RI., to reduce sample complexity. Additionally, parallelized or hardware-accelleatte imple- mentations could help mitigate runtime concerns, making ELEMENTAL more feasible for real-world deployment.
0.2
Eureka ELEMENTAL
0.0
2
3
Iteration
Figure 3. A comparison of the code execution rate between ELE- MENTAL and EUREKA A in three iterations in the nine IsaacGym environments. The shades show the standard error.
Participants taught three core skills: go grasp mushroom, go drop mixture bowl, and mix bowl with spoon, using both ELEMENTAL and the baseline EUREKA (order random- ized). Both methods used the same hardware, interaction rounds, and input text format, ensuring a fair comparison. The key distinction is that ELEMENTAL uses demonstra- tions and IRL to learn and refine reward functions, while EUREKA relies on LLM-generated reward code from lan- guage only. The remaining skills were predefined and shared across both conditions. Teaching followed two interaction rounds per method: for each skill, participants provided a kinesthetic demonstration and natural language intent de- scription, observed robot execution, and gave feedback used to refine the feature function and/or the reward function. After teaching, participants observed blind executions of the full salad mixing task and evaluated both methods using 7-point Likert scales on two criteria: task performance (suc- cessful execution) and strategy alignment (intent matching). Each criterion included four questions (max score: 28).
ELEMENTAL also assumes access to reasonably high- quality demonstrations. While our real-world user study showed that ELEMENTAL can recover from imperfect in- puts through feedback, we observed that performance de- grades with completely random or low-quality demonstra- tions (e.g.. Table 1, EL ELEMENTAL w/ random visual demo). Future research could explore filtering and weighting strate- gies better handle noisy or suboptimal demonstrations, For example, filtering could be based on user confidence, VLM scores, automated ranking mechanisms. Moreover, recent advances in learning from suboptimal demonstra- tions (Brown et al., 2019; Chen et al., 2021; Beliaev et al., 2022) could be integrated into ELEMENTAL'S IRL. pipeline to further improve robustness.
Finally, given the demonstrated utility of visual inputs, fu- ture work could explore alternative demonstration modali- ties, such as continuous video, which may be more natural for users and improve generalization across robotics tasks. Incorporating real-time human feedback during the self- reflection process also remains a promising direction for adapting the fcature and reward refinement loop.
ELEMENTAL significantly outperformed EUREKA on both metrics. Task scores were 20.58=4.99 4.3 vs. 1272
((11) -4.65, p < .001; strategy scores were 6. vs. 10.50 + 4.32, t(11)= = -4.20, p .001. Participants remarked positively on ELEMENTAL's robustness to noisy demonstrations. For example, one participant noted: "Even if my demonstration was slightly to the left of the mixture bowl, ELEMENTAL can help me fix this when give it feed- buck and successfully put ingredients in the mixing bowl."
7. Conclusion
We introduced ELEMENTAL. novel framework that inte- grates VLMs with LfD to draft feature functions from visual demonstrations. ELEMENTAL leverages IRL to match feature functions with demonstrations and introduces an iterative self-reflection mechanism for continuous improve- ment. Experiments showed that ELEMENTAL outperforms previous -tate-of-the-art methods by 42.3% standard IsaacGym tasks and 41.3% in generalizing to novel tasks.
These results support the hypothesis that ELEMENTAL aligns better with user intent, even under imperfect input. Moreover, the study further supports the system's 's scala- bility: no manual keyframe engineering was needed, and the same hyperparameters used in simulation were applied without tuning. Notably, the weal-world pipeline required no hand-laheling demonttraions were automatically ex-
9

---

ELEMENTAL
Acknowledgement
Canada, July 4-8, 2004, volume 69 of ACM Inter- national Conference Proceeding Series. ACM, 2004. doi: 10.1445/10153300.01144330 URL https: /tttp:://oo... 2zg/10..145/11153000.11 10154433
We wish to thank our reviewers and area chairs for their valuable feedback in revising our manuseript. This work was supported by the National Science Foundation (NSF) under grant IIS-2340177, 77, grant IIS-21 12633, and grant CPS 2219755, as well as the National Institutes of Health (NIH) under grant 15715L
Abouelazm, A.. Michel, J., and Zoellner, J. M. A re- view of reward functions for reinforcement learning in the context of autonomous driving. ArXiv preprint, 0ass22405.01440, 2024. URL https:// hatp:://arxivorr/ abs/2405, 01440.
Impact Statement
Adeniji, A., Xie, A., Sferrazza, C., Seo, Y., James, S. and Abbeel, P. Language reward modulation for pretraining reinforcement learning. ArXiv preprint, 010/2308.127000 2023. URL htttttt//axxiiii abs
ELEMENTAL advances LID by integrating VLMs with IRL improve reward function engineering. Our contribu- tion enables robots to learn from both natural language and visual demonstrations, reducing the need for hand-crafted reward functions and making robot learning more accessible to non-experts. By improving the alignment of learned be- haviors with human demonstrations, ELEMENTAL has the potential to enhance human-robot interaction in areas such as assistive robotics, industrial automation, and adaptive learning systems.
Adkins, J., Bowling, M., and White, A. A method for evaluating hyperparameter sensitivity in reinforcement learning. Advances in Neural Information Processing Systems, 37:124820-124842, 2024.
Arora, S. and Doshi, P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Despite its benefits, ELEMENTAL, like all Al and robotics research, is a dual-use technology. While ELEMENTAL offers significant benefits, responsible use will require safe- guards around demonstration quality, oversight of iterative reward refinement, and transparency in how VLM-derived features influence learned behaviors. Additionally, while ELEMENTAL improves transparency in reward learning, its dependence on VLMs introduces potential challenges such as ballucinations or overfitting to language priors.
Beliaev, M., Shih, A., Ermon, S., Sadigh, D., and Pedarsani, R. Imitation learning by estimating expertise of demon- strators. In International Conference on Machine Learn- ing. pp. 1732-1748. PMLR, 2022.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportuni- ties and risks of foundation models. ArXiv preprint, abs/2108.0720 072558 2021. URL https: httpp::s//axxv.oop//
Another ethical consideration is the potential misuse of ELE- MENTAL for applications beyond its intended scope. While the framework is designed for positive use cases, such as making Al training more interpretable and user-aligned, the ability to iteratively refine reward functions could be lever- aged in ways that raise ethical concerns, such as in surveil- lance or military applications. We advocate for responsible AI development and encourage the use of human-in in-the- the- loop safeguards, bias auditing, and clear ethical guidelines for deployment.
Booth, S., Knox, W. B.,, Shah, J., Niekum, S., Stone, P., and Allievi, A. The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifica- tions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Brown, D. S., Goo, W., Nagarajan, P., and Niekum, S. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. Chaudhuri, K. and Salakhutdinov, R. (eds.). Pro- ceedings of the 36th International Conference chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 783-792. PMLR, 2019. URL
Overall, ELEMENTAL represents a step toward more intu- itive and generalizable AI systems by integrating demonstra- tions with language guidance. While its potential benefits are significant, careful attention must be given to ensuring fairness, transparency, and responsible use as these tech- nologies continue to evolve.
Ma-
References
mwr. p.r.srses/
Abbeel, P. and Ng, A. Y. Apprenticeship learning via in- verse reinforcement learning. In Brodley, C. E. (ed.), Machine Learning, Proceedings of the Twenty-first In- ternational Conference (ICML 2004), Banff, Alberta,
v97 /0/bbrwn11 1 9a html.
Chen, L., Paleja, R., Ghuy, M., and Gombolay, M. Joint goal and strategy inference across heterogeneous demonstra- tors via reward network distillation. In Proceedings of the
10

---

ELEMENTAL
International Conference Human-Robot Interaction (HRI), pp. 659-668, 2020.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. In Mcllraith, S. A. and Weinberger, K. Q. (eds.), Proceedings of the Thirty-Seconel AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Ap- plications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3207-3214. AAAI Press, 2018. URL https: / www www.iaaiaa...o..... orggi..rg//ooo hctp///c////// /idex. php/ AAAT/AAAAAAAAAA www.aa/////appeeeeeee/aaaaa/a/// 6669.
Chen, L., Paleja, R., and Gombolay, M. Learning from sub- optimal demonstration via self self-super reward regres- sion. In Conference robot learning, pp. 1262-1277. PMLR, 2021.
Chen, L., Jayanthi, S., Paleja, R. R., Martin, D., Zakharov, V., and Gombolay, M. Fast lifelong adaptive inverse rein- forcement learning from demonstrations. In Proceedings of Conference on Robot Learning (CoRL), 2022.
Hoffman, R. R., Shadbolt, N. R., Burton, A. M., and Klein, G. Eliciting knowledge from experts: A methodological analysis. Organizational behavior and human decision processes, 62(2):129- 158, 1995.
Chernova, S. and Thomaz, A. L. Robot Learning from Human Teachers. Morgan & Claypool Publishers, 2014.
Chi, V. B. and Malle, B. F. Interactive human-robot teaching recovers and builds trust, even with imperfect learners. In Proceedings of the 2024 ACM/IEEE International Con- ference on Human- Human-Robot Interaction, pp. 127-136, 2024.
Hussenot, L., Andrychowicz. M., Vincent, D., Dadashi, R., Raichuk, A., Ramos, S., Momchev, N., Girgin, S., Marinier, R., Stafiniak, L.,, et al. Hyperparameter selec- tion for imitation learning. In International Conference on Machine Learming, pp. 45 4-11-2 PMLR, 2021.
Di Stefano, G., Gino, F., Pisano, G., and Staats, B. Learn- ing by thinking: How reflection improves performance. Harvard Business Review, HBS Working Paper, 2014.
Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-1271 2013.
Du, Y., Konyushkova, K., Denil, M., Raju, A., Lan- don, J. Hill, F., de Freitas, N., and Cabi, S. Vision- language models success detectors. ArXiv preprint, abs/2303.07280. 2023, URL
Kojima, T., Gu, S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Ericsson, K. A. and Simon, H. A. Verbal reports as data. Psychological review, 87(3)2215, 1980.
Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Re- ward design with language models. ArXiv preprint, Abs/2303.00000 2023. URL. abs/2303. 00001.
Fan, L., Wang, G., Jiang. Y., Mandlekar, A., Yang, Y., Zhu, H.. Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Informa- tion Processing Systems, 35:18343-18364 2022.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-44 2016.
Lin, J., Du, Y., Watkins, O., Hafner, D., Abbeel, P., Klein, D., and Dragan, A. Learning model the world with language. ArXiv preprint, abs/2308.01 01399, 2023. URL https:
Feldon, D. F. The implications of research on expertise for curriculum and pedagogy. Educational Psychology Review, 19:91-110, 2007.
/at.aa////320 2308. 01399.
Fu, J., Luo, K., and Levine, Learning robust rewards with adverserial inverse reinforcement learning. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://///// http///pwwweeeeeeeeeee/eee id=rkHywl-A-
Locke, E. A. Social foundations of thought and action: A social-cognitive view, 1987.
Ma, Y. J., Kumar, V., Zhang, A., Bastani, O., and Jayaraman, D. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301-23320, PMLR, 2023a.
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar,
Gupta, D., Chandak, Y., Jordan, S., Thomas, P. S., and C da Silva, B. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems, 36, 2024.
A. Eureka: Human-level reward design via coding large language models. ArXiv preprint, abs/2310.12931. 2023b. URL wwt.tttpps//// / /www/vivvv.//////////////////aa. http//rxx.....bbbaa.a.. http:/aab/231.11999.

---

ELEMENTAL
Mahadevan, K., Chien, J., Brown, N., Xu, Z.,, Parada, C., Xia, F., Zeng, A., Takayama, I.., and Sadigh, D. Gen- erative expressive robot behaviors using large language models. ArXiv preprint, 0os/24001.16633, 2024. URL https: / aaaiiiii.. www..v......aa..eeeeoooeee wwo///gg//ss... 2401 14373.
Ravichandar, H., Polydoros, A. S., Chernova, S., and Billard, A. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3, 2020.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction to no-regrel online learning. In Proceedings of the International Confer- ence on Artificial Intelligence and Statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. Isaac gym: High performance gpu-based physies simulation for robot learning. ArXiv preprint, abs/2108. 10470, 2021. URL ptttps://arxivy org/ 0bss2110.100
Sontakke, S., Zhang, J., Arnold, S., Pertsch, K., Biylk, E., Sadigh, D., Finn, C., and Itti, L. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36, 2024.
Matignon, L., Laurent, G. J., and Le Fort-Piat, N. Reward function and initial values: Better choices for acceler- ated goal-directed reinforcement learning. In Artificial Neural Networks - ICANN 2006, pp. 840-849, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
Suay, H. B., Brys, T., Taylor, M. E., and Chernova, S. Learn- ing from demonstration for shaping through inverse rein- forcement learning. In AAMAS, pp. 429-437, 2016.
Myers, V., Zheng, B. C., Mees, O., Levine, S., and Fang, K. Policy adaptation via language optimization: De- composing tasks for few-shot imitation. ArXiv preprint, (bs/2408.16228,, 2024. URL https::///// /ttp//axivvv.////// org/ abs/2408.0 16228.
Tai, L., Paolo, G., and Liu, M. Virtual-to-real deep rein- forcement learning: Continuous control of mobile robots for mapless navigation. 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 31-36. IEEE, 2017.
16228.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: universal visual representation for robot manipulation. ArXiv preprint, abs/2203.1 2022. URL https:// /ttss////xvvvvvv/ wwtp:vvvvorrraaeooo htrrg//bb/22033 12601.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. ArXiv preprint, 13971 2023. URL https:tpp//....... / arxiv, .rrg abs 1397.1
Ng. A. Y. and Russell, S. Algorithms for inverse rein- forcement learning. In Langley, P. (ed.), Proceedings of the Seventeenth International Conference Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29- July 2, 2000, pp. 663-670. Morgan Kaufmann, 2000.
Wang, Y., Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., and Erickson, Z. RI-vIm-f: Reinforcement learn- ing from vision language foundation model feedback. ArXiv preprint, abs/2402.03681, 2024. URL https: / //arxiv. wrw. wtwpg//bab///a./// 03681.
Nisbett, R. E. and Wilson, T. D. Telling more than we can know: Verbal reports on mental processes. Psychological review, 84(3):231, 1977.
Wang, Y.-J., Zhang, B., Chen, J., and Sreenath, K. Prompt robot to walk with large language models. ArXiv preprint, abs/2309. 009669,99999.99 2023. URL https:/ http///axvvvvo.// 012/2239..0999
Peng, A., Bobu, A., Li, B. 7., Sumers, T. R., Sucholutsky, I. Kumar, N., Griffiths, T. L., and Shah, J. A. Preference- conditioned language-guidees abstraction. ArXiv preprint, fas/2400.0308// 2024a. URL https: //axxiv.///ivv///a org/ abbs/402..0 03081.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., et al. Emergent abilities of large language models. ArXiv preprint, wtt.s//22..00622... 2022. URL https: aaaaaaiivv..oo wttpaaaww....oaaaa/a/o//aaa/aaea /200 07682.
Peng, A., Li, B. Z.. Sucholutsky, I., Kumar, N., Shah, A., Andreas, J., and Bobu, A. Adaptive language-guiie guided abstraction from contrastive explanations. ArXiv preprint, abs/2409.08211 2024b. Utt.httss://arxii
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Gon- zalez Arenas, M., Lewis Chiang, H.-T., Erez, T., Hasen- clever, I.,, Humplik, J.. Ichter, B., Xiao, T., Xu, P., Zeng, A., Zhang, T., Heess, N., Sadigh, D., Tan, J., Tassa, Y., and Xia, F. Language to rewards for robotic skill syn- thesis. ArXiv preprint, abs/2306.08647, 2023. URL https: http::///rrii.orrga/////....
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S. H., and Ridge, P. G. Evaluating a large language model's abil- ity to solve programming exercises from an introductory bioinformatics course. PLOS Computational Biology, 19 10)11011511, 2023.
12

---

ELEMENTAL
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L.., and Farhadi, A. Target-driven visual naviga- tion in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
Zhu, Z. and Hu, H. Robot learning from demonstra- tion in robotic assembly: A survey. Robotics, 7(2), 2018. ISSN 2218-6581. doi: 10.3390rrobotics700001..
URL https:/ / www. mdpi www.//1/-/// 6581 /7/ 2/17.
Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, Carnegie Mellon University, 2010.
Ziebart, B. D., Maas, A., Bagnell, A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the National Conference on Artificial in- telligence (AAAI), pp. 1433-1438, 2008.
13

---

ELEMENTAL
A. User Study Details
To evaluate ELEMENTAL in practical, real-world settings, we conducted a user study. This section provides additional details on the task setup and experimental design.
A.1. Task and Robot Setup
The study centers around salad mixing task, where participants teach Kinova JACO robotic arm sequentially transport ingredients and perform mixing motion. To enable targeted evaluation within limited study duration, we decomposed the task into modular falls-thrre of which were taught by participants, while others were predefined.
Taught skills (via ELEMENTAL and EUREKA):
1. Go to mushroom: Navigate to the mushroom ingredient.
2. Go to mixture bowl: Move the bowl location in prepara- tion to drop ingredients.
3. Mix bowl with spoon: Stir the bowl contents using spoon.
Predefined skills:
1. Go to pepper, Go to tomato, Go to spoon, Go to home
Figure 4. Real-world salad-m dalad-mxxing user study setup. Users taught three skills: go grasp mushroom, go drop mixture bowl, and mix bowl with spoon.
2. Pick Drop: Grasp or release ingredients utensils.
The robot was controlled via low-level joint position commands. Visual demonstrations were automatically extracted from ZED stereo camera footage as ten equally spaced keyframes per skill, requiring manual annotation, This setup enabled scalable and consistent demonstration capture across participants.
A.2. Experiment Design
We used a within-subject design where each of the twelve participants taught the three skills using both ELEMENTAL and EUREKA, with method order randomized. Each teaching session consisted of two interaction rounds per method: participants first provided a kinesthetic demonstration and textual intent description, then observed a robot rollout and gave natural language feedback. The learning algorithm used this input to refine the reward and policy in EUREKA, or the feature representation, reward, and policy in ELEMENTAL. This ddssevvtioon-eeddbeee loop was repeated twice for each skill and each method.
After completing all skill teachings, participants watched blind rollouts of the full salad mixing task using both systems. Each rollout followed the full sequence of ingredient pick-and-drop and bowl mixing using both learned and predefined skills. Participants then completed final survey evaluating both methods two dimensions: task performance (e.g.. successful execution of actions) and strategy alignment (e.g., matching user preferences). Each criterion was rated via four 7-point Likert scale questions (range 4-28). ELEMENTAL significantly outperformed EUREKA on both metrics, as shown in Figure 5.
A.3. System Responsiveness
Despite relying on IRL and VLM-based reward learning, ELEMENTAL was tuned to support interactive teaching. Each learning round-ineed innluddinuiiig training and policy/reward refinement was completed in under 4 minutes using a remote server with an NVIDIA A40 GPU. The same hyperparameters were used across simulation and real-world settings, demonstrating ELEMENTAL's sample efficiency and robustness without task-specific tuning.
14

---

ELEMENTAL
A.4. Example in Reducing Language Ambiguity through Demonstrations
key motivation for ELEMENTAL is that visual demonstrations can reduce ambiguity underspecified natural language instructions by grounding them in observable behavior. In our study, participants frequently used vague spatial or temporal terms when describing tasks. For instance, when teaching the mix bowl with spoon skill, one participant instructed: "First, the robot should lower its gripper toward the inside of the bowl with the spoon pointing downward. Then, the robot should move in a way to make the spoon move in circular motion for mixing." This instruction includes vague temporal phrases and spatial instructions.
EUREKA, which relies solely on language, interpreted this instruction by constructing a reward function that encourages a fixed orientation using static dot-product term, shown in Box 1.
Box I: EUREKA REWARD FUNCTION.
def compute_rewarre (ee_pos: torch. Tensor, Tuple[torch. [torch. Tensor, Dict[str, torch, Tensor]
4 MorrhhTenson # other contents omitted
Orientation reward
ie_orientation = ee_pos [:, 3:7]
dot dot_product torch.abs (torch. sum (ee_orientation * desired_ ~ dim=-1))
orientation_rewart torch. borchheegpprriinttttto.eerrrrrrr (dot_product.
1))
In contrast, ELEMENTAL incorporates visual demonstrations to disambiguate when and how the robot should reorient (Box 2). Based on the demonstration, infers that the robot should only begin reorienting when far from the bowl, according the "first... then..." temporal structure. This feature shows that the VLM correctly inferred that reorientation should occur only when the robot is far from the bowl, capturing the "first... then..." structure described in the language.
Box 2: ELEMENTAL FEATURE FUNCTION.
def compute_feate (obs_buf; torch. Tensor) Dict [str, torch, Tensor)
# other contents omitted
# 3. Reorient while distant to avoid collision
down_direction = torch. ([0.0, 0.0, -1.0], device=obs_buf. device) brientatioon_siiiaaiity...
torch. www..iica..cctticalcccco cww.eaneiioooa...iieeeeeieeiiiei www.nnosssmmmmlmiiteee...... (ee (ee_orientation[:: :3],
down_direction. uns.uueeeeeuuuuue (0), dim=-1)
iar so_taacee_t__boooe
to >= 0.2
reorientation_eart berch.whers fis_far, peteetattioossillaiiitt...
torch. tensor (0.0,
This comparison demonstrates how ELEMENTAL reduces ambiguity by grounding vague or implicit instructions. Whereas EUREKA encodes the user's description constant reward over orientation, ELEMENTAL uses demonstration context to condition behavior on spatial distance-capturingg capturing temporal dependencies not specified explicitly in language.
Such distinctions are not just academic: participants frequently expressed uncertainty when relying on language alone. One participant asked, "Should describe this (for example, moving to the left/right side) from my perspective or the robot's ss? This highlights the challenge of specifying tasks precisely through language. ELEMENTAL mitigates this challenge by aligning demonstrations with language, enabling more robust and intent-aligned reward inference.
15

---

ELEMENTAL
30
30
25
Strategy Score (4-28)
Score (4-28)
20
15
15
10
10
Eureka
ELEMENTAL
ELEMENTAL
Algorithm
Algorithm
(a) Comparison of task scores between Eureka and ELEMEN- TAL. Participants rated how well the robot completed the overall task after watching the full execution sequence. MENTAL achieved significantly higher task scores.
(b) Comparison of strategy scores between Eureka and ELE- MENTAL. Participants rated how closely the robot's execu- tion matched their intended strategies. ELEMENTAL outper- formed Eureka with statistical significance.
Figure 5. Participant ratings from the final evaluation phase. ELEMENTAL significantly outperformed Eureka both task success and strategy alignment. Error bars denote interquartile ranges; stars indicate statistical significance (*** p < .001).
B. Prompts
We show the prompts used in ELEMENTAL for feature drafting and self-reflection in Box 3-6.
Box 3: INITIAL SYSTEM PROMPT.
You are feature engineer trying to write relevant features for the reward (inverse reinforcement learning) tasks as effective as possible.
function to solve
Your goal is to write a feature function for the environment that will help
the agent construct a linear reward function with the constructed features via inverse reinforcement learning to accomplish the task described in text and the demonstration.
Your feature function should use useful variables from the environment as + inputs. The feature function signature must follow: @torch. jit.script def compute_feature
torch. Tensor)) Dict [str,
return
Since the feature function will be decorated with étorch. jit script, please make sure that the code is compatible with TorchScript (e.g., use torch + tensor instead of numpy array).
You should wrap the function within class.
Make sure any tensor variable you introduce is the same device the input tensors.
16

---

ELEMENTAL
The code output should be formatted as a python code string: """python Box 4: INITIAL USER PROMPT.
The Python envi ronment is (task_obs_codeesttrin))
Write a feature function for the following task: [task_description).
Three keyframes of a demonstration for how to accomplish the task are shown in the image (superimposing agent pos).
Box 5: CODE OUTPUT INSTRUCTION.
The input of the feature function is torch. Tensor named "obs_buf" that is batched state (shape: [batch, pum_obs]).
The output of the feature function should be dictionary where the keys - are the names of the features and the values are the corresponding + feature values for the input state.
You must respect the function signature.
Some helpful tips for writing the feature function code:
(1) You may find it helpful to normalize the features fixed range by applying transformations
(2) The feature code's input variables must be obs_buf: torch. Tensor, which corresponds to the state observation
returned by the environment compute_obsere computee_oservvtiionn)) function. Under no circumstance can you introduce new input variables.
(3) Each output feature should only single dimension (shape: [batch]).
(4) You should think step-by-steph first, think what is important in the task based on the task description and the demonstration and come up with names the features, then, write code to calculate each feature
(5) You should be aware that the downstream inverse reinforcement
learning only creates reward functions that are linear function of the constr features; thus, it is important to construct expressive features that humans do care in this task
(6) use unicode anywhere such \u03c0 (pi)
17

---

ELEMENTAL
Box 6: SELF-REFLECTION PROMPT.
We trained reward and policy via inverse reinforcement learning using the provided feature function code with the demonstration.
We tracked the feature values as well as episode lengths.
The mean values the last avval_ave_horiaee steps from the learned 4 policy are: (insert)
Please carefully analyze the feedback and provide a new, improved feature function that can better solve the task. Some helpful tips for analyzing the feedback:
(1) If the episode lengths are low, it likely means the policy is unsuccessful
(2) If the feature counts are significants di different between demo and learned behavior, then this means IRL cannot match this feature with the demo as it is written. You may consider
(a) Change its scale
(b) Re-writing the feature: check error in the feature computation (e.g., indexing the observation vector) and be careful about outlier values that may occur in the computation
(c) Discarding the feature
(3) If feature has near-zero weight, the feature may be unimportant. You consider discarding the feature or rewriting it.
(4) You may add/ remove features as see appropriate.
Please analyze each existing features in the suggested manner above first, and then write the feature function code.
18

---

ELEMENTAL
Table 4. This table shows benchmarking results on IsaacCyym tasks between HLEMENTAL. and Fureka. Fach cell reports the mean (standard deviation) across five seeds. The statistical significance shows if p < .05 and ** if < .01. The statistical test (p) shows the p-value for independent t-test (one-sided) for comparisons that satisfies the normality and homoscedasticity assumptions, and otherwise shows the p-values for Mann- Wainnyy U-test (denoted with f). Bold denotes the better performance between ELEMENTAL and Eureka. Additionally, we added a baseline, VLM+BC, which utilizes the VLM drafted feature functions to train Behavior Cloning (BC) by transforming all observations into the feature space. We also include preliminary results of ELEMENTAL and Bureka with OpenAl ol model with 1-5 seed experiments done to show ablation on the VLM used.
IsaneGym Envinaments
Method
Ball Balance
Franka
Allegro Hand
Shadow Hand
Caripolo
Quadeopter
Cabinet
Ant
Humanoid
Anymal
EUREKA ELEMENTAL (Ours)
192. (26.17) 17)
412.16 (33 20) 438.90 (43.64)
-0.27 (0.10)
0.10 (0.14)
4.09 (2.26)
3.69 (0.56)
-2.14 (1.59)
2.36 (4.91)
0.00 (0.00)
258.50 (42.67)
-0.44 (0.15)
0.18 (0.17)
7,00 (1.03)
4.70 (0.12)
-0.37 (0.34)
20.54 (3.98)
1.57 (1.35)
Statistical significance
Statistical test (p)
0.009
0
0,970
0.265
0.015
0.002
0.021
0.004!
0.004
VLM+BC
80.95 (17, 19)
68.77 (17.01)
-1 -1.44 (0.12)
0.00 (0.00)
0.01 (0.09)
0.08 (0.23)
-2.5 (0.27)
0.00 (0.00)
0.01 (0.01)
EUREKA ol
143.49 (24.70)
434,14 (12.99)
.0.26 (0.03)
0.02 (0.00)
6.91 (0.42)
(1.19)
-1.78 (1.59)
8.43 (2.46)
2.14 (0.91)
OL
264.92 (69.70)
456.51 (26. 16)
-0.77 (0.15)
(0.01)
7.00 (1.05)
4.52 (0.33)
-0.54 (0.05)
21.62 (0.73)
2.91 (1.16)
Table This table shows benchmarking results on IsaacGym tasks between ELEMENTAL and Eureka. Each cell reports the max across five seeds. Bold denotes the better performance.
IsaacGym Environments
Method
Ball Balance
Franka Cabinet
Allegro Hand
Shadow Hand
Cartpole
Quadcopter
Ant
Humanoid
Anymal
EUREKA ELEMENTAL (Ours)
215.91 309.38
454,18 474.63
-0.18 -0.30
0.34
6.88
4.36
-0.70 -0.02
11.12 23.38
0.00
0.36
8.49
4.81
2.71
C. Additional Simulated Results
In this section, we present expanded experimental results that provide deeper insight into the performance of ELEMENTAL relative to baselines across benchmark and generalization tasks. These results are computed across five random seeds with statistical tests to assess the significance of the observed improvements.
Benchmark Performance (Table 4 and 5). Table 4 reports the mean and standard deviation of task performance across five seeds on the nine IsaacGym environments. ELEMENTAL achieves better performance than EUREKA on eight out of nine tasks with statistically significant improvements (p- < .05 or p < .01) in five tasks. Table 5 further confirms these gains by showing that ELEMENTAL reaches higher maximum performance than EUREKA in eight out of nine tasks. We also include results from new VLM+BC baseline, which trains behavior cloning policy using features extracted by VLMs. This baseline performs poorly across all tasks, underscoring that simply combining VLM-generated features with demonstrations is insufficient- suffers from covariate shift and lacks ELEMENTAL'S self-reflection loop, which iteratively refines the feature function.
Generalization to Variants (Table 6 and 7). We also assess the generalization capabilities of ELLEMENTAL. by evaluating performance on four Ant-environment variants. Table 6 shows that ELEMENTAL outperforms EUREKA in all four variants, achieving statistical significance in two of them (p < .05). The consistent improvement across these perturbed settings demonstrates that ELEMENTAL is better equipped to handle out-off-istrrbbun alignment between learned rewards and demonstrations enabled by IRI. Table 7 confirms that ELEMENTAL attains superior max performance across all variants.
environments, due in part to the tighter
Reward Function Quality (Table 8 and 9). Tables 8 and 9 compare the reward correlation between learned reward functions and the ground-truth rewards provided in IsaacGym. ELEMENTAL yields better or comparable correlation in all tasks. These results support the claim that ELEMENTAL produces reward functions that are more consistently aligned with task objectives due to its iterative self-reflection loop and embedded IRI. that improve feature and reward alignment.
19

---

ELEMENTAL
Table This table compares the generalization performance of FLEEMENTAL and EUREKA Ant-variant environments. Fach cell reports the mean (standard deviation) across five seeds. The statistical significance shows if < .05 and ** if < .01. The statistical test (p) shows p-value independent t-test (one-sided) for comparisons that satisfies the normality and homoseedasticity assumptions. and otherwise shows the p-values for Mann-Wiiee Whitney U-test (denoted with f). Bold denotes the better performance.
Method
Ant Original
Reversed Ohs
w/ Lighter Gravity
Ant Running Backward
EUREKA ELEMENTAL
4.09 (2.26)
2.56 (2.93)
2.64 (1.04)
3.94 (2.76)
7.00 (1.03)
5.71 (3.23)
3.77 (1.68)
7.41 (1.25)
Statistical significance Statistical test (p)
0.015
0.072
0.119
0.017
Table 7. This table compares the generalization performance of ELEMENTAL and EUREKA on Antivariant environments. Each cell reports the max across five seeds. Bold denotes the better performance.
Method
Ant Original
w/ Reversed Obs
w/ Lighter Gravity
Ant Running Backward
EUREKA
6.88
5.96
4.39
6.90
ELEMENTAL
8.49
8.47
5,89
9.30
Ablation: VLM Variants. We also report results using OpenAl's ol model, in addition to the GPT-4o-bar boseb results in Table 4. ELEMENTAL continues to outperform EUREKA when both use the ol model (seven out of nine tasks, on average 37% gain), suggesting that ELEMENTAL's architecture is robust to the specific VLM backbone and benefits from the integration of demonstrations and iterative refinement regardless of the underlying model.
C.1. Case Study
In this subsection, we present a case study illustrating the iterative process of ELEMENTAL on the Humanoid task. The initial feature function drafted by the Vision-Language Model (VLM) is shown in Box 7. The proposed features-foree veroeit_velocity. uprightness, and heading Beadiingalignmenttto balance and direction. These features provide strong starting point for the learning process.
well-aligned with the task objectives of running efficiently while maintaining
Using this initial feature function, ELEMENTAL trains the IRL process, calculates the feature counts for both the generated trajectories and the demonstration, and feeds this feedback back to the VLM, shown in Box 8. The feedback reveals key discrepancies, such as lower forward velocity. Based on this analysis, the VLM revises the feature function, as shown in Box Notably, the revised function introduces a new feature, lateral velocity, which captures stride consistency by taking the absolute value of the lateral movement. This demonstrates the VLM's capability to construct nonlinear features, expanding the expressiveness the feature function.
Finally, ELEMENTAL trains the IRL process again using the updated feature function and compiles the feedback, as shown in Box 10. The resulting reward weights assign the highest importance to forward. forcird.velocitt, with relatively minor contributions from stability objectives such as uprightness and leloril. This distribution aligns well with human intuition for the task, where speed is the primary objective, and stability features serve as secondary constraints. The evolvement the feature function illustrated Figure 6.
Feature Function #2
Feature Function #1
Reflection
1. forward _elocity (adjust norm.): :): 0.549
1. forward_velocity 0.693
2. uprightness (adjust norm.): 0.253
uprightness: -0.107
3. alagnmnga (adjust norm.): -0.079
3.
-0.201
4. lateral_velocity: 0.119
Figure 6. This figure illustrates the feature evolution in example.
This case study highlights the potential of ELEMENTAL to produce models at every stage of its pipeline. The feature functions generated by the VLM are human-readable and meaningful, allowing practitioners to inspect and refine them as needed. Additionally, the linear weights learned by ELEMENTAL during the IRL process indicate the relative importance
20

---

ELEMENTAL
Table This table shows reward correlation of inferred reward functions for EUREKA and ELEMENTAI. (ours) IsaacCym tasks. Each cell reports the mean (standard deviation) across five seeds.
Isaceyym Environments
Method
Caripole
Bail
Quadcupler
Franka
Humanoid
Anymal
Allegro Hand
Shadow
Balance
Cabinet
Iland
EUREKA
0.77 (0.09)
0.48 (0.61)
0.94 (0.02)
0.49 (0.36)
0.97 (0.04)
0.84 (0.17)
0.32 (0,70)
0.32 (0.40)
0.00 (0.00)
ELEMENTAL (Ouns)
(0.15)
(0.16)
(0.64)
(0.33)
(0.17)
(0.11)
(0.08)
(0.20)
(0.42)
Table 9. This table shows reward correlation of inferred reward functions for BUREKA and ELEMENTAL (ours) on IsaacGym tasks. Each cell reports max across five seeds.
IsaacGym Environments
Method
Ball Balance
Franka Cabinet
Allegro Hand
Shadow Hand
Cartpole
Quadcopter
Ant
Humanoid
Anymal
EUREKA ELEMENTAL (Ours)
0.85
0.95
0.97
0.93
1.00
0.98
0.99 1.00
0.77
0.00
0.99
0.95
0.89
0.98
1.00
0.98
0.77
0.82
of each feature in matching the demonstration. The iterative self-reflection mechanism ensures that the learned policies and reward functions evolve in alignment with the demonstration.
21

---

ELEMENTAL
Box 7: 1ST ROUND VLM RESULT
create Function simed ranning speed ensuring forward velooity, belance topriggt gig ness,, and periapss a conponent that ensures the hunanoid ne inaina forward momentum xithout falling.
hunanoid
pessible, teatures
stability.
Bere's fonction based on
structuro:
"'pyhhn
import
typing ixport
ftarch.fit.sec. script
(obs (obs_ torch, Tensor) Diet letr, Corch.Tenson Extract relevant featurea from the abservacion
Aesuming ess_zut shapc: [batch_size,, nun[obs]
Feature: forward valocity (x direction lccal torwars_relosity obs_Euf 1]
Feacure: uprightaess (ase the profection)
uprightners eas_acf
Feacure: haading clienment (ese the heading profectior) toading_elignnen cac_butli, 11]
Normalize features
Aesuming value normalization suth as the velocity rorge, uprightness [-1, 1], acading aligmment [-1, 11
www_ooowadd_vlacllee 10.0 This is arbitrary cornalization forwarivvelozity fceward_velocitt max_fornard_volccitty
ferward
Coobine features into octput dictionasy features
"Eorvard_wwloctyss forvard_velociyyy
"uprisheneass:: uprightness, "bsaddng__ligmment" hoomanngallommen,
sotern teateres
Explanation:
"foonard *o: The observaticn baffer's secord column ("oba_bofls, 117) captures the foraard volcoity in the local Trams
"uupprghtnesss've observation buffer's 11th colunn ("aba_bud[l, 101") captures upright projaction, us anial to ensuce the humanoid las!, falling over.
se'beading igmment
observation buffer's 12ch ('oba_huf(s, 111)) captures heading igmment, which snaueM the hunanuid noving forward coszect direction.
These Feacures will wnable the inverse reinforcenent learning construct rexard funct ion that emphaxizes cunning while maintaining belanse atsaight heading.
22

---

ELEMENTAL
Box 8: 1ST ROUND FEEDBACK
Coloo lat features the denonacrat ion resultx the fol lowingg
gpiaude_lengiunn 1030.3
focward_velocity. upr inhhreees:: 0.94
aaddiingaliggnnen 0.91
trained rexard and policy inverse roinfereament learning grevided feature function code demonxtiral
We Lracked the festure values as well as spisode lengtha. The mear. values of the last 100 steps from the learned policy are: fnswerd_velocitty 0,4
aprichtraas: 0.35
heading al alingeent: 0.99
IRL reward: 0,05
apiaodo_leng:ha: 691.47
feature weigght: ('forward velonity's y 0,693, spprightnesst -0.107, "heading sliggmentt' -0,201)
Please carefully analyae the feedback and provide new, inproved fealucH funct ion Lhat can beitez sulve the Lask, Sune eo telpful cipa for analysing the teedback:
egisode lengtha
low, it likely
policy unsucoesaful
12) II the feeture uounts are
different beiveen demu Searned behavior, then this means camoot metch
4 this teature with the demo as it is written, You may conside:
Change scale
check + careful about outlier values that may occur the computation
curputation le.g., indexing obseevattion
(c) biscarding the festure
(31 If Teatuze has
miight, the feature be uningustant. You can consides discerding Use Dealuee
rexriting
141 nay adé/remove festures es you see approgriate.
2leace analyze each existing teatures in the suggested nanaer shove tirst, and then write the reacure runction code, The input the festure function is torah. naned "ohe but" that is batched state (shape: batch, obell. dictionary
outpur
keys
corresponding teature values zor the input state.
You must respect the fanction signature.
The code output should be formatted
python code string:
"tyyhhonn
Some helgful tips for writing the feature function code:
[1] You nay find it heiptul to normalize the foatures to fixed range applying transtormations
121
variables nex input variables.
code's
corpete Porch.Tenses. which
observation
can you
13: Each output reature should only one a single dimension tshape: [batch]1.
14: You should think step-by-stegg first, think what is inportant in the task based she taak description and the demonstration and cane with names of seatures, then,
to calculate teature
151 You should se aware that the courstream inverse reintorcement learning only creates revard tunctions tast are linear
- function of the constructed fearures] thus, it inportant zonstruct expressiva foatures that hunans do care in this tack not
anywhere \o03cd (pi)
23

---

ELEMENTAL
Box 9: 2ND ROUND VLM RESULT
Based or the provided tocdback and analycis of hox the cezeent restures axe pertorinng, we can observe few key pointo:
.xFowwadd Velocity:ee
the demonscration. sugezsta that cuzzent method *y calculating or normalise ixing this feacuze night need ad adjustneat,
Ihe policy schicving lowez forvard velocity
The forward velocity weight is quize significant ("0.693").
.:lpzightnesaa:
The policy has aigner uprightaess than the demonstration, The veight for uprisacness is necative ("-0,107",, which may ( indicate that the model learns to be overly caul ioua about alaying uptight, polencially the excenis of noving fast. need rethink how this feature is inflvancing reward.
negative weight
3. relleading Aliguneni:ee tve
The pollcy has highez heading aliennent than the denonatration. similar to uprichineas, the veight also negative relat ively signt ficant ("-0.201),, indicating might overephhasiaing this
diver. chase points, ve sin to improve the feacure function by:
Adjasting the sealing nornalixet for of the Forward velacity.
Revisi ing polent inlll cevising uprightness leading aligmnent fealures bester match demosttattin
charactariacied
3. Considering new or additional features that night ancourage the dexi behavior such as stabil Hyy or at stride consistency.
ixproved
fonctions
""pychon import torch
from typing iagort
def feeture buff: Larch, Tensur) > Biet Jatr, Sorch.Pessor]] from the abservation butfer Aesuming bof shapes [bstch sise, nun_obs]
relevant
Teature: torward velocity (x direction in local velocity obs_bofls,,
Adjust scaling of torxard velocity to enphasize higher speess, aesuning range [-1, nex_ forward velocity = 1.0 New wialing factor to acourately ref eat achievable sgeeds pas__orreed_ Hosmalizing forward velooity
forvard_velosily forward_.eloo..ly
Yeacure: uprightness (use the projection)
uprightness oas_aut
Te Re-nornalize uprightness to give nore incent to upright but not overly consevvacive uprightness (uprightnese - 0.51 + Asjusting the range to new scaling [-1, 1]
Teacure: heading eligmennt heading hooding__lignment aligmeent cac_butli, 131
beading projection)
Mornalize heading aligament
aliganent direction
heading_alignnent alignnent (neading_aliggee
Adjusting range similariy
Introducing new toature: lateral volocity Ito cacure stability) Lateral_ velosity bes_rutli, lateral velocity
velocity) (enoott.
aigg/sabbl aidewsy
Normllize lateral velcaity, supposing it ranges [-1, 1]
max_lators__vooeitt Latoral_yelocity max_latrra__vooociee isteral_veiozity
Combine features into octput dictionary tostures
"torward_veloccty"" berward_velocityy
"uprimheness";: uprightaess,
"heacing_elionne aligimentt alsgmen,,
"Istezal velocity: iateral_veloctty
rerern foatures
#f# Explanation:
foorardd veleotty Adjusted sealing achsevable velocities impactrul.
"pprightness""e: Adjusted
normalization to try so balance upeight positioning wichout naking the
overly cautisus.
"* "heading alligagaalg Similar adjustnent for nornalization.
Iataral_veicctty"... Added
sideways
stability atswigat notion.
Thane changes ain to provide a nore nuanced reward fanction, hopefully promoting behaviors closer to the denonatration while balzncing speed and stability.
24

---

ELEMENTAL
Box 10: 2ND ROUND FEEDBACK
Caloclat ing neen features the demorssracion results the fol lowing:
episose__ergee lengths: 1000.3
corward velocity: 5.57
uprichaness: 0.89
maading_alignmeent 0.03
lstersl velocity: 2,31
inverse roinfereement learring previded runction codo tracked the feature values as well as spisode lengths. The mzar. values of the last 100 steps from the learned policy are: corwerd_veloctty: uprichinass: 0.65
rexord
demonotration.
hesding_aiigmment: 0.92
eateral_velocttg 2,05
3.54
episose_lergths: 932.42
IRG teature weighte: ('terward_veloosty''s 0,548, 'uprigatnees's 0,253, 'heedinggaligmment'' -0.079, "Iateral Lvelocity'; + 0.119)
Pisass carefully
teesback provide improved resture function helptul cips for anslyzing feedback:
solve Some
[1] If the cpisade lsnoths are low, it likely neans the policy is unsuzessstul.
121 It the teature ocunts are cigriticantly differeat between demo and Learned behevior, then this mocns 185 cannot motch is written.
this
consides
Change scals
(b) Ro-wisting the testure: check error in the teature computation le.gu indexiag she observation vectorl and be
b careful about outller valuas that occur the conputation
Discarding the feature
131 It teature near-zero Neight, the teature may be unimpostant, can ciscarding reature
b zewriting it.
101 You nay add/rcmove festures you see appropriate. existing feature function is tozch. naned "ebs_buf"
the auggested
and weite state (shase: (batch, pur_sbs]].
code. The impus
The outpu: of the fecture Function should be dictionaty whare the keys are the nancs of the fectures and the values are the oq coriwaponding featare values for the input scate.
output
formatted python string: """pythaa ...
Bome helpful tips for writing the feature function code:
nay it helpful nomnelise Seatures
range applying Leansfornel acate
obs tozch, Tensoz, which cocrasponds " [self.obs_buf] seturned by the envirennent sonotteubborrvatioos) functicn. Under no circunstance can you introduce varialles.
[2] feature codets lsput variables
Each feature should
dinensicn
[batch]).
141 should thisk step-by-stap: flest,
shat
the task based
task dascripticn
< denossiaation and come up with names of the Sealires, then, wrils cude to caloulate eadi fealure
[5) You should be aware that the dowrsczean Laverse zeinfozoement leazning only creates revard functions that are linear feasures;
isportant
expressive
this Lank
161 Do not use unicode anywhare auch as \u03eD (oi)
25

---

ELEMENTAL
Hyperparameters
Value
Reward learning rate a
1.0
Approximate MaxEnt-IRL number of iterations m Policy training steps
5
500
Number of algorithm iterations
Number of code generations per iteration
(1 for ShadowHand due longer compile times)
Policy Neural Network Architecture
Fully-connected [32, 32] with ReLU activation
Table 10. Hyperparameters and their values
D. ELEMENTAL Hyerparameters
tune hyperparameters via a grid search. We summarize the ELEMENTAL hyperparameters Table 10, All other hyperparameters follow EUREKA's default setup.
26


Reviews:
[{"review_id": "RCyN37NcjN", "reviewer": "Reviewer_5aJa", "summary": "The paper proposes an approach to inverse reinforcement learning (IRL) that uses the knowledge of a VLM to construct code the computes state features from the environment. These features are then used with MaxEnt IRL, and iteratively refined online to match the demonstration trajectories. Experiments show better performance than past imitation (no language) and reward inference (no demos) methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, and for recognizing that ELEMENTAL presents a novel approach to resolving reward ambiguity in IRL by using VLM knowledge as a semantic prior. In response to the reviewer’s comments, we have added new experiments, expanded statistical analysis, and revised figures to address the concerns raised. All updated tables and figures are included in https://shorturl.at/YHEDU (referred to as Response Table and Response Figure), following ICML rules. We address each point below.\n\n**[Q1 (Issues 1 & 2, Weakness 1, Questions 3 & 4)]** Statistical significance\n\n**[A1]** We thank the reviewer for highlighting the importance of statistical tests. We increased the number of random seeds from 3 to 5 for both ELEMENTAL and Eureka across all benchmark and generalization tasks. We report the mean and standard deviation in Response Table 1 (benchmark) and Response Table 3 (generalization), along with statistical tests. ELEMENTAL performs better in 8/9 benchmark tasks (5/9 statistically significantly, $p < .05$ or $p < .01$) and in 4/4 generalization tasks (2/4 statistically significantly, $p<.05$). Notably, ELEMENTAL achieves a 122.5% average gain across benchmarks and an 81.2% gain in generalization. We also: 1) updated original Tables 1 and 3 (max success across three seeds) to be Response Tables 2 and 4 (across five seeds); 2) updated original Table 2 (max reward correlation across three seeds) to be Response Tables 5 and 6 (mean and max reward correlation across five seeds).\n\nThe original Figure 3 used standard deviations, leading to large shaded areas. We have updated this in Response Figure 1 to report standard errors instead. A two-way repeated measures ANOVA (across nine paired tasks) shows significant main effects for algorithm, $F(1, 8) = 7.00, p = .030$, and round, $F(2, 16) = 10.03, p = .002$; the interaction is not significant, $F(2, 16) = 2.21, p = .144$. This indicates ELEMENTAL achieves statistically significantly higher code execution rates than Eureka. \n\nRegarding whether prompts impact execution rate: we agree prompt design can influence execution rates. However, as detailed in Supplementary Section A, our prompts are developed based on Eureka's and kept as similar as possible. \n\n**[Q2 (Issue 3)]** Baseline combining LfD and VLM-generated feature code\n\n**[A2]** We thank the reviewer for this valuable suggestion. We implemented a VLM+BC baseline that uses the same VLM-generated feature functions as ELEMENTAL, transforms observations into feature space, and trains a BC policy mapping features to actions. As shown in Response Table 1, this baseline performs poorly—more than 50% worse than both ELEMENTAL and Eureka. This highlights that combining demonstrations and language alone is insufficient: BC suffers from covariate shift and lacks ELEMENTAL’s self-reflection loop, which iteratively refines the feature function.\n\nWe agree that exploring more advanced IRL methods (e.g., GAIL or AIRL) in place of Approximate MaxEnt-IRL would be a promising direction, as we noted in Section 6.\n\n**[Q3]** More Related Works\n\n**[A3]** We thank the reviewer for pointing out these relevant works. ELEMENTAL distinguishes itself from prior work by coupled reward inference and VLM-based feature drafting, as well as grounding in both demonstration and textual input.  The referenced papers explore the use of language for modeling environments, reward shaping, or task decomposition, often treating language as a prior for pretraining or few-shot adaptation. In contrast, ELEMENTAL uniquely integrates VLMs into the IRL process by generating executable feature functions from visual-language prompts and iteratively refining them through self-reflection. We will incorporate discussion of these papers in the revised manuscript.\n\n**[Q4 (Weakness 2, Questions 1 & 2)]** Justification and empirical support for Phase 3 (Self-reflection)\n\n**[A4]** We thank the reviewer for these important questions. To clarify: the VLM does not directly penalize feature count discrepancies in Eq. (7). Instead, the discrepancies are provided as feedback, and the VLM interprets them—deciding whether to add, remove, or adjust features to better capture task-relevant behaviors and demonstration preferences.\n\nWe show an empirical evidence in Supplementary C.2 (Humanoid domain):\n\n- The 1st-round feature function (Box 1) included forward_velocity, uprightness, and heading_alignment.\n\n- Feedback (Box 2) showed underperformance in forward_velocity and overly conservative uprightness.\n\n- The VLM revised the feature function (Box 3), adding lateral_velocity and adjusting normalizations.\n\n- The 2nd-round result (Box 4) showed improved alignment: forward_velocity increased, and uprightness decreased.\n\nThis demonstrates that VLM self-reflection correctly improves feature alignment, and Eq. (7) provides information for VLM to revise features based on the comparison.", "claims_and_evidence": "The paper claims that the method outperforms prior imitation, IRL, and reward design with language methods in terms of reward recovery and task success trained on recovered rewards. The results show quantitative comparisons on IsaacGym control environments that are consistent with these claims.\n\nIssues:\n- Most of the results are reported without standard deviation/error or an statistical testing\n- When error bars are included (Figure 3, Table 5), they do not seem to show significant differences between the method and baselines\n- The baseline methods all either utilize the demonstrations or the language, never both. A natural baseline to include for a fair comparison would be some for of learning from demonstrations method (such as GAIL or BC) applied to the VLM-generated feature code.", "methods_and_evaluation_criteria": "Yes, the idea of combining IRL with VLM semantic knowledge is sound, and the environments and baselines are good, with the exception of the issues mentioned above.", "theoretical_claims": "None provided, though the paper would benefit from more theoretical justification for \"phase 3\" (Eq. 7).", "experimental_designs_or_analyses": "- There appear to be no error bars or statistical significance testing, except in Table 5 and Figure 3.\n- It is unclear how the \"successful code execution rate\" per iteration reflects on the two methods. The paper claims that it shows that reward features are better than reward functions, but could this be a function of the prompts used? The figure also appears to have overlapping error bars at each point, making its value questionable.", "relation_to_broader_scientific_literature": "The paper provides a useful synthesis of ideas in IRL and recent advances in using VLM knowledge for decision making.\n\nSome related work that could be discussed as well include existing approaches that have incorporated language in modeling an environment [1,2,3,4] or as a semantic prior for learning from other data [5,6]\n\n### References\n\n[1] Lin, J. et al., 2024. ''Learning to Model the World With Language.'' ICML\n\n[2] Ma, Y. J. et al., 2023. ''LIV: Language-Image Representations and Rewards for Robotic Control.'' ICML\n\n[3] Fan, L. et al., 2022. ''MineDojo: Building Open-Ended Embodied Agents With Internet-Scale Knowledge.'' NeurIPS\n\n[4] Nair, S. et al., 2022. ''R3m: A Universal Visual Representation for Robot Manipulation.'' CoRL\n\n[5] Myers, V. et al., 2024. ''Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation.'' CoRL\n\n[6] Adeniji, A. et al., 2023. ''Language Reward Modulation for Pretraining Reinforcement Learning.'' arXiv:2308.12270", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Reward ambiguity is a major challenge in IRL. The proposed approach is a novel attempt to use VLM knowledge as a semantic prior to resolve this ambiguity.\n\n\nWeaknesses:\n- The overall results (without error bars) don't provide strong support for the claims\n- I was confused by the justification for the \"phase 3\" component", "other_comments_or_suggestions": "- Line 777: Restuls ⇒ Results\n-", "questions_for_authors": "> The two feature count vectors are then fed back to the VLM, which uses the feature count differences to revise the feature function ϕ(s).\n\n- What is the theoretical justification for Eq. (7), which penalizes these differences?\n\n- Is there evidence that the VLM is able to improve these feature count differences (Eq. 7) in \"phase 3\"? The \"self-reflection\" prompt tells it to, and it improves overall performance, but there is no ablation showing that it actual improves the feature discrepancy.\n\n> Experiments\n\n- Why is Table 5 different from table 3?\n- Which results are statistically significant?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZ8PvdlCKB", "reviewer": "Reviewer_tQa4", "summary": "The paper introduces ELEMENTAL, which combines VLMs with Learning from Demonstration (LfD) to address challenges in reward design for robotic tasks. ELEMENTAL leverages visual demonstrations and natural language descriptions to generate task-relevant feature functions, which are optimized through an enhanced Maximum Entropy Inverse Reinforcement Learning (MaxEnt-IRL) algorithm. The framework incorporates a self-reflection mechanism to iteratively refine feature functions, reward functions, and policies, ensuring alignment with user demonstrations. Experimental results on IsaacGym benchmarks show that ELEMENTAL outperforms state-of-the-art methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for highlighting the strengths of our IRL-VLM integration, self-reflection mechanism, and experimental results. In response, we have added real-world user study results, runtime analysis, and experiments using OpenAI o1 model. All updated tables and figures are included in https://shorturl.at/YHEDU (referred to as Response Table and Response Figure), following ICML rules. We address each point below.\n\n**[A1 (Question 3)]** Real-world experiment\n\nWe conducted a within-subject user study and show ELEMENTAL achieves significantly better ratings from user than Eureka. In the study, 12 participants taught a Kinova JACO arm to complete a salad mixing task (illustration shown in Response Figure 2). For user study time consideration, participants were asked to teach three core skills—Go grasp mushroom, Go drop at mixture bowl, and Mix bowl with spoon—while the remaining skills—Go grasp pepper, Go grasp tomato, Go to home—were predefined. At the beginning of the study, we informed participants of the skill set and how the skills would be composed into a final full-task execution during the evaluation phase.\n\nEach skill was taught twice per participant, once using ELEMENTAL and once using Eureka (order randomized). For each skill, after the initial kinesthetic demonstration and a natural language description of intent, participants observed the learned robot policy and provided textual feedback. This observation–feedback cycle was repeated twice per algorithm, consistent with our simulated experiments. After teaching all three skills with both algorithms, participants observed blind executions of the full salad mixing task (using each method’s learned and predefined skills) and rated them using 7-point Likert scales on two criteria:\n\n- Task performance (i.e., whether the robot accomplishes the task)\n- Strategy alignment (i.e., whether the robot’s execution matches user intent/preferences)\n\nEach criterion consisted of four questions, resulting in the summed scores ranging from 4 to 28.\n\nThe user study results showed ELEMENTAL outperformed Eureka significantly:\n\n- Task score: ELEMENTAL $20.58 \\pm 4.93$ vs. Eureka $12.42 \\pm 4.72$, $t(11) = -4.65, p < .001$\n- Strategy score: ELEMENTAL $19.83 \\pm 6.13$ vs. Eureka $10.50 \\pm 4.32$, $t(11) = -4.20, p < .001$\n\nThese results demonstrate ELEMENTAL’s superior alignment with user intent and effectiveness in real-world settings. To achieve interactive real-time user study, both algorithms were tuned to complete each learning round in under 4 minutes by training via IsaacGym-based simulation on servers with NVIDIA A40 GPUs. This also demonstrates ELEMENTAL’s feasibility on real-world, out-of-distribution problems.\n\n**[A2 (Question 1)]** Keyframe and scalability\n\nIn the four locomotion domains, we used temporally equally spaced frames and superimposed them into a single image (automatic). In our simulated manipulation domains, keyframes were selected by experts, though [1] shows that keyframe-selection in robotics tasks is user-friendly. In the real-world user study, we used ten equally spaced frames captured via a ZED camera during each demonstration—an automatic and scalable process.\n\nAcross all settings, ELEMENTAL performs robustly, suggesting low sensitivity to the keyframing method. We agree exploring automated keyframe selection via VLMs is a promising direction for future work.\n\n[1] Akgun, B., Cakmak, M., Jiang, K., & Thomaz, A. L. (2012). Keyframe-based learning from demonstration: Method and evaluation. International Journal of Social Robotics.\n\n**[A3 (Question 2)]** Runtime\n\nAs reported in Section 5.1, with the same policy training environment steps, Eureka averaged 68.2 minutes across the nine tasks, while ELEMENTAL averaged 168.4 minutes. Importantly, our user study demonstrates ELEMENTAL can be deployed interactively in real time, with each learning round completing in under 4 mins. We agree reducing runtime is a valuable future direction possibly via more advanced IRL algorithms, as discussed in Section 6.\n\n**[A4 (Question 4)]** VLM ablation, prompt design, and reporting mean result values\n\nFor mean values, please refer to [A1] of Reviewer 5aJa, where we report results over five seeds with statistical tests, showing that ELEMENTAL significantly outperforms Eureka.\n\nTo study the effect of VLM choice, we include preliminary results using OpenAI’s o1 model in Response Table 1. While full 5-seed runs are ongoing due to time limits, current results show that both ELEMENTAL and Eureka improve with o1, and ELEMENTAL continues to outperform Eureka in 7 out of 9 tasks (on average 37% gain). This suggests that ELEMENTAL’s advantages are robust across some VLM choices. We will update the table once full results are available.\n\nRegarding prompt design, our prompts are developed based on Eureka’s and kept similar (Supplementary Section A), minimizing the likelihood that performance differences arise from prompt tuning.", "claims_and_evidence": "The paper demonstrates the effectiveness and generalization capabilities of ELEMENTAL through experiments and validates its individual components. \n\nHowever, the experiments are conducted solely in the IsaacGym environment. It would be better if additional experiments were performed in other simulation environments to further validate the approach.\n\n Additionally, can ELEMENTAL be deployed in real-world settings? Including real-world experiments would strengthen the paper. \n\nFurthermore, I suggest that the authors add an additional column in Tables 1 and 2 to report the mean values for better clarity.", "methods_and_evaluation_criteria": "This paper primarily utilizes VLM to design rewards and introduces LfD (IRL) to address the issues that VLMs struggle to balance the importance of different features, generalize poorly to out-of-distribution robotic tasks, and cannot properly represent the problem with text-based descriptions alone.\n\nThe evaluation mainly focuses on the rewards in IsaacGym Environments, which is reasonable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments evaluation focuses mainly on nine tasks in IsaacGym. I suggest that the authors validate their approach in additional simulation environments or on more **complex manipulation-related tasks**.\n\nIf the authors could demonstrate its application in the **real-world**, that would be even better.", "relation_to_broader_scientific_literature": "This paper primarily builds upon the previous use of VLMs for reward design by introducing LfD (IRL). In a sense, this requires more supervision, but overall, it leads to better results.", "essential_references_not_discussed": "As far as I know, the authors have discussed the related works.", "other_strengths_and_weaknesses": "Strengths 1: Combining inverse reinforcement learning (IRL) to enhance reward design in visual-language models (VLMs) is both novel and interesting, while also making sense conceptually.\n\nStrengths 2: The experimental results demonstrate that the proposed ELEMENTAL framework is highly effective.\n\nStrengths 3: The techniques used, such as self-reflecting on features, optimizing the reward function, and initial prompt design, are reasonable and convincing, and their effectiveness is validated through experiments.\n\nWeaknesses: The main drawbacks lie in the scalability of the method and the experimental evaluation. For details, please see the \"Questions For Authors\" section.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Question 1: How are the key frames obtained? Can they be derived or parsed using the VLMs? I am concerned that using key frames may **make it difficult for this method to scale up**, especially for complex manipulation-related tasks.\n\nQuestion 2: What is the **runtime** of the ELEMENTAL algorithm, and how efficient are its individual modules compared to other methods? Providing these details would improve the paper.\n\nQuestion 3: It would be better if the authors could provide results or experiments demonstrating deployment in **real-world** scenarios.\n\nQuestion 4: This paper lacks ablation studies on **prompt design and the selection of VLMs**, which would also be very valuable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPjKSXGefh", "reviewer": "Reviewer_g569", "summary": "This paper introduces ELEMENTAL, a framework for reward design in robotics that integrates vision-language models (VLMs) with an inverse reinforcement learning (IRL) backbone. The authors aim to address the shortcomings of purely language-based reward engineering, particularly the difficulty of specifying nuanced features and balancing them properly. Instead, they propose using visual demonstrations and language-based prompts to construct an initial feature function, then iteratively refine that feature function and the learned policy through a self-reflection loop that compares the policy’s behavior (in terms of feature values) to the demonstration. This loop adjusts the reward function so that the final policy better matches the user’s intended behavior. Empirical evaluations on challenging IsaacGym tasks (locomotion and manipulation) show that ELEMENTAL outperforms both standard IRL methods (that lack VLM-powered feature extraction) and prior language-based reward-design approaches like EUREKA.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We are glad that the reviewer found our integration of VLMs with IRL to be a compelling combination and appreciated our empirical comparisons and ablation studies. All updated tables and figures are included in https://shorturl.at/YHEDU (referred to as Response Table and Response Figure), following ICML rules. We respond to the reviewer’s insightful suggestions below: \n\n**[Q1]** Sensitivity to demonstration quality\n\n**[A1]** We thank the reviewer for this important question. We assess ELEMENTAL’s sensitivity to demonstration quality in both simulation and real-world settings:\n\n- High-quality visual demonstrations are informative for VLMs to extract meaningful task semantics, as illustrated in Table 1 *ELEMENTAL w/ random visual demo* condition.\n- In our real-world user study (see Reviewer tQa4 [A1]), demonstrations were provided by human participants—potentially noisy and imperfect compared to RL-generated ones. ELEMENTAL still achieved significantly higher task and strategy scores than Eureka. As one participant noted when teaching the Go to mixture bowl skill:\n*Even if my demonstration was slightly to the left of the mixture bowl, ELEMENTAL can help me fix this when I give it feedback and successfully put ingredients in the mixing bowl.*\nThis highlights ELEMENTAL’s ability to recover from imperfect input by constructing intent-aligned features and optimizing them through IRL.\n\nWe agree that handling low-quality or partial demonstrations is an important future direction. Techniques such as demonstration filtering (e.g., based on user confidence, VLM scoring, or automatic ranking algorithms) or Learning from suboptimal Demonstration methods [1–3] could enhance robustness. We will include these discussions in the future work section.\n\n[1] Brown, D., Goo, W., Nagarajan, P., & Niekum, S. (2019, May). Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning (pp. 783-792). PMLR.\n\n[2] Chen, L., Paleja, R., & Gombolay, M. (2021, October). Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning (pp. 1262-1277). PMLR.\n\n[3] Beliaev, M., Shih, A., Ermon, S., Sadigh, D., & Pedarsani, R. (2022, June). Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning (pp. 1732-1748). PMLR.\n\n**[Q2]** Hyperparameter and design choices sensitivity\n\n**[A2]** We appreciate the reviewer’s concern. We validated our design choices and hyperparameters across nine simulated domains and the real-world salad mixing user study (see Reviewer tQa4 [A1]), which together span diverse robotic settings—locomotion, manipulation, and human-in-the-loop learning. We used the **same ELEMENTAL hyperparameters and design components (e.g., gradient and weight normalization) across all tasks**, demonstrating robustness without per-environment tuning.\n\nThat said, RL and IRL can still be sensitive to hyperparameters—an open challenge in the field [4-6]. ELEMENTAL is the first framework to integrate VLMs with IRL using multimodal inputs, and we agree future work can improve its IRL backend. Our current normalization strategies help stabilize IRL optimization, and more advanced approaches (e.g., AIRL) could further improve robustness. We will include these discussions in the future work section.\n\n[4] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018, April). Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).\n\n[5] Hussenot, L., Andrychowicz, M., Vincent, D., Dadashi, R., Raichuk, A., Ramos, S., ... & Pietquin, O. (2021, July). Hyperparameter selection for imitation learning. In International Conference on Machine Learning (pp. 4511-4522). PMLR.\n\n[6] Adkins, J., Bowling, M., & White, A. (2024). A method for evaluating hyperparameter sensitivity in reinforcement learning. Advances in Neural Information Processing Systems, 37, 124820-124842.", "claims_and_evidence": "The main claim of the paper is that incorporating a VLM and demonstration enables better reward design than a pure LLM based approach like EUREKA. This claim is supported effectively in the paper via various comparisons against EUREKA and ablations of the proposed method itself. For example, Experimental ablations show that when ELEMENTAL is provided only with text demonstrations (or no demonstrations at all), performance drops significantly compared to the default setting with high-quality visual demos.", "methods_and_evaluation_criteria": "Yes, it makes sense. The IsaacGym benchmark used in the paper has been used in prior works (e.g., EUREKA).", "theoretical_claims": "The paper does not introduce theoretical claims.", "experimental_designs_or_analyses": "The paper compares ELEMENTAL to a variety of baselines: standard IRL, behavior cloning (BC), random policies, the ground-truth reward, and a prior language-based reward method (EUREKA). This set is sufficiently comprehensive to showcase where their method sits in terms of performance bounds (random and ground truth) as well as direct competitors in LfD and LLM-based approaches.", "relation_to_broader_scientific_literature": "ELEMENTAL extends decades of IRL research by introducing a language-based mechanism to produce reward features, circumventing the heavy reliance on manually crafted feature representations. At the same time, ELEMENTAL uses a VLM to do reward specification via IRL, which is a less common usage of VLMs for robotics. So overall, I think this paper offers a nice combination of existing ideas.", "essential_references_not_discussed": "Essential related works are discussed and compared in great length.", "other_strengths_and_weaknesses": "The proposed approach has couple weaknesses that I'd like see authors address during the rebuttal. First, I'd like to understand how sensitive ELEMENTAL is to the quality of the provided demonstrations. This is mentioned in the limitation section, but a table that shows the sensitivity would be helpful. An interesting question is whether future versions could incorporate “demonstration filtering” or handle partial demonstrations. Second, ELEMENTAL seems to require a lot of hyperparameter tuning. Some of the approach’s success presumably hinges on certain design decisions (e.g., gradient and weight normalization). One might wonder how robust it is if we alter those normalizations or if a certain environment has drastically different scale.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "My suggestions are stated above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ck1HD0RKeN", "reviewer": "Reviewer_KANf", "summary": "In this paper, the authors propose a framework that combines natural language guidance with visual user demonstration to align robot behavior. Using inverse RL and iterative self-reflection, ELEMENTAL improves task success by 41.3% over previous methods in out-of-distribution tasks.\n\nIn the first stage, features related to the task are inferred through the VLM. In the next stage, a reward function is optimized using the feature functions to match the demonstrations, and do IRL. The final stage, called reflection, iteratively improve the feature functions created in stage one and complete the learning loop.\n\n## Update after rebuttal\nThe authors addressed my questions, and I am keeping my original score, 4:accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation and for recognizing ELEMENTAL’s contribution in automating not only reward design but also feature construction. We are glad the reviewer found the paper clear and the method well-motivated. All updated tables and figures are included in https://shorturl.at/YHEDU (referred to as Response Table and Response Figure), following ICML rules. Below, we address the comments. \n\n**[Q1]** Feature refinement through self-reflection\n\n**[A1]** We agree that visualizing the evolution of the feature set and feature weights would improve reader understanding, and we will add such visualizations to the supplementary material. Below, we show further analysis of the Humanoid case study in Appendix C.2: \n\n- 1st round: The VLM proposes three features—forward_velocity, uprightness, and heading_alignment. The learned policy is overly conservative and slow, achieving low episode lengths and high uprightness and heading alignment.\n\n- 2nd round: The VLM revises the feature function by (1) adjusting normalization of the existing features and (2) introducing a new feature, lateral_velocity, to capture stride consistency and stabilize side-to-side movement.\n\n- Outcome: The revised reward weights assign positive weights to both forward_velocity and lateral_velocity, improving alignment with the demonstration. Heading_alignment receives a smaller weight but still matches the demo, suggesting the overemphasis in the previous round was corrected. Episode length increases from 691 to 932, reflecting the learned reward function is now more aligned with the ground-truth objective.\n\nThis example illustrates how the self-reflection loop enables meaningful revisions to the feature function and their relative importance. During each self-reflection round, ELEMENTAL compares the feature counts from the learned policy against those from the demonstration and feeds this discrepancy to the VLM. The VLM interprets this feedback to revise the feature function: adding missing features, modifying existing ones, or discarding those deemed unhelpful. While the VLM operates as a black box, the output feature code and IRL weights are transparent and human-readable—making it possible to inspect how ELEMENTAL adapts its reward representation over time. \n\n**[Q2]** Flexibility in reward design compared to Eureka\n\n**[A2]** We thank the reviewer for this insightful suggestion. A key advantage of ELEMENTAL over Eureka is its ability to construct richer, context-sensitive features by combining multimodal inputs (language + demonstrations). While Eureka interprets task objectives solely from text, ELEMENTAL leverages visual demonstrations to ground ambiguous or under-specified instructions, leading to more expressive reward features.\n\nFor example, in our real-world user study (see Reviewer tQa4 [A1]), one participant taught the robot to \"mix bowl with spoon\" using the instruction:\n\"First, the robot should lower its gripper toward the inside of the bowl with the spoon pointing downward. Then, the robot should move in a way to make the spoon move in a circular motion for mixing.\"\n\nThis instruction contains temporal dependencies (e.g., “first...then”) and spatial relations that are difficult to resolve with language alone. The reward function from Eureka misses this temporal nuance and encodes the task with static orientation reward:\n```python\ndef compute_reward(ee_pos: torch.Tensor, bowl_position_tensor: torch.Tensor) -> Tuple[torch.Tensor, Dict[str, torch.Tensor]]:\n    # other contents omitted due to space limit\n\n    # Orientation reward\n    ee_orientation = ee_pos[:, 3:7]\n    dot_product = torch.abs(torch.sum(ee_orientation * desired_orientation, dim=-1))\n    orientation_reward = torch.exp(orientation_reward_temp * (dot_product - 1))\n```\n\nIn contrast, ELEMENTAL interprets the demonstration to encode timing and conditional dependencies. It defines a feature that encourages early reorientation only when distant from the bowl:\n```python\ndef compute_feature(obs_buf: torch.Tensor) -> Dict[str, torch.Tensor]:\n    # other contents omitted due to space limit\n    \n    # 3. Reorient while distant to avoid collision\n    down_direction = torch.tensor([0.0, 0.0, -1.0], device=obs_buf.device)\n    orientation_similarity_far = torch.nn.functional.cosine_similarity(ee_orientation[:, :3], down_direction.unsqueeze(0), dim=-1)\n    is_far = distance_to_bowl >= 0.2\n    reorientation_early = torch.where(is_far, orientation_similarity_far, torch.tensor(0.0, device=obs_buf.device))\n```\n\nThis example highlights ELEMENTAL’s greater flexibility in reward design: it constructs temporally-aware and spatially-grounded features by aligning language with visual demonstrations, something difficult to express in language alone. \n\nParticipants in the study often asked, “Should I describe this (for example, moving to the left/right side) from my perspective or the robot’s?”—underscoring the inherent ambiguity in language-only reward design.", "claims_and_evidence": "The framework proposed in this paper appears to be convincing. In particular, the Eureka, which is considered the most similar to this research, has already demonstrated the effectiveness of RL automation using VLM. This study goes a step further by not only automating the reward function but also automating the feature extractor with VLM, showing even more improved results.\n\n\nHowever, regarding the process of improve the feature function based on the feature counts of the trajectories generated by the trained rollouts and the demonstration trajectories, it’s little bit unclear how exactly they changes across the updates. It is understanable since this part leverages black-box VLM, but if the authors could provide more insights of this process with thorough analyzation, it would be helpful for the readers.", "methods_and_evaluation_criteria": "The proposed method, evaluation criteria, and the baselines are considered appropriate.", "theoretical_claims": ".", "experimental_designs_or_analyses": "The experimental design and analysis seems valid.", "relation_to_broader_scientific_literature": "This study represents one direction in the line of research on RL automation based on VLMs/LLMs and is part of the same context as Eureka, which the authors have cited. However, it achieves a higher level of automation compared to Eureka and can be considered a novel pipeline.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "It is considered one of the studies that increases the efficiency in reward function automation, improving over previous research.", "other_comments_or_suggestions": "The paper is well-written and easy to follow. However, I believe readers could gain better insights if the following two points were included:\n- The process described in Appendix C.2., where the VLM autonomously updates the feature function, is helpful for understanding the effectiveness of the proposed pipeline. I suggest the authors visualize how the VLM adds new features over time, and how each feature's contribution to the reward evolves.\n- Provide a comparison showing how much more flexibility in reward design is achieved compared to Eureka.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TK2rLkpqIr", "reviewer": "Reviewer_jnmY", "summary": "The paper proposes incorporating user demonstrations into LLM-based reward design methods in robotics. Their proposed approach is a direct contender of EUREKA. The main motivation of this work is that language can be ambiguous for task requirement specification and hence using user demonstrations is a good way to reduce this ambiguity and create a better interface for human task specifications.\n\nFor this purpose the authors propose ELEMENTAL, a method that leverages demonstrations to achieve this exact goal. The method has 3 different phases. In the first phase, a vision-language model is prompted with demonstrations and text and is expect to produce a feature extraction function. The prompt also includes the environment code in addition to the demonstrations (presented as either superimposed images for locomotion or keyframes for manipulation). The second phase is an inverse RL phase where the agent is expected to learn a reward from the previously extracted features and learn a policy with the obtained reward and using PPO. The final phase allows the agent to self reflect about the quality of the feature extractor so that it can improve it. \n\nFor evaluation, the proposed method is evaluated using a series of IsaacGym environments, and compared to baselines from inverse RL and to Eureka. The results show an improvement in performance in comparison to the baselines and some hints of better generalization. All results are in simulation and using IsaacGym. The authors also ablate multiple design choices from their method and report wall clock time comparisons to Eureka, in which their method is around 2.5 times slower.\n\n\n## Update after rebuttal\n\nThe rebuttal successfully addressed most of my concerns. I have now raised my score to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and for recognizing the value of combining demonstrations with language for reward design, as well as ELEMENTAL’s improved performance. All updated tables and figures are included in https://shorturl.at/YHEDU (referred to as Response Table and Response Figure), following ICML rules. Below, we address the reviewer’s concerns.\n\n**[Q1 (Claim Concern 1, Weakness 4, Supplementary)]** Demonstrations reducing language ambiguity\n\n**[A1]** In our user study (see Reviewer KANf [A2]), a participant taught the “mix bowl with spoon” skill using language that included vague temporal phrases (“first… then…”) and spatial instructions (“lower into the bowl”). While such language hints the intent, it does not fully define the user’s intent. Nonetheless, ELEMENTAL was able to extract temporal-spatial alignment from visual input of the demonstration. \n\nWe argue that while it is possible to articulate complex relationships using language (e.g., with coordinate systems or math), doing so is burdensome and error-prone for users [1]. ELEMENTAL reduces ambiguity by grounding vague or underspecified language in visual demonstration, a more natural way to convey task details. We will add this example to supplementary to show the comparison of “prompts + demos” features in comparison to “just prompts” rewards. \n\n[1] Doğan, F. I., Gillet, S., Carter, E. J., & Leite, I. (2020). The impact of adding perspective-taking to spatial referencing during human–robot interaction. Robotics and Autonomous Systems.\n\n**[Q2 (Claim Concern 2, Weakness 3, Experiment Concern)]** Generalization Claim\n\n**[A2]** In addition to our Ant generalization experiments in simulation, we conducted a real-world user study (see Reviewer tQa4 [A1]) on a salad mixing task using a Kinova JACO arm—a different robot and unseen domain.\n\nDespite this domain shift and user-provided demonstrations, ELEMENTAL significantly outperformed Eureka in both task success ($20.58 \\pm 4.93$ vs. $12.42 \\pm 4.72$, $p < .001$) and strategy alignment ($19.83 \\pm 6.13$ vs. $10.50 \\pm 4.32$, $p < .001$). These results demonstrate ELEMENTAL’s ability to generalize to novel robots, real-world interactions, and imperfect user input.\n\n**[Q3 (Experiment Concern, Weakness 6, Question 3)]** Random seeds and statistical significance\n\n**[A3]** In our revised experiments (see Reviewer 5aJa [A1]), we increased the number of random to 5 across all benchmark and generalization tasks. On average, ELEMENTAL outperforms Eureka by 122.5% in benchmark settings and 81.2% in generalization (better in 8/9 and 4/4 tasks respectively). ELEMENTAL shows statistically significant improvement over Eureka in 5/9 benchmark tasks ($p < .05$) and in 2/4 generalization tasks ($p < .05$). Regarding Figure 3, we updated in Response Figure 1 and confirmed ELEMENTAL has significantly higher execution rate ($p = .030$). \n\n**[Q4 (Weakness 1)]** Novelty\n\n**[A4]** To our knowledge, ELEMENTAL is the first to enable multimodal, self-improving reward learning with VLMs in robotics. ELEMENTAL combines VLMs with LfD, introducing a novel, three-phase framework that includes: (1) multimodal feature extraction from both demonstrations and text, (2) inverse reinforcement learning to optimize reward and policy, and (3) a self-reflective loop that iteratively revises the feature space using feedback from learned behavior. We show improvements over SOTA on standard benchmarks and real-world deployment, and would be grateful if the reviewer could specify which aspects they feel are insufficiently novel so we can better address them.\n\n**[Q5 (Weakness 2, Question 2, Question 4)]** Design choices for VLM and linear reward\n\n**[A5]** We chose GPT-4o in our experiment due to its strong multimodal reasoning capabilities, and we provide additional experiments with OpenAI’s o1 model in Response Table 1 (see Reviewer tQa4 [A4]). Preliminary results show that both ELEMENTAL and Eureka improve under o1, and ELEMENTAL continues to outperform Eureka in 7 out of 9 tasks—suggesting our framework is robust across some VLM choices. We will explore open-source VLMs in future work. \n\nWe agree that exploring richer reward representations is a promising direction for future work. We opted for linear combinations of features for potential human interpretability, providing insight into the importance of each feature. While prompting a VLM to write features and then write a reward function is possible (Eureka’s prompt effectively does this), we find that pairing feature construction with IRL leads to better alignment with demonstrations, as IRL naturally handles balancing feature weights through optimization rather than relying on one-shot reward drafting. \n\n**[Q6 (Weakness 5, Question 1)]** Keyframe\n\n**[A6]** Our real-world user study uses 10 equally spaced frames captured, demonstrating that ELEMENTAL works well with simple, automated keyframing. Please refer to Reviewer tQa4 [A2] for our full response.", "claims_and_evidence": "The main claim of the paper is that integrating demonstrations can reduce the ambiguity of task specification and hence improve performance of LLM/VLM inverse RL. While the results do show an improvement in performance, the evaluation lacks a clear connection between demonstrations and ambiguity being reduced. Perhaps this can be shown with qualitative examples where the demonstrations clearly induce some reward components that could not have been produced by a language-only method.\n\nA second major claim of this paper is that it improves generalization to  out-of-distributions tasks. Here the main concern is that VLM are trained with data online and can only produce reward within the support of reward they have seen during training (reasonable assumption). The authors attempt to validate this claim by designing some custom tasks within IsaacGym and showcase their method being successful on those as well. While this experiment is a good hint for generalization, it is not sufficient to understand the generalization capability of the method to tasks from completely unseen domains (different simulation with not much code online, different robots...).", "methods_and_evaluation_criteria": "Integrating demonstrations into VLM-based reward design is a good idea and a promising direction. The proposed method is reasonable, the assumption of reward being linear to some features is limiting but enough for a large set of tasks. \n\nThe evaluation benchmark is a good start to properly validate the method. I believe real-world results and different simulators would strengthen the claim of generalization and better highlight the contributions of the paper.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "I do not like the usage of only 3 random seeds for testing the method and comparing it to baselines, especially that some of the results are not statistically significant as can be seen in figure 3. Besides that experimental design is good. Analyses is also mostly good, except that claiming generalization to out-of-distribution tasks is a bit of an over claim given that the tested tasks are all very similar and from the same domain.", "relation_to_broader_scientific_literature": "The paper is a logical next step to previous work on LLM/VLM-based reward design.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The paper is well-written and enjoyable to read.\n- Leveraging demonstrations to automated reward design is a reasonable next step and a goo idea to reduce the ambiguity of language-based methods.\n- The experiments show good performance improvement\n\n**Weaknesses:**\n\n- The paper's novelty is quite limited.\n- Many design choices of the method are not well motivated, e.g., choice of VLM, choice of linear-feature rewards..\n- The paper lacks real world experiments or experiments with environments that are truly out-of-distribution, despite the paper boldly claiming generalization to such domains.\n- The paper clearly misses qualitative evaluation to showcase how the additional demos reduce ambiguity.\n- The paper relies on keyframes (selected by an expert) for manipulation tasks. Such information is not easy to get and a more automated approach is desirable.\n- The evaluation uses only 3 seeds.", "other_comments_or_suggestions": "None", "questions_for_authors": "- Would your method work if the keyframe selection is automated and hence some not all that well picked frames are used?\n- How would your method work with different VLMs? ideally it would be interesting to see it working with open-source VLMs.\n- Can you perform more experimental runs with random seeds (at least 5, ideally > 10)?\n- Why did you pick the reward to be a linear combination of the features instead of also prompting the VLM to write a reward function based on the features? how would your method compare to such an approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces ELEMENTAL, a framework for reward design in robotics that integrates vision-language models (VLMs) with inverse reinforcement learning (IRL). It leverages visual demonstrations and natural language descriptions to generate task-relevant feature functions, addressing shortcomings of purely language-based reward engineering. The framework incorporates a self-reflection mechanism to iteratively refine feature functions, reward functions, and policies, ensuring alignment with user demonstrations. Experimental results on IsaacGym and real-world tasks show that ELEMENTAL outperforms state-of-the-art methods such as standard IRL methods (that lack VLM-powered feature extraction) and prior language-based reward-design approaches like EUREKA.

The main motivation of this work is that language can be ambiguous for task requirement specification and hence using user demonstrations is a good way to reduce this ambiguity and create a better interface for human task specifications. The authors provide good examples for this is in the discussion with reviewers. All reviewers agree that the idea of combining IRL with VLM semantic knowledge is novel and sound, and the environments and baselines are good. The experimental results demonstrate that effectiveness of the proposed approach. The paper is well-written and easy to follow.

The authors should include newly experiments, expanded statistical analyses, revised figures, real-world user study results, runtime analysis, discussion related to improvements in feature refinement through self-reflection + reward design compared to Eureka -- as part of the final revised paper draft.
Decision: Accept (poster)