Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Authors: Tung Minh Luu
Younghwan Lee
Donghoon Lee
Sunho Kim
Min Jun Kim
Chang D. Yoo
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning From Human Feedback
Reinforcement Learning From AI Feedback
Rating-based Reinforcement Learning
Primary area: reinforcement_learning->deep_rl
TL;DR: A rating-based reinforcement learning framework that learns rewards from large vision-language model feedback with enhancements to reduce instability from data imbalance and noisy ratings.

Abstract: Designing effective reward functions remains a fundamental challenge in reinforcement learning (RL), as it often requires extensive human effort and domain expertise. While RL from human feedback has been successful in aligning agents with human intent, acquiring high-quality feedback is costly and labor-intensive, limiting its scalability. Recent advancements in foundation models present a promising alternative--leveraging AI-generated feedback to reduce reliance on human supervision in reward learning. Building on this paradigm, we introduce ERL-VLM, an enhanced rating-based RL method that effectively learns reward functions from AI feedback. Unlike prior methods that rely on pairwise comparisons, ERL-VLM queries large vision-language models (VLMs) for absolute ratings of individual trajectories, enabling more expressive feedback and improved sample efficiency. Additionally, we propose key enhancements to rating-based RL, addressing instability issues caused by data imbalance and noisy labels. Through extensive experiments across both low-level and high-level control tasks, we demonstrate that ERL-VLM significantly outperforms existing VLM-based reward generation methods. Our results demonstrate the potential of AI feedback for scaling RL with minimal human intervention, paving the way for more autonomous and efficient reward learning.

Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Tung M. Luu
Younghwan Lee
Donghoon Leel
Sunho Kim
Min Jun Kim
Chang D. Yoo
Abstract
Vinyals al., 2019; Perolat et al., 2022), robotics (Kalash- nikov et al., 2018; Chen et al., 2022), and autonomous sys- tems (Bellemare et al., 2020; Zhou et al., 2020; Kaufmann et al., 2023). A critical factor driving these successes is the design of an appropriate reward function, which both captures the task objective and allows an RL agent to learn efficiently. This reward function is often manually defined and refined by human domain experts, process that is not only time-consuming but also susceptible errors (Skalse et al., 2022). In this work, we aim to eliminate the depen- dence on manually designed reward functions by training agents with reward functions derived from data. In this context, RL from human feedback (RLHF) has emerged as powerful paradigm, enabling the learning of reward functions directly from human feedback (Knox & Stone, 2009; Christiano et al., 2017; Ibarz et al., 2018).
Designing effective reward functions remains a fundamental challenge in reinforcement learning (RL), as it often requires extensive human effort and domain expertise. While RL from human feedback has been successful in aligning agents with human intent, acquiring high-quality feed- back costly and labor-intensive, limiting its scal- ability. Recent advancements foundation mod- els present promising alternative-leveraee Al-generated feedback to reduce reliance on hu- supervision in reward learning. Building on this paradigm, we introduce ERL-VLM, an en- hanced rating-based RL method that effectively learns reward functions from AI feedback. Un- like prior methods that rely on pairwise compar- isons, ERL-VLM queries large vision- languag models (VLMs) for absolute ratings of individual trajectories, enabling more expressive feedback and improved sample efficiency. Additionally, propose key enhancements to rating- based RL, addressing instability issues caused by data im- balance and noisy labels. Through extensive ex- periments across both low-level and high-level control tasks, we demonstrate that ERL-VLM sig- nificantly outperforms existing VIM-based ward generation methods. Our results demon- strate the potential of AI feedback for scaling RL with minimal human intervention, paving the way for more autonomous and efficient re- ward learning. The code available at: https: //github. www.eooo///ooee henm//uungm22233 20//0/////llllmm /erlvm.
One obstacle for employing RLHF at scale is its reliance on a substantial amount of human feedback, which is both costly and labor-intensive. Recent advancements in vision- language models (VLMs) have demonstrated their ability to understand intricate relationships between images and text. When integrated with large language models, large VLMs have shown high degree of alignment with human judgment (Gilardi et al., 2023; Ding et al., 2023; Bao et al., 2024; Yoon et al., 2025). These capabilities position large VLMs as promising proxies for human feedback training RL agents.
Recent studies have explored this approach for training robotic tasks by querying large VLMs for pairwise pref- erences between trajectories (Guan et al., 2024; Wang et al., 2024a). However, querying VLMs for preferences has few disadvantages. First, individual preferences convey little information, requiring a large number of queries to learn meaningful reward functions, which can lead to sample inefficiency. In addition, since preferences derived by comparing least two trajectories, the number of input and (potential) output tokens processed by the VLM effectively doubles. This leads to higher computational costs and a slower querying process, especially for long trajec- tories. Finally, when querying VLMs with trajectories of similar quality (e.g., rrgative-negativ or pairs), the model often provides invalid preferences by "hal-
1. Introduction
inherently
Reinforcement Learning (RL) has achieved remarkable suc- cess in various domains, including games (Silver et al., 2017;
'Korea Advanced Institute of Science and Technology (KAIST). Correspondence to: Chang D. Yoo <cd_yoo@ @aaiskkka.kk.>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
lucinating" that one trajectory is better than the other (Guan et al., 2024). To address this, (Wang et al., 2024a) intro- duces "unsure" label for indistinguishable trajectories and discards them during reward learning. While this miti- gates query ambiguity, it is inefficient as it prevents the full utilization of all queried samples.
expert demonstrations, enabling the learned reward func- tions to guide agents in replicating the demonstrated behav- iors. Various IRL approaches have been developed, includ- ing maximum entropy IRL (Ziebart et al.. 2008; Wulfmeier et al., 2015), non-linear IRI. (Levine et al., 201 I; Finn et al., 2016), adversarial IRL (Ho & Ermon, 2016; Fu et al., 2018), and behavioral cloning IRL (Szot et al., 2023). While these methods have shown success across different domains, they heavily rely on high-quality demonstrations, which are ex- pensive and labor-intensive to obtain. In contrast, ERL- VIM eliminates the need for expert demonstrations by lever- aging only human-prow ouman-provided task descriptions in language to learn a reward function.
To address these challenges, we propose ERL-VLM (Enhancing Rating-based Learning to Effectively Lever- age Feedback from Vision-Language Models), method that efficiently utilizes feedback from large VLMs, such as Gemini (Reid et al., 2024), to generate reward functions for training RL agents. Unlike prior approaches that rely on pairwise comparisons, ERL-VLM queries large VLMs for absolute evaluations of individual trajectories, represented on Likert scale (i.e., range of discrete ratings such as "very bad", "bad", "okk, "good", and "very good"). This approach allows VLMS to provide more expressive feed- back, reduces query ambiguity, and ensures that all queried samples are fully utilized for reward learning. Additionally, we introduce simple yet effective enhancements to existing rating-bassen based RL methods to mitigate instability in reward learning caused by data imbalance and noisy rating labels from VLMs. We demonstrate that ERL- VLM can generate reward functions that successfully train RL agents across a variety of vision- vision-language navigation discrete action and continuous robotic control environments. Notably, ERL- VLM operates solcly on image observations of the agent and a language task description, without requiring privileged state information or access to environment source code, as required in prior work (Ma et al., 2024; Xie et al., 2024; Wang et al., 2024b; Venuto et al., 2024; Han et al., 2024). Our contributions are summarized as follows:
Learning from Feedback. Learning reward functions di- rectly from human feedback has gained significant attention as it typically requires less effort compared to collecting expert demonstrations. Among the most widely used ap- proaches is comparative feedback, where human evaluators perform pairwise comparisons of trajectories to express preferences. These preferences are then transformed into targets for the reward model (Wirth al., 2017; Christiano et al., 2017; Ibarz et al., 2018; Leike et al., 2018; Lee et al., 2:21a;b; Hejna III & Sadigh, 2023). Alternatively, evalua- tive feedback focuses on providing absolute assessments for individual trajectories (Knox & Stone, 2009; Warnell et al., 2018; MacGlashan et al., 2015; Arumugam et al., 2019; White et al., 2024), potentially offering more comprehen- sive insights than relative preferences (Casper et al., 2023; Yuan et al., 2024).
While learning from feedback shows promise, it is often constrained by the requirement for extensive human par- ticipation to generate the necessary feedback. Recently, learning from Al feedback has emerged as a compelling al- ternative. Pretrained foundation models have been leveraged to approximate human feedback for RL finetuning in work on large language models (Bai et al., 2022; Li et al., 2023; Lee et al., 2024; Yu et al., 2024). Building on this paradigm, several studies have explored reducing the reliance on hu- man feedback in training RL tasks by utilizing Al Al-generated preferences (Kwon ct al., 2023; Klissarov et al., 2023; Wang et al., 2024a; Lin et al., 2024; Wang et al., 2025). In contrast to these preference-based approaches, our method obtains absolute evaluations instead of relative preferences, which we hypothesize to provide more expressive and informative feedback for RL tasks leveraging AI models.
We present ERL-VLM, novel method for learning re- ward functions from VL.M-provdded feedback. ERI-VIM enables agents to learn new tasks using only a human- provided language task description, significantly reducing the need for extensive human effort in reward design. We propose simple yet effective enhancements to existing rating-based RL methods, improving stability in reward learning from VLM feedback and thereby improving the overall agent performance.
We demonstrate that ERL-VLM outperforms prior VLM- based reward generation methods, successfully training RL agents across a variety of lanionnl navigation and robotic control tasks. Additionaly, we conduct ex- tensive ablation studies to identify key factors driving ERL-VLM's performance gains and provide deeper in- sights into its effectiveness.
Vision- Languu-L Models as Reward Functions. Recent research has also explored leveraging vision-language mod- els (VLMs), such Contrastive Language- Image Pretrain- ing (CLIP; (Radford et al., 2021)), to directly generate re- ward signals for training agents. These rewards are typically computed by measuring the cosine similarity between the embeddings of a language task description and a visual rep- resentation of the current state or a sequence of states. For
2. Related Work
Inverse Reinforcement Learning. Inverse Reinforce- ment Learning (IRI.) aims to recover reward functions from
2

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Trajectory Segment
(s, a
Vision- Language Model
Task Description
Rating Dataset
"Pick up the mug in the coffee maker"
Reward learning from ratings
(s,,,s', a))
Replay Buffer
Reward Model
Policy
Policy leamning from experiences
Figure 1. Overview of ERL-VLM:
The figure illustrates an example of PickupObject task in ALFRED along with corresponding task description. During each feedback session, trajectory segments are randomly sampled from the replay buffer and sent to the VLM teacher to obtain rating feedback ÿ. The reward model fs is then trained using the collected rating dataset and subsequently used to compute rewards as the agent interacts with the environment.
example, (Fan et al., 2022; Jiang et al., 2024) train CLIP model and use it to reward agents in MineCraft environ- ments. However, this approach requires a lot of labeled, guvironment-speeifa datasets, which are costly and difficult to scale. In robotics, similar approaches have been used to reward agents, either through fine-tuning in zero-shot settings (Mahmoudieh et al., 2022: Cui et al., 2022; Baumli et al., 2023; Ma et al., 2023; Rocamonde et al., 2024; Son- takke et al., 2024). Despite their potential in some domains, these similarity score-based reward signals are often noisy, and their accuracy highly dependent on task descriptions and alignment of image observations with the pretraining data distribution (Rocamonde et al., 2024; Sontakke al., 2024; Wang et al., 2024a). Moreover, most of these meth- ods focus on utilizing the embedding space of VLMs rather than leveraging their reasoning capabilities. In contrast, our approach harnesses the emergent reasoning capabilities of large VI.Ms, such as Gemini (Reid et al., 2024), to provide rating feedback, which is then distilled into reward func- tion for training agents. The work most similar to ours is RL-VLM-F (Wang et al., 2024a), which generates prefer- ence feedback using VLMs. However, querying VLMS for preferences introduces the risk of query ambiguity. Also, prompting VLMs with multiple trajectories can be compu- tationally inefficient. ERL-VLM, instead queries VLMs for absolute ratings for individual trajectories, that improves computational efficiency and ensures all queried samples are fully utilized in reward learning, thereby enhancing the overall performance of training RL agents.
2018). At each time step f. the agent observes the current state 8t and selects an action at according to its policy (-|s), which defines a probability distribution over actions for a given state. Then the environment responses a reward
re = rst,, az) and transitions to the next state 8t+1- The objective of RL algorithms is to find the optimal policy that maximizes the expected return, Ro = E 7'r., which is defined as discounted cumulative sum of the reward with the discount factor 7/
Rating-based RL. In scenarios where a reward function is unavailable, standard RL approaches may not be directly applied to derive policies. RI. from human feedback shown as an effective paradigm for transforming human feedback into guidance signals (Lee al., 2021b; Casper et al., 2023; Yuan et al., 2024). In this work, we build on the recently introduced rating-basee RL framework (White et al., 2024), where a teacher provides discrete ratings for the agent's 's behaviors (Knox & Stone, 2009; MacGlashan et al., 2017: Arumugam et al., 2019), and an estimated reward function As is trained to ensure that the distribution of predicted ratings matches the distribution of collected ratings. Formally, given segment as sequence of states and actions {(51, a1),..................... ($H, aH)}, where H 2 1, a teacher assigns a rating label ÿ from the discrete set C = (0, 1,...,, n - 1), to indicate the segment's quality. Here, 0 denotes the lowest possible rating, while n represents the highest. Descriptive labels can also be associated with rating levels; for example, with three rating levels, level 0 could be labeled "bad", level as "average", and level 2 as "good", Each feedback is then stored in dataset D as tuple (a, V). Given the estimated reward function fg. the estimated return for length-k trajectory segment is computed as R(o) Σk= Pv (8t, az). Then, the probability (i( of assigning
3. Preliminaries
In standard reinforcement learning (RL), an agent interacts with an environment in discrete time steps (Sutton Barto,
3

---

Enhancing Rating-Band Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
the segment to the i-th rating class is formulated as:
1.0
Sasaane (x109)
Anstage Good
e
RR41)
Accuracy
Po(1)
(1)
2 e e
(-(o)) R())
0.5
22
where i((o) E [0,1] denotes the normalized version of R(o), computed using min-max normalization within the current training batch. R₂ represents the rating class bound- aries, constrained as =- Ro < Ri S .... I Rs == 1. These boundaries are determined ensure that, within the batch, the number of samples in each rating category mod- eled by Equation (1) reflects the numbers of ratings assigned
Unioom Samalng
Scaciied Samp bng
0.0
0.5
10
0.0
0.5 Timestops (x109)
1.0
Timesteps (x110))
(a) Accuracy of VLM teacher.
(b) Samples per rating class.
500
100
Un Sumpinng
Estimated Raturn
250
Straliiid Sarrphig
Success Rate
80-
Strliial Sumpling
60
by the teacher. For details on how the boundaries are com- puted, we refer readers to (White et al., 2024).
40
-250
20
-500
Based on the probabilistic rating model in Equation (1), the estimated reward function Fg is optimized by minimizing multi-class cross-entropy loss, defined as:
0.0
0.5
100
0.0
0.5 Timestops (x30%)
1.0
Timesteps (x10f)
(c) Estimated return from fzp. (d) Training performance.
Figure 2. Reward learning from rating feedback in the Drawer Open environment. Due to the imbalance of ratings, optimizing fo using the conventional objective in Eq. (2) with uniform batch sampling often leads to poor performance. To mitigate this issue, we use stratified sampling to ensure that every training batch con- tains samples from all rating classes.
LC D)= -R( En (o) log Po(i)
(2)
where Mo (i) is an indicator function that takes the value if y equals i, and otherwise, and U represents uniform sampling. The policy can then be updated using any RL algorithm to maximize the expected return under the estimated reward function P.).
complete procedure is detailed in Algorithm 1.
Vision-Language Models for Evaluative Feedback. In this work, we leverage large pretrained vision-language models (VLMs; (Zhang et al., 2024)) as teachers to pro- vide feedback. We define VLMs as models capable of pro- cessing both language inputs, p = (Io...., Tm). where
Rating Generation with VLMs. In this work, we focus on tasks where the quality or success of an agent's 's behaviors can be evaluated based on single image or sequence of images. Building on prior work (Wang et al., 2024a), we elicit feedback from the VLM through a two-stage process: first analyzing the agent's behavior and then generating rat- ings based on this analysis. We observe that the quality of ratings heavily depends on the effectiveness of the analy- sis stage. For tasks with simple descriptions, the VLM is capable of producing plausible ratings from single image observation. However, for tasks with complex and composi- tional descriptions, single observations often fail to capture the full context. To address this, we design a prompt that directs the VLM to analyze trajectory segments composed of multiple image observations and, if applicable, associ- ated actions. This approach provides greater granularity, enabling more comprehensive analysis that captures criti- cal nuances in the agent's 's behavior. The prompt templates used in our experiments are shown in Table 6 and 7. Given these prompts, during each feedback session, we sample N segments from the replay buffer and query the VLM to rate the segments according to the task description. The sampled segment, along with its corresponding rating, is then stored in the rating dataset D (lines 12-14 in Algorithm 1).
Im E V. and visual inputs, I E I. Here, V represents a finite vocabulary, and I denotes the space of RGB images. Given these inputs, a VLM T generates language outputs as h [p, where h = (ho.- ..... h4)) and h4 E V. Our focus is on VLMM trained on diverse text and image datasets, which equip them with the ability to generalize effectively across a variety of environments. Additionally, these models must possess strong visual comprehension and question-anse answering capabilities, critical requirements for accurately evaluating and rating trajectories. In this work, Gemini (Reid et al., 2024) as the large pretrained VLM that meets these requirements.
4. Method
In this section, we introduce ERL-VLM, a method for learn- ing reward functions from VLM feedback. We first deseribe the prompt design used to elicit rating feedback from VLMs for the agent's 's behaviors. Next, as the core contribution of this paper, we introduce enhancements to the existing raseng-based RL framework to effectively utilize this feed- back in learning the reward function. Finally, we outline how the learned reward function is used for policy learning. An overview of ERL-VVLM is presented in Figure 1, and the
Reward Learning from Ratings. Given the rating dataset 'D collected from the VLM teacher, the estimated reward function f.. can be optimized using the conventional ob- jective in Equation (2). However, directly applying this
4

---

Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Algorithm 1 ERR-VLM
Entropy (Ghosh al., 2017). The reward function fgg is then optimized using the following objective:
1: Input: Text description of task goal I.
2: Initialize: Policy Ne and reward function P.. RL replay buffer t 0. dataset of rating < VLM query frequency K, number of queries per feedback session, horizon T.
LMAR(W, D) ((o.))uls(D)
Σ (ur(1)) (o()) (3)
3: for each iteration do
4: MAAA COLLECTION
where, Us denotes the stratified sampling strategy. Prior works in preference-based RL often adopt label smoothing (Wei et al., 2021; Christiano et al., 2017; Ibarz et al., 2018) to handle corrupted labels, assuming a fixed rate of uniform noise in the preferences provided by the teacher. However, in our experiments, we find this approach to be ineffective, likely because is challenging to estimate the percentage of corrupted labels produced by the VLM teacher, especially in the context of multi-class ratings.
5: for to T do
6: Collect $14-1, image 14+1 by taking a ~ ss(ar(s:)
7: Update B + B {(st, Le, ar, 8441, J441,
8: end for
9:
I/ RATING BY VLM AND REWARD LEARNING
10: if itteraaion %K == then
11:
for m = 1 to N do
12:
Randomly sample segment a from B
13:
Query VLM with a and task goal for label y Update D 4- D (a)
14:
15:
end for
16:
for each gradient step do
Following prior works in RLHF (Lee et al., 2021a;b; White et al., 2024), we query the VLM teacher every K training steps (Algorithm lines 11-15). At the end of each feedback session, we optimize the reward function Py using the objective defined in Equation 3 (Algo- rithm I lines 17-18). Subsequently, all experiences in the replay buffer are relabeled with the newly updated reward function. Finally, the policy is trained using RL algo- rithm with the relabeled data from the replay buffer. In this work, we adopt SAC (Haarnoja et al., 2018) and a variant of IQL (Kostrikov et al., 2022) as the base RL algorithms.
Implementation Details.
17:
Stratified sampling minibatch {(a, b) D Update For according to Equation (3)
18:
19:
end for
20:
Relahel entire replay huffer B using updated fv. end if
21:
22:
// POLICY LEARNING
23:
for each gradient step do
24:
Sample random batch [(*,, dts, st-1, f ~ B
25: Update the policy To with any off- off-policy RL algorithm
26:
end for
27:
end for
5. Experiments
approach often leads to instability during reward learning, even when the teacher provides reasonable ratings, as shown in Figure 2a. This instability arises from two key challenges.
The goal of our experiments is to evaluate ERL-VLM's ef- fectiveness in generating reward functions for training RL agents. We compare ERL-VLM to VLM-based reward gen- eration methods, which also operate on image observations and language task description, across three domains: low- level manipulation control tasks in MetaWorld (Yu et al., 2020), high-level vision- language navigation tasks in AL- FRED (Shridhar al., 2020), and real-world robotic manip- ulation using Sawyer robotic arm. Concretely, we aim to answer the following questions: (1) Can ERL-VLM gener- ate effective reward functions for policy learning? (2) How do reward functions generated by ERL-VLM compare to those produced by other pretrained VLMs? (3) What the impact of our proposed enhancements on the overall effectiveness of ERL-VLME
First, the distribution of samples across rating classes is often highly imbalanced. For example, in the early stages of training, "bad" ratings dominate other rating classes, re- sulting in skewed datasets (Figure 2b). This imbalance negatively impacts reward learning because multiple train- ing batches may contain samples from only a single rating class, which disrupts the effective selection of rating class boundaries. Consequently, the reward function tends to consistently predict the dominant rating class, leading to degraded performance, as shown in Figure 2c, 2d. It is worth noting that this issue is less prevalent in preference- based RL, as the binary comparisons inherently balance the preferences between two samples. To address this issue, we employ stratified sampling, ensuring that samples from all rating classes are represented each training batch. Addi- tionally, we use a weighted loss function, assigning weights based on class frequency account for data imbalance.
5.1. Experimental Setup
World. In this environment, the agent generates low- level continuous actions to control a simulated Sawyer robot, enabling it to interact with objects on a table to complete tasks. We evaluate ERI-VVLM on three tasks, following prior work (Sontakke et al., 2024; Wang et al., 2024a):
Second, the VLM teacher may provide noisy or imprecise labels due to hallucinations (Zhang et al., 2023b) or other errors. To mitigate the impact of noisy labels, we adopt the Mean Absolute Error (MAE) as the training objective, as it provably more robust to label noise compared to Cross
Sweep Into: Sweep a green puck into a hole on the table. Drawer Open: Pull open a drawer on the table.
5

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Swaep Into
Drawer Opan
Soccer
PldapOtject
PutOlject
CleanOhject
CoolOhject
Real robot tasks
Figure 3. We evaluate ERL-VLM across three domains: two simulated environments and one real-world robotic manipulation setup. The first three images depict tasks from MetaWorld, the next four images illustrate examples from ALFRED, and the final image shows the real-world robotic setup.
Soccer: Kick soccer into the goal on the table.
eration approach has also been explored in several prior works (Cui et al., 2022; Mahmoudieh et al., 2022). RoboCLIP Score: We use the text-based version of Robo- CLIP, where rewards are computed based on the similarity between the video embedding of trajectory and task description in S3D embedding space (Xie et al., 2018). RL-VLM-F (Wang et al., 2024a): Similar to our ap- proach, this baseline leverages a pretrained VLM to pro- vide feedback for reward learning: however, it queries for relative preferences rather than absolute ratings.
ALFRED. this domain, an embodied agent is randomly
placed in room and must select appropriate discrete ac-
tions to complete tasks specified by language instructions.
ALFRED abstracts away low-level control into 12 discrete actions and along with 82 discrete object types. We leverage modified version of the ALFRED simulator (Zhang et al., 2023a) that enables online RL training. We consider a set of 20 tasks drawn from 10 scenes in the Kitchen environment.
These tasks can be categorized into four types:
Environment Reward: This baseline trains the agent using the reward function from environments. The reward function is dense in MetaWorld but sparse in ALFRED.
PickupObject: Pick up specified object from a location.
PutObject: Place object into a target location.
CoolObject: Interact with fridge to cool an object.
Training and Evaluation Procedure. For Meta World tasks, we adopt Soft Actor-Critic (SAC) (Haarnoja et al., 2018) as the base RL. algorithm, while for ALFRED tasks, we use an online variant of Implicit Q-Learning (IQL) (Kostrikov et al., 2022), which has been shown effectively train agents in ALFRED (Zhang al., 2023a). To ensure a fair comparison, we use the same hyperparameters for policy training across all methods, with the only difference being the reward func- tion. For methods that require learning reward functions, we use Gemini- 1 1.5-Pro (Reid et al., 2024) as the VLM for all tasks and use same query budget (10000 for MetaWorld and 1500 for ALFRED), and apply a uniform sampling scheme for queries. Additionally, the reward function in ALFRED also conditions in the task instruction, similar to the pol- icy. For RL-VILM-F, we perform pairwise comparisons between trajectories generated from the same instruction in ALFRED. We evaluate performance using task success rate, as defined by each benchmark. Each evaluation run computes success rates over 10 episodes MerlWorld and 100 episodes in ALFRED. Performance are reported the mean and standard deviation over 3 runs.
CleanObject: Use a faucet to clean an object.
The object types and desired locations are specified in the instructions. We use the original crowd-sourced language instructions from the benchmark as task descriptions. This allows us test the robustness of the VLM teacher in pro- viding feedback based various numan-provided task de- scriptions. Detailed task instructions are provided in Table 2. Unlike MetaWorld, World, this domain requires multi-task learning, where the policy is conditioned on a language instruction. At the start of each episode, a scene is randomly sampled from a set of 10 scenes, with objects initialized based on the scene's 's configuration. Each scene includes two sequential instructions, requiring the agent to complete the first task before proceeding to the next.
The visualizations of these tasks are shown in Figure 3. Further details about the tasks and environment setups can be found in the Appendix.
Baselines. We compare ERL-VLM to established methods that also leverage pretrained VLMs to generate rewards for robotic tasks based on the language task description and the agent's image observations. These methods involve (1) using similarity scores as reward signals and (2) utilizing feedback from VLMs to learn reward functions. Concretely, we compare against the following baselines:
5.2. Main Results
We present the learning curves of success rates for all com- pared methods across two simulated domains in Figure 4.
As shown, ERL-VLM outperforms all baselines in 6 out of 7 tasks, demonstrating its effectiveness in learning reward functions from VLM feedback. For similarity score-based approaches, while they provide useful learning signals, their
CLIP Score (Rocamonde et al., 2024): Generate rewards by computing the cosine similarity between the current image observation and the task description in CLIP em- bedding space (Radford et al., 2021). This reward gen-
6

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Sweep Into
Drawer Open
Soccer
100
100
100
Success Rate
75
75
75-
50
50
50
25
25
25-
0.5
0.75
0.0
0.25
0.5
0.75
0.75
1.0
Timesteps (x10%)
Timesteps (x10f)
Timesteps (x109)
PijkupObject
PutObject
CoolObject
CleanObject
100
100
100
100
Success Rate
75
75
75
75-
50
50
50
50-
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Timesteps (X(10%)
Timesteps (x10*)
Timesteps (x109)
Timesteps (x10%)
Environment Reward
CLIP Score
RoboCLIP Score
RL-VLM-F
ERL-VLM (ours)
Figure 4. Success rate for FRRL-LMM and baselines across manipulation tasks from MetaWorld and 20 vision-language navigation tasks from ALFRED. Tasks of the same type in ALFRED are aggregated for clearer visualization. Our method achieves prominent performance on all tasks, and significantly outperforms baselines on ALFRED tasks. Results are means of 3 runs with standard deviation (shaded area).
performance often fluctuates throughout training. This in- consistency arises from the inherent noisiness of the reward signals, as also observed in prior works (Ma et al., 2023; Sontakke et al., 2024; Rocamonde et al., 2024; Wang al., 2024a). This suggests that relying on direct embedding similarities results in unreliable reward values, leading to instability in policy optimization.
Table I. Success rates on the 3 real-world robot evaluation tasks table-top environment.
Sweep Bowl
Drawer Open
Pickup Banana
Method
BC
0.50 + 0.10
0.23 0.06
0.17 + 0.06
Sparse Rewards
0.57 0.06
0.37 0.06
0.30 + 0.10
ERI-VLM (Ours)
0.73 + 0.06
0.60 + 0.10
0.47 + 0.12
For reward learning from VLM approaches, RL-VLM-F performs comparably to ERL-VLM in Drawer Open but struggles significantly in other tasks, particularly in AL- FRED, where it nearly fails to learn meaningful policies. We attribute this to the limited query budget, which results in a relatively small number of preference feedback samples per task in ALFRED (e.g.. 75 in our experiments). Given that each preference only conveys little information about relative ranking between two segments, the reward function struggles capture the desired behavioral nuances effec- tively. In contrast, ERL-VVL leverages absolute ratings, which offer greater global value than preferences within the same query budget. This results in a more informative reward signal for policy training, enabling more effective utilization of VLM feedback.
5.3. Real Robot Evaluation
We further evaluate ERL-VLM in a real-world robotic ma- nipulation setting using 7-DOF Rethink Sawyer robot inter- acting with objects in a tabletop environment. We compare our approach against two baseline approaches: (1) Behavior Cloning (BC), supervised policy pretraining method, and (2) IQL with Sparse Rewards, an offline RL setting where the agent learns from sparsely provided rewards. Image ob- servations are captured by a RealSense camera, as illustrated in Figure 3 (rightmost). We consider following tasks:
Sweep Bowl: Sweep bowl across red boundary line. Drawer Open: Pull open drawer.
Pickup Banana: Grasp banana from plate and lift it.
Interestingly, for the tasks of PickupObject and PutOb- ject tasks, ERL-VLM outperforms even the sparse reward function. This suggests that learning a reward function from absolute ratings not only provides useful learning sig- nals-similar to sparse rewards- also introduces more shaping signals that emphasize critical states for task com- pletion. We attribute this to the improved analysis in our prompt design, which enables the VLM to conduct a deeper assessment of agent behaviors. This result highlights the po- tential of VLMs to serve as effective substitutes for human providing feedback in reward learning.
For each task, we collect 50 demonstrations via human teleoperation and query Gemini to obtain rating feedback offline. The reward model then trained using the objective in Equation 3. The learned reward function is subsequently used to relabel rewards in the offline dataset. We employ IQL (Kostrikov et al., 2022) to train the policy in an offline manner, following the setup in (Yuan et al., 2024). Results in Table I demonstrate that ERL-VLM is capable of gener- ating useful reward functions that facilitate policy learning, indicating its potential for real-world robot manipulation tasks. More visualization results are shown Figure 9.
7

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Languag, Models
Sweep Into
Drawer Open
Soccer
100
100
100
Success Rate
75
75
75
50
50
50
25
25
0
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
Timesteps (x109)
Timesteps (x109)
Timesteps (x10%)
ERL-VLM
No MAE
No Stratified Sampling
Vanilla RbRL
Figure 5. Ablation studies with different enhancements in ERL-VLM. Results means of 3 runs with standard deviation (shaded area). Drawer Open
Drawer Open
Sweep Into
100
100
100
Ours w/ LS
Success Rate
n-3
n-3
75
50
50
50
25
25
25
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
Timesteps (x10%)
Timesteps (x10%)
Timesteps (x10*)
(a)
(h)
(c)
Figure 6. (a) The effect of label smoothing in handling the corrupted labels. (b/c) ERL-VLM with different numbers of rating classes.
5.4. Ablation Studies
replace the MAE loss with cross-entropy loss using a soft label: (1 r) 5 + r/3 [1, 1JT where 0.1, follow- ing prior work (Ibarz et al., 2018; Lee et al., 2021b). The results, shown in Figure 6 (a), indicate that label smoothing is ineffective in our experiment.
Enhancements to Rating-basee based RL. To better analyze the impact of our core contributions, we conduct ablation studies on Meta World tasks using the following variants:
Vanilla RbRL: The original rating-based RL framework (White et al., 2024), which optimizes the reward fune- tion using cross-entropy loss with uniform sampling, as defined in Equation (2).
Different Number of Rating Classes. To evaluate the impact of the number of rating classes n. we modify the prompt to query VLM with n {2, 3, 4] and evaluate performance on Sweep Into and Drawer Open. The re- sults, shown in Figure 6 (b/c), reveal that increasing (e.g., 4) degrades performance in both tasks, likely due to increased ambiguity and inconsistency in VLM-generatee performs best in Sweep Into, n = yields superior results in Drawer Open. We attribute this to task task-specific differences in evaluation clar- ity: in Sweep Into, evaluating whether the puck has placed in the hole is binary and unambiguous, whereas in Drawer Open, the degree of opening is more subjective, making three-class rating scheme more suitable. We anticipate that more refined prompt engineering, with carefully crafted rating descriptions, could further exploit the potential of VLMs for feedback.
No MAE: A variant of ERL- VLM that replaces mean absolute error (MAF) loss with cross-entropy loss while retaining stratified sampling and weighted loss, mirroring
Vanilla RbRL with data balancing improvements.
ratings. Interestingly. while
No Stratified Sampling: variant of ERL-VI VRL-VLM that removes stratified sampling and weighted loss, leaving only MAE as the enhancement over Vanilla RbRL.
We report results in Figure 5.
The analysis shows that each enhancement contributes to the performance of ERL- VLM when learning from rating feedback. Notably, adopt- ing MAE loss provides the most significant improvement over Vanilla RbRL, demonstrating its robustness in handling noisy labels generated by VLMs. Additionally, stratified sampling and weighted loss effectively mitigate data im- balance, particularly in Sweep Into and Drawer Open. We further investigate the impact of label smoothing (LS) (Wei et al., 2021) as an alternative strategy for handling corrupted labels in Drawer Open (3 rating levels). Specifically, we
6. Conclusion
In this work, we propose ERL-VLM, a method for learning reward functions by querying VLMs for rating feedback based on a numan-provided language task description and
8

---

Enhancing Rating-Base Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
image observations. Unlike preference-based approaches, ours leverages VLM feedback more effectively and effi- ciently by querying for absolute evaluations rather than pair- wise comparisons. We demonstrate its effectiveness across low-level continuous control, high-level vasion-language navigation, and real-world robotic manipulation tasks. Our results highlight ERL-VLM's potential to reduce human effort in reward design. paving the way for more scalable and autonomous RL systems.
feedback. arXiv:2212.080073 2022.
Bao, H.,, Huang, Y., Wang, Y., Ye, J., Wang, X., Chen, X., Elhoseiny, M., and Zhang, X. Autobench-v: Can large vision-language models benchmark themselves? arXiv:2410.2125/ 2024.
Baumli, K., Baveja, S., Behbahani, F., Chan, H., Comanici, G., Flennerhag, S., Gazeau, M., Holsheimer, K., Horgan, D., Laskin, M., al. Vision-language models as source of rewards. arXi::2312.09188 2023.
Impact Statement
Bellemare, M. G., Candido, S.. Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 2020.
Our approach offers a promising approach to aligning RL agents with more fine-grained human intent by leveraging vision-language models (VLMs) for reward learning. By carefully designing natural language prompts, our method enables rapid and scalable feedback collection, significantly reducing human effort in prototyping and deploying RI sys- tems in real-world applications. However, our method relies on large foundation models, it inherits their inherent biases, which could propagate into RL agents. This raises important considerations regarding robustness and safety, especially in safety-critical applications. Ensuring the reli- ability of Al-driven feedback remains crucial challenge. In this work, we use only visual observations and descrip- tive actions to analyze agent behavior. Future research can explore integrating additional sensing modalities to further enhance the accuracy and reliability of feedback from foun- dation models, ultimately leading to more trustworthy and effective RL systems.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of re- inforcement learning from human feedback. Transactions on Machine Learning Research (TMLR), 2023.
Chen, Y., Wu, T., Wang, S., Feng, X., Jiang, J., Lu, Z., McAleer, S., Dong, H., Zhu, S.-C., and Yang. Towards human-level bimanual dexterous manipulation with rein- forcement learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
Acknowledgements
Cui, Y., Niekum, S., Gupta, A., Kumar, V., and Rajeswaran, A. Can foundation models perform zero-shot task specifi- cation for robot manipulation? In Learning for Dynamics and Control Conference (L4DC), 2022.
This work was supported by Institute for Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea goVe))men((MSITT (No.RS- 1021-1121181,, Development of Causal Al through Video Understanding and Reinforcement Learning, and Its Ap- plications to Real Environments) and partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea 84, Development and Study AI Technologies to Inexpensively Conform to Evolving Policy on Ethics).
Ding, B., Qin, C.,, Liu, L., Chia, K., Joty, S., Li, B., and Bing. L. Is gpt-3 good data annotator? In Association for Computational Linguisties (ACI.), 2023.
government
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
References
Finn, C., Levine, S., and Abbeel, P. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICML), 2016.
Arumugam, D., Lee, K., Saskin, S., and Littman, M. L. Deep reinforcement learning from policy-dependent hu- man feedback. arXiv: (tt.:::902.0227... 2019.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., al. Constitutional ai: Harmlessness from ai
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. International Conference on Learning Representations (ICLR), 2018.

---

Enhancing Bating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Ghosh, A., Kumar, H., and Sastry, S. Robust loss func- tions under label noise for deep neural networks. In Association for the Advancement of Artificial Intelligence (AAAI), 2017.
Klissarov, M., D'Oro, Sodhani, S., Raileanu, R., Bacon, P.-..,, Vincent, P., Zhang, A., and Henaff, M. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv:2310.00166, 2023.
Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outper- forms crowd workers for text-annotation tasks. In PNAS, 2023.
Knox, W. B. and Stone, P. Interactively shaping agents via human reinforcement: The tamer framework. K-CAP, 2009.
Guan, L.. Zhou, Y., Liu, D., Zha, Y., Amor, H. B., and Kambhampati, S. Task success is not enough: Investigat- ing the use of video- Languala models as behavior critics for catching undesirable agent behaviors. Conference on Language Modeling (COLM), 2024.
Kostrikov, I,, Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. International Confer- ence on Learning Representations (ICLR). 2022.
K Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In International Confer- ence on Learning Representations (ICLR), 2023.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- oef-cpolcy maximum entropy deep reinforce- ment learning with stochastic actor. In International Conference Machine Learning (ICML), 2018.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In International Conference on Machine Learning (ICML), 2024.
Han, X., Yang, Q.. Chen, X., Cai, Z., Chu, X., and Zhu, M. Autoreward: Closed-loop reward design with large language models for autonomous driving, IEEE Transac- tions on Intelligent Vehicles, 2024.
Lee, K., Smith, L., and Abbeel, Pebble: Feedback- efficient interactive reinforcement learning via re- labeling experience and unsupervised pre-training. arXiv:2106.0500 2021a.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-pref: Benchmarking preference-based reinforcement learning. Conference on Neural Information Processing Systems (NeurIPS), 2021b.
Hejna III, D. J. and Sadigh. D. Few-shot preference learn- ing for human-in-the-loop rl. Conference on Robot Learning (CoRL), 2023.
Leike, J., Krueger, D.,, Everitt, T. Martic, M., Maini, V., and Legg. S. Scalable agent alignment via reward modeling: research direction. arXiv: 0eee///1110077711 2018.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. In Conference on Neural Information Processing Systems (NeurIPS), 2016.
Levine, S., Popovic. Z., and Koltun, V. Nonlinear in- verse reinforcement learning with gaussian processes. Conference on Neural Information Processing Systems (NeurIPS), 2011.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. In Conference on Neural Information Processing Systems (NeurlPS), 2018.
Li, L.,, Xie, Z., Li, M., Chen, S., Wang, P., Chen, L.., Yang, Y., Wang. B., and Kong, L. Silkie: Preference distillation for large visual language models. arXiv:23 10.66655,. 2023.
Jangg H., Yue, J., Luo, H., Ding, Z., and Lu, Z. Rein- forcement learning friendly vision-language model for minecraft. In European Conference on Computer Vision (ECCV), 2024.
Lin, M., Shi, S., Guo, Y., Chalaki, B., Tadiparthi, V., Pari, M., Stepputtis, S., Camphell, J., and Sycara, K. Navi- gating noisy feedback: Enhancing reinforcement learning with error- error-prone language models. Empirical Methods in Natural Language Processing (EMNLP), 2024.
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Van- houcke, V., et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning (CoRL), 2018.
Ma, Y. J., Kumar, V., Zhang, A., Bastani, O., and Jayaraman D. Liv: Language-iman representations and rewards for robotic control. In International Conference on Machine Learning (ICML., 2023.
Kaufmann, E., Bauersfeld, L.,, Loquercio, A., Müller, M., Koltun, V., and Scaramuzza, D. Champion-lewe drone racing using deep reinforcement learning. Nature, 2023.
10

---

Enhancing Rating-Ban Bardd Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L.,, and Anandkumar, A. Eureka: Human-level reward design via coding large language models. International Conference on Learning Representations (ICIR), 2024.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T.,, Baker, I.,, Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. Nature,
Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward gaming. In Conference Neural Information Processing Systems (NeurIPS), 2022.
MacGlashan, J., Babes- Vroman, M., desJardins, M., Littman, M. L., Muresan, S., Squire, S., Tellex, S., Aru- mugam, D., and Yang, L.. Grounding english commands to reward functions. In Robotics Science and Systems (RSS), 2015.
Sontakke, S., Zhang, J., Arnold, S.,, Pertsch, K., Bıyık, E., Sadigh, D., Finn, C., and Itti, L. Roboclip: One demon- stration is enough to learn robot policies. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D. L., Taylor, M. E., and Littman, M. L. Interac- human feedback. In
tive learning from International Conference on Machine Learning (ICML), 2017.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Mahmoudieh, P., Pathak, D., and Darrell, T. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning (ICML), 2022.
Szot, A., Zhang, A., Batra, D., Kira, Z., and Meier, Bc-irl: Learning generalizable reward functions from demonstra- tions. wrXhv23303.1614 2023.
Venuto, D., Islam, S. N., Klissarov, M., Precup. D., Yang, S., and Anand, A. Code as reward: Empowering rein- forcement learning with vlms. International Conference on Machine Learning (ICML), 2024.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: A universal visual representation for robot manipulation. Conference on Robot Learning (CoRL), 2022.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathicu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft using multi- agent reinforcement learning. Nature, 2019.
Perolat, J., De Vylder, B., Hennes, D., Tarassov, E., Strub, F., de Boer, V., Muller, P., Connor, T., Burch, N., Anthony, T., et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 2022.
Wang, R., Zhao, D., Yuan, Z., Obi, I., and Min, B.-C. Pref- clm: Enhancing preference based reinforcement learning with crowdsourced large language models. IEEE Robotics and Automation Letters (RA-L), 2025.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Wang. Y., Sun, Z., Zhang, J., Xian, Z., Biyik, E., Held, D., and Erickson, Z. Rl-vim-f: Reinforcement learning from vision language foundation model feedback. In- ternational Conference on Machine Learning (ICML), 2024a.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap. T., Alayrac. J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. 3xXiv22433005330 2024.
Wang, Y., Xian, Z., Chen, F,, Wang. T.-H., Wang, Y., Fragki- adaki, K., Erickson, Z., Held, D., and Gan, C. Robo- gen: Towards unleashing infinite data for automated robot learning via generative simulation. In International Con- ference on Machine Learning (ICML), 2024b.
Rocamonde, J., Montesinos, V., Nava, E., Perez, E., and Lindner, D. Vision-language models are zero-shot re- ward models for reinforcement learning. International Conference on Learning Representations (ICLR), 2024.
Warnell, G., Waytowich, N., Lawhern, V., and Stone, P. Deep tamer: Interactive agent shaping in high- dimensional state spaces. In Association for the Advance- ment of Artificial Intelligence (AAAT), 2018
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi. R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Conference on Computer Vision and Pattern Recognition (CVPR),200 2020.
Wei, J., Liu, H., Liu, T., Niu, G., Sugiyama, M., and Liu, Y. To smooth or not? when label smoothing meets noisy labels. arXiv:2 106.00 04 /49 2021.

---

Enhancing Rating-Basce Baaed Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
White, D., Wu, M., Novoseller, E., Lawhern, V. J., Way- towich, N., and Cao, Y. Rating-based reinforcement learning. In Association for the Advancement of Artificial Intelligence (AAAI), 2024.
Zhang, Y., Li, Y., Cui, L.,, Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., al. Siren's song in the ai ocean: survey hallucination in large language models. arXiv:3309002219 2023b.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. A survey of preference-based reinforcement learning meth- ods. Journal of Machine Learning Research (JMLR), 2017.
Zhou, M., Luo, J., Villella, J., Yang, Y., Rusu, D., Miao, J., Zhang, W., Alban, M., Fadakar, I., Chen, Z.., Huang, A. C., Wen, Y., Hassanzadeh, K., Graves, D., Chen, D., Zhu, Z., Nguyen, N., Elsayed, M., Shao, K., Ahilan, S., Zhang, B., Wu, J., Fu, Z.., Rezaee, K., Yadmellat, P., Rohani, M., Nieves, N. P., Ni, Y., Banijamali, S., Rivers, A. C.,, Tian, Z., Palenicek, D., bou Ammar, H., Zhang, H., Liu, W., Hao, J., and Wang, J. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. Conference Robot Learning (CoRL), 2020.
Wulfmeier, M., Ondruska, P. and Posner, I. Max- imum entropy deep inverse reinforcement learning. arXiv: 04888, 2015.
Xie, S., Sun, C.,, Huang, J., Tu, 7.,, and Murphy, K. Re- thinking spatiotemporal feature learning: Speed-accuracy trade-offss in video classification. In European Conference Computer Vision (ECCV), 2018.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intellignnce (AAAI), 2008.
Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q.. Zhong, V., Yang, Y., and Yu, T. Text2reward: Automated dense reward funetion generation for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
Yoon, E., Yoon, H. S., Hasegawa-Johnson. M. A., and Yoo, C. D. Can video Iims refuse to answer? alignment for answerability video large language models. In Interna- tional Conference on Learning Representations (ICLR), 2025.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. deta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning milms through open- spercs ai feedback for super gpt-4v trustworthiness. arXiv:2405.17222 2024.
Yuan, Y., Hao, J., Ma, Y., Dong, Z., Liang, H., Liu, J., Feng, Z.,, Zhao, K., and Zheng, Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. International Conference on Learning Representations (ICLR), 2024.
Zhang, J., Zhang, J., Pertsch, K., Liu, Z.,, Ren, X., Chang, M., Sun, S.-H., and Lim, J. J. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. Conference on Robol Learning (CoRI.), 2023a.
Zhang, J., Huang, J., Jin, S., and Lu, S. Vision-languaal models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
12

---

Enhancing Bating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Appendix
A. Details on Tasks and Environments
A.I. MetaWorld Tasks
(a) Sweep Into
(b) Drawer Open
(c) Soccer
Figure 7. Examples of image observations in World for querying the VLM teacher and reward lcarning.
Figure 8. Example of trajectory segment from "PickupObject" task in scene 4.
We evaluate our methods on three tasks from MetaWWrrd (Yu et al., 2020), following prior work (Sontakke et al., 2024; Wang et al., 2024a). We describe tasks as follows:
Sweep Into: The goal is to sweep green puck into a square hole on the table. The task is considered successful when the puck remains inside the hole.
Drawer Open: The goal is to pull open drawer on the table. The task is considered successful when the drawer extends
beyond certain threshold.
Soccer: The goal is to kick soccer ball into the goal. The task is considered successful when the ball remains inside the goal area.
Observation Space. We state-based observations for policy learning, whereas for reward learning,
RGB
image-based observations rendered from the simulator. Following the original MetaWork setup (Yu et al., 2020), the state observation is a 39-dimensional vector consisting of the end-effector position, the gripper status, the object's 's position, the object's 's orientation, and the goal position. For image observations, we render RGB images at resolution of 224 x 224. Additionally, as these tasks are all object-centric, we follow (Wang et al., 2024a) to remove the robot from the image to facilitate the analyzing process VL.Ms. The examples of image observation are shown Figure 7.
The action is 4-dimensional vector, normalized within [1. The first three dimensions represent the displacement in the position of the end-effector, and the last dimension is a normalized torque applied to the gripper fingers.
Action Space.
A.2. ALFRED Tasks
We evaluate our methods in modified version of the ALFRED simulator (Zhang et al., 2023a) to assess its effectiveness in solving high-level vision-language navigation tasks. We focus on 20 tasks drawn from 10 scenes in the Kitchen environment, which requires both navigation and object manipulation. Each scene includes two sequential instructions, and the agent must complete the first instruction before proceeding to the next. The detailed task instructions are provided Table
Observation Space. For both policy learning and reward learning, we use input observations composed of the 224 x 224 egocentric RGB image and the robot's pose. The robot's pose is 4-dimensional vector, consisting of its (I, y, 2) coordinates and camera horizon angle. The camera horizon can take values from (-45, -30, ......., ,0, 45}. For all methods, we preprocess the image observation using a frozen ResNet-18 18 encoder (He et al., 2016) pretrained on ImageNet, obtaining a visual representation of shape 512 x 7. The example of image observations are shown in Figure
13

---

Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Action Space. In ALFRED, the agent operates with discrete action space of 12 high-level actions, including navigation MoveAhead, RighteRight, Rotateleeft, LookUp, and LookDown, and 7 interaction actions: Put, Pickup, Open, Close, ToggleOn, ToggleOff, and Slice. For interaction actions, the policy additionally selects one of 82 object types to interact with. Note that for the VLM analysis prompt, we use only actions and not object types. Due to large discrete action space (5 + 82), we perform same masking as (Zhang et al., 2023a) to prevent agents from taking actions that are not possible (e.g., the policy cannot output Close for object Tomato).
actions:
Scene Task Type PickupObject
Task Instruction
Pick up the spoon from the counter Put the spoon in the white cup on the shelf.
PutObject
PickupObject
Pick up egg that is beside the fork in the sink.
2
Open the refrigerator, then place the egg on the glass shelf and close the fridge. Wait then open the fridge and pick up the egg, then close the fridge.
CoolObject
PickupObject
Pick up the tomato from the sink.
3
Open the fridge door, put the tomato inside of the fridge, close the door, open the door, take the tomato out, close the door.
CoolObject
PickupObject
Pick up the mug in the coffee maker
Open the fridge. put the cup in the fridge, close the fridge, wait, open the fridge, pick the cup, close the fridge
CoolObject
PickupObject
Pick up the bread.
5
Open the fridge, put the bread in the tridge, close the fridge, open the fridge, get the bread, and close the fridge.
CoolObject
PickupObject
Pick up the white coffee cup to the right of the trophy.
OleanObject
Put the coffee cup in the sink, turn on the water, turn off the water and pick up the coffee cup.
PickupObject
Pick up the smaller silver knife the counter.
7
PutObject
Put the knife in the green cup in the sink.
PickupObject
Pick up bowl from shelf
8
PutObject
Put the bowl on the counter
PickupObject
Grab the knife from the counter
PutObject
Put the knife in the pan the stove
PickupObject
Pick up the knife from the counter.
CleanObject
Place the knife the sink and turn the water on. Turn the water off and pick up the knife.
Table 2. Task instructions from the Kitchen environment in ALFRED.
A.3. Real Robot Tasks
Observation Space. For policy learning, we use input observations composed of the 480 x 480 RGB image captured by an Intel RealSense D435i and the end-effector's state. For all methods, we preprocess the image by first downsampling to 224 x 224 and then transforming it into a 512-dimensional feature vector using a pretrained R3M image encoder (Nair et al., 2022). The end-effector state is 5-dimensional vector consisting of the end-effector's position (z, y, 2), yaw orientation, and gripper's binary state (open/close). We concatenate the image feature with the end-effector state to form the final input observation, resulting in a 317-dimensional vector.
The action space consists of (1) the displacement the end-effector's position in Cartesian space and (2) a discrete value indicating whether the gripper should open or close. These actions are transmitted to the robot with 10 Hz and converted to the desired joint poses using the Sawyer robot's Intera SDK inverse kinematics solver.
Action Space.
B. Training Implementation Details and Hyperparameters
B.1. Meta World
Policy Training. We build on RbRL (White et al., 2024) for training policy, which is similar to PEBBLE (Lee et al., 2021a), except that it learns from rating feedback. Original RbRL's implementation only supports locomotion tasks, we extend it to support manipulation tasks MetaWorld. We use SAC (Haarnoja et al., 2018) as the off-policy RL algorithm, the detailed hyperparameters are shown in Table 3. For all methods, the policy is learned with state-based observations.
Phttps://// thub. com D nn- based- weweeddReinforeemee wewereemmeeee--eennnnene
14

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Langune Models
Parameter
Value
Number of layers
3
Hidden units per each layer
256
Steps of unsupervised pre-training # Training Steps
9000
1e6
Batch Size
512
Learning rate
0.0003
Optimizer
Adam
Initial temperature
0.1
Critic target update freq
2
Critic EMA
0.005
(81,82)
(0.9, 0.999)
Discount 5y
0.99
VLM query frequency K
5000
Number of queries N per session
50
Maximum budget of queries
10000
Table 3. Hyperparameters for SAC in MetaWorld.
Reward Learning. Following (Wang et al., 2024a), we use an image-based reward model consisting of 4-layer Convolutional Neural Network, followed by a linear layer and a Tanh activation. Additionally, we employ ensemble of three reward models. For all tasks, use segment size of 1 for reward learning. The reward models are optimized using loss function defined in Equation (3). We use batch size of 512 and Adam as the optimizer with the learning rate of 0.0003.
B.2. ALFRED
Policy Learning. We use Implicit Q-Learning (IQL) (Kostrikov et al., 2022) with transformer-based architecture for both the policy and critic networks, similar to (Zhang et al., 2023a). The hyperparameters are provided in Table 4. The main difference is that we set the quantile parameter = 0.5, making IQL a standard off- online RL algorithm,, rather than one suited for offline RL as in (Zhang et al., 2023a). During training, to reduce exploration time time-which particularly challenging AL we seed the buffer with 3 human-collected demonstrations for each task. important to note that while these demonstrations complete the task, they are not necessarily optimal. This is applied consistently across all baselines. Additionally, we relabel the rewards in the seed buffer to align with each baseline's reward function, ensuring compatibility during training. For all methods, the policy takes input sequence of states and actions as input, and each state is structured as mentioned in Section A.2.
Reward Learning. For the reward model, we use the same transformer-basee architecture as the policy, which also takes inputs same the policy. The output of the reward model is bounded by a Tanh activation function. We also employ an ensemble of three reward models, similar to the setup in MetaWorld Additionally, since the buffer is seeded with demonstrations, we first query the VL.M for roughly 1000 feedback samples and perform warmup training on the reward model using these feedback. During online training, we continue training the reward model with remaining query budget. The reward models are optimized using the loss function defined in Equation (3). We use batch size of 512 and AdamW as the optimizer with a learning rate of 0.0001. This setup is applied to both RL-VLM-F and ERL-VLM.
B.3. Real Robot
Policy Learning. We train the policy and critic networks using Implicit Q-Learning (IQL) (Kostrikov et al., 2022), employing a transformer-base
architecture similar to ALFRED tasks. The hyperparameters are provided in Table 5. For the Sparse Rewards baseline, we assign reward of one at the final step of the demonstrations. Since the gripper action space discrete and the dataset is imbalanced, we adjust the gripper loss by applying inverse frequency weighting based on the number of examples in each class.
Reward Learning. The reward model utilizes the same Convolutional Neural Network architecture as MellW environment. We use a Tanh activation function to bound the output. We also employ an ensemble of three reward models.
15

---

Enhancing Rating-Basse Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Languaa Models
Parameter
Value
Batch Size
128
# Trriniin Steps
5e5
Learning Rate
0.0001
Optimizer
AdamW
Dropout Rate
0.1
Weight Decay
0.1
Discount y
0.97
Q Update Polyak Averaging Coefficient
0.005
Policy and Q Update Period
8 per train iter
IQ Advantage Clipping
[0, 100]
IQL Advantage Inverse Temperature 3
5
IQL Qunatile т
0.5
Maximum Context Length
8
Warmup query
1000
# Epochs for warmup training reward model
100
VLM query frequency
50 epochs
Number of queries N per session
50
Maximum budget of queries
1500
Table Hyperparameters for IQL in ALFRED.
We use a batch size of 512 and the Adam optimizer with learning rate 0.0001.
Parameter
Value
Batch Size
256
# Training Steps
1c5
Learning Rate
0.0001
Optimizer
Adam
Discount Y
0.99
Q Update Polyak Averaging Coefficient
0.005
IQL Advantage Clipping
[0, 100]
IQL Advantage Inverse Temperature 3
3
IQL Qunatile T
0.7
Table Hyperparameters for IQL in Real Robot.
C. Baselines
We rerun RL-VLM-F using its original implementation", ensuring a fair comparison by using the same Gemini-1.5 P.o-Pr (Reid al., 2024) as the VLM. While we can achieve similar reported performance in Sweep Into and Open Drawer, we observe lower performance in Soccer. For CLIP Score, we also use the implementation from RL-VLM-F, and we extend RoboCLIP Score based the RL-VLM-F framework. Additionally, we observe that both CLIP and RoboCLIP achieve higher performance than previously reported in RL-VLM-F. In Meta World, task descriptions for CLIP, RoboCLIP, and RL-V remain the same as those used in (Wang et al., 2024a), as shown in Tables 11, 12, and 13. For ALFRED, we use task instructions as task description for all methods. The prompt template used in RL-VIL-F is shown in Table 10.
D. Prompts and Task Descriptions
The prompt templates used obtain ratings in ERL-VLM are shown in Table 6 (for MetaWorld), Table 7 (for ALFRED), and Table 8. The prompt templates for obtaining preferences in RL- RL-VLM-FF are provided in Table 9 (for Meta Norld))ld) and Table 10 (for ALFRED).
Phttps://gtt thub. com
16

---

Enhancing Rating-Basc Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Prompt Template for ERL-VVLM
Analyzing:
<Image>
on>. You will presented image a rohot arm performing the task <Task Descri on?, Please focus on target object in the task and carefully analyze the image in term of completing the task.
Rating:
<Analyses from previous response>
From the above analyses, based this rating category: <Ratiin C1 Classes>, how would you this image in of completing lask <Task Descriptien>?
Table 6. Prompt Template for ERL-VLM used in MetaWorld environment: Rating classes can be [Bad, Average, Good].
Prompt Template for ERL-VLM
Analyzing:
< <magee>>
You will be presented with an image containing segment of the trajectory of robot performing the task < <Inst ruct ruction>, The trajectory segment contains <N> time steps of visual observations, corresponding to <N-1> intermediate actions.
The intermediate actions are: <Actioni,..... Act -on--12:
Note that when an object is picked up, appears close to the bottom the view.
Please analyze the visual differences between consecutive time steps, reply the changes between consecutive time steps in cach
line explicitly. For example:
Timestep 0 1 (Executed action): Your analysis
Timestep 2 (Executed action): Your analysis
and so on
The task is to <Instructions analyze this segment in terms of completing the task.
Rating:
<Anal CAnalyses from previ response>
You are tasked with rating the agent's 's performance in completing task <Instructions From the above analyses for
<N 1> transitions, based on this rating category: <Bad, Average, Good>, where:
BBad: The task is not completed.
Average: The task is partially completed, or you helieve the transition is critical for achieving the goal, such as the target object
appear in the agent's view, but falls short of completion.
Good: The task completed.
How would you rate each transition in terms of completing task?
Please reply a single line of list of ratings for <N 1> transitions.
(For example: [rating of transition 1, rating of transition 2, ... rating of transition <N 1>).
Table Prompt Template for ERL-VLM used ALFRED environment.
Prompt Template for ERL-VLA
Analyzing:
< <Image>>
You will be presented image of robot arm performing the task <Task Description>. Please focus on the target object the task and carefully analyze the image term achieving the task.
Rating:
<Anaayyses from previous response>
From the above analyses, based on this rating category: <Rating Classes>, how would you rate this image in terms of completing task <Task Descriptien:?
Table Prompt Template for ERL-VLM used Real Robot environment.
17

---

Enhancing Rating- Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Prompt Template for RI.-VLM-F
Analysis Template:
Consider following two images:
Image I:
<Image 1>
Image
<mmage
2>
What shown Image
What shown Image 2?
3. The goal is to <Task Description>, Is there any difference between Image and Image in terms of achieving the goal?
Labeling Template:
Based on the text below to questions:
[Repeat the 3 questions the Analysis Template]
<VLM response>
the goal better achieved in Image 1 or Image 2? Reply single line of 0 if the goal is better achieved in Image 1, or is hetter achieved in Image 2.
Reply -I text is unsure or there no difference.
Table 9. Prompt Template for RL-FF used in MetaWorld environment.
Prompt Template for RL-VLM-F
Analysis Template:
Consider the following two trajectory segments:
Trajectory Segment 1:
<Image 1>
Trajectory Segment
< <mmage 2>
Trajectory segment contains <N1>> time steps of visual observations, corresponding to <N1 1> intermediate actions.
The intermediate actions are: <Actioon,- Act Lon, -1>,
Trajectory segment 2 contains <2>>>> time steps of visual observations, corresponding <N₂, 1> intermediate actions.
The intermediate actions are: <Actioni, Act 1o
Note that when an object is picked up, appears close the bottom view.
Please analyze the visual differences between consecutive time steps of two trajectory segments, reply the changes between consecutive time steps in cach line explicitly. For example:
Trajectory Segment 1:
Timestep 0 1 (Executed action): Your analysis
Timestep 2 (Executed action): Your analysis
and so on
Trajectory Segment 2:
Timestep 0 to 1 (Executed action): Your analysis
Timestep (Executed action): Your analysis
and so on
The goal <IInIInutuutt ion>> there any diffcrence between Trajectory Segment and Trajectory Segment terms of achicving the goal?
Labeling Template:
Based the text below to the questions:
<Analyses from previous response>>
the goal better achieved the Trajectory Segment 1 or Trajectory Segment Reply single line of if the goal is better achieved in Trajectory Segment 1, or if is better achieved Trajectory Segment 2.
Reply -1 if the text is of there is difference.
Table Prompt Template RL-VLM-F used ALFRED environment.
18

---

Enhancing Rating-Basce Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Languea Models
Task Name
Task Description
Sweep Into
The green cube is in the hole.
Drawer Open
The drawer is opened.
Soccer
The soccer ball is in the goal.
Table 11. Task descriptions for CLIP Score used in Meta World.
Task Name
Task Description
Sweep Into
robot sweeping the green cube into the hole on the table robot opening green drawer
Drawer Open
Soccer
robot pushing the soccer ball into the goal
Table 12. Task descriptions for RoboCLIP used in MetaWorld.
Task Name
Task Description
Sweep Into
minimize the distance between the green cube and the hole to open the drawer to move the soccer ball into the goal
Drawer Open
Soccer
Table 13. Task descriptions for RL-VLM-F used in Meta World.
Task Name
Task Description
Sweep Into
place the green cube so that lies on the square hole open the drawer
Drawer Open
Soccer
place the soccer ball so that it lies inside the goal
Table 14. Task descriptions for ERL-VLM used in MetaWorld
Task Name
Task Description
Sweep Bowl
Sweep the green bowl beyond the red line.
Drawer Open
Open the drawer with the blue handle until the green line fully visible.
Pickup Banana
Pick up the banana on the gray plate.
Table 15, Task descriptions for ERL-VLM used in Real Robot environment.
19

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Languag Models
Task: Sweep Bowl
BC
Sparse Rewards
ERL-VLM
(a) The Sweep Bowl task relatively simple, where the agent learns push the bowl across the red line. As shown, all three methods achieve high success rates.
Task: Drawer Open
BC
0.27
Sparse Rewards
ERL-VLM
(b) Drawer Open task, both BC and Sparse Rewards failed to position the gripper correctly on the handle, In contrast, ERI--VIM successfully placed the gripper in the right spot and pulled the handle horizontally to complete the task.
Task: Pickup Banana
BC
Sparse Rewards
ERL-VLM
(c) In the Pickup Banana task, FRI--V.M successfully grasped and lifted the hanana from the plate. In contrast, BC and Sparse Rewards moved close the banana but failed close the gripper the position, resulting in lifting without proper grasping.
Figure 9, Visualizations of policy rollouts three tasks in the real robot environment.
20

---

Enhancing Bating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
E. Impact of MAE loss in Preference-based RL
We further investigate the impact MAE loss in handling corrupted labels RL-VLM-F (VLM-based PbRL) by replacing Cross- cooss-Entropy with MAE loss. The results, shown Figure 10, indicate that MAE provides a slight improvement. This suggests that the lower performance may stems more from the ambiguity of preference queries rather than noisy labels.
Sweep Into
Drawer Open
Soccer
100
100
100
Success Rate
75
bi-ed MARL
75
75
50
50
50
25
25
25
VuM-based PARL
VLM VLM-based P
VLM-based +
0
0
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
Timesteps (x10%)
Timesteps (x10%)
Timesteps (x109)
Figure 10. The impact of MAE on handling corrupted labels in RL-VLM-F.
F. Additional Results & Analysis
E1. Distribution of Ratings
The rating distribution for the tasks is shown in Figure 11. As shown, for the three MetaWorld tasks, the percentage of Good ratings increases the timestep progresses. For ALFRED tasks, the percentage of Average ratings increases over time, while the percentage of Good ratings remains almost constant. This is because, in ALFRED tasks, successful states occur at the end of the trajectory, so the VI.M tends to assign Good ratings mostly at the final step, resulting in an increase in number of Good ratings, although the increase is relatively small compared to other ratings.
Sweep Into
Soccer
Drawer Open
ALFRED
Percentage (86)
100
100
100
100
75
75
75
80
60
20
0.0
0.0
0.5
1.0
300
600
900
1200
Timesteps (x10%)
Timesteps (x10F)
Timesteps (x109)
Figure 11. Typical rating distribution over the course of training for ERI-VIIM agents. The first three plots show the distribution ratings in MetaWorld tasks, while the final plot displays the distribution in ALFRED tasks, aggregated across four different task types.
F2. Examples of Segments Labeled with Different Ratings
We provide examples of states in Meta MetaWorld World and trajectories in ALFRED, each labeled with different ratings, in Figures 15, 16, 17, and 18. The reasoning behind the VLM's ratings (obtained during the analysis stage of our prompts) is also illustrated in these figures. As shown, the VLM assigns appropriate ratings based on the status of the target object in relation to the task objectives specified in the task description.
E3. Visualization of the Learned Reward
We visualize the outputs of the learned reward models from ERL-VLM, RL- RLM-F, and CLIP along expert trajectories in three MetaWorld tasks and four ALFRED tasks, as shown in Figures 19 and 20. In MetaWorld tasks, CL.PP rewards are generally noisy and poorly aligned with task progress. While both ERL-VLM and RL-VLM-F exhibit increasing reward trends along expert trajectories, ERL-VLM aligns more closely with the ground-truth task progress and shows significantly less noise compared to RL-VLM-F. In ALFRED, ERL-VLM produces smoother and more consistent reward signals along expert trajectories than the other methods.
F.4. Consistency of Ratings
In practice, we observe that the large VLMs (e.g., Gemini) can produce stochastic ratings; identical segments may receive different labels, even when the temperature is set zero. To investigate the consistency and reliability of these ratings, we
21

---

Enhancing Bating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
repeatedly query the VLM along expert trajectories across different tasks. The results, shown in Figures 12 and 13, illustrate the degree of rating consistency. While some noise is present, the overall ratings along expert trajectories remain stable and reasonable. For example, by calculating the accuracy of ratings based on task success/failure (for 3 rating levels, we only count Bad and Good), we observe the following rating accuracies: for Sweep Into, 90 + 4.6; for Drawer Open, 65 + 3.8; and for Soccer, 66.8 + 3.7. The low variance across trials suggests that the inconsistencies are minor and can be effectively addressed by our MAE-basc objective.
Sweep Into (n-2)
Drawer Open (n- 3)
Soccer (n-2)
Different Thas
Time stop along the expert trajectory
step along expert
trajectory
Time stop along the expert trajectory
Figure 12. Consistency of VLM-assigned ratings for MctaWorld tasks: Using the expert trajectories from Figure 19, we perform 10 repeated trials to query the VLM for ratings (using the prompt template from Table 6, with the VLM temperature set to 0). The top row indicates task-specific success or failure at each timestep. The bottom 10 rows show the ratings from these trials along the expert trajectories (with every other timestep skipped for hetter visualization). The number of ratings, n, matches the value used in Figure 4.
Objett
PutObject
CoolObject
CleanObject
Tirm along the Mccpett
Time along the scpar ejectory
Tirr alory
trajecturyy
Time step along the Mapert Inajectory
Figure 1.3. Consistency of VLM-asiigned ratings for ALFRED tasks: Using the expert trajectories from Figure 20, we perform 10 repeated trials to query the VLM for ratings (using the prompt template from Table 7, with the VLM temperature set to 0). The top row indicates task-specific success or failure at each timestep. The bottom 10 rows show the ratings of trials along the expert trajectories. The number of ratings is equal to three across tasks.
The influence of rating levels on the consistency of VLM feedback in the Drawer Open task is further examined in Figure 14. With n 3 ratings levels, the VLM produces more consistent and intuitive feedback, with ratings generally increasing as the trajectory progresses toward task completion. However, with other values (e.g., inconsistency. For instance, with n 2, the VLM sometimes assigns Bad ratings even when the state reflects successful task completion. Similarly, with = 4, it occasionally assigns Very Bad or Bad ratings to successful states. We hypothesize that this behavior stems from inherent biases in the VLM. Based on these observations, we recommend performing simple consistency check when designing prompts for new tasks to guide the selection of an appropriate number of rating levels.
2 = 4), the VIM exhibits greater
Drawer Open (n-2)
Drawer Open (n 3)
Drawer Open (n-4)
Difforent Thails
12 Time step along the expert trajectory
12
30 40 00
Time step along the expert trajectory
Time step along the expert trajectory
Figure 14, VLM-assigned ratings for the Drawer Open task with f2, 3, 4). Using the expert trajectory from Figure 19, we perform 10 repeated trials to query the VLM for ratings (using the prompt template from Table 6, with the VLM temperature set to 0). The top row indicates task-specific success or failure each timestep. The bottom 10 rows show the ratings from these trials along the expert trajectories (with every other timestep skipped for hetter visualization).
22

---

Enhancing Bating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Task: Sweep Into
Gemini
Analyzing: image shows green cube not the square hole. The task is therefore incomplete. The robot arm needs to move the cube and position correctly over and within the hole to complete the task.
Rating: Bad
Prompt Table 6
Gemini
The image shows the green cube already placed within the square hole. Therefore, the task of "place the green cube so that lies the square hole" complete. Rating: Good
Analyzing:
Promptin in Table 6
Task: Drawer Open
Gemini
Analyzing: The image shows small, single-drawer drawer cabinet with the drawer closed. The task is open the drawer. Therefore, the task is not completed in this image. The robot arn would need to grasp the handle and pull forward to complete the task.
Rating: Bad
Prompt in Table 6
Gemini
Analyzing: The drawer is already slightly open, the task is partially completed. To fully complete the task of opening the drawer, the robot arm would need to grasp the handle and pull it towards itself until drawer fully extended. Rating: Aveage
Prompt in Table 6
Gemini
Analyzing: The image shows the drawer already in open state. Therefore, the task of opening the drawer is already completed. Rating: Good
Prompt in Table 6
Task: Soccer
Gemini
The image shows the soccer ball outside and in front of the goal. The task is not complete. The ball needs be moved into the space hetween the posts and behind the net order for it to be considered "inside the goal. Rating: Bad
Analyzing:
Prompt in Table 6
Gemini
Analyzing: The image shows the soccer ball already inside the goal. Therefore, the task of placing the soccer ball inside the goal is complete. Rating: Good
Prompt in Table 6
Figure 15. Examples of states with different ratings in Meta World tasks. We use the prompt template from Table 6 to obtain ratings. The images shown were used query VI.M, and the text outputs are directly taken from the model's response. Following (Wang al., 2024a), remove the robot from the images to facilitate the VL.M's analysis.
23

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Langune Models
Task Instruction: "Pick up the mug in the coffee maker"
Tinwacep
Einwateg
Elenatap
LLawatep
Stawatep
Tineacep Tamwacep
Einwstep
Fieescap
Tiaeatep 10
Ftewatep 11
Vineacep
Prompt in Table 7
Gemini
Analyzing:
Timestep 0 to (RotateRight): The robot rotates to the right, the view changes from the front of the counter to the side of the counter.
Timestep 1 2 hMoveAheed): The robot moves forward, the view shows more of the counter and the coffee maker.
Timestep heav:: The tobut moves forward, view shows more the counter and the coffee maker.
counter and coffee maker.
Timestep heeveA The robot
forward, view shows Timestep to (MoveAhred); The robot moves forward, the view shows
the counter and the coffee maker.
Timestup 5 to (MovaAhead): The robot moves forward, the view shows more of the counter and the coflee maker.
Timestep 6 to (MoveA head): The robot moves forward, the view shows more of the counter and the coffee maker.
Timestep to (MoveAheed):: The robot moves forward, the view shows more of the counter and the coffoe maker.
Timestep heava) The tobot moves forward, view shows more
counter and coffee maker.
Timestep 10 (RotateLaft): The robot rotates the left, the view changes from the side of the counter to the front of the counter.
Timestep 10 11 (LookDown): The robot looks down, the view changes from the front of the counter closer view of the coffee maker.
Timestep 11 to 12 (PickupObjec)): The robot picks up the mug. the mug is now close to the bottom of the view.
Rating: ("Bad", "Bad", "Bad", "Bad", " "Bad", "Bad", 'Bad", "Bad", "Bad", "Bad", "A 'Average" "Good")
Task Instruction: "Pick up the mug in the coffee maker"
Stnescep
TEawecap
Tlawsoep
Timestep
Prompt in Table
Gemini
Amalyzing:
Timestep 0 to (RotateLeftt: The agent tums left from facing the sink to facing the window.
Timestep (MoveA head): The agent moves closer the window.
Timestep 2 to (MoveAhcad) The agent moves closer to the window.
Timestep 3 to (MoveA The agent moves closer to the window.
Timestep 4 to (MoveA head): The agent moves closer to the window.
Timestep to heoveAhead); The agent moves closer to the window.
Timestep to (RotateLeft): The agent tums from facing window facing the coffee maker.
Timestep (RotateLLeft) The agent
left from facing coffee maker facing kitchen island.
Timestep 8 to (RotateRight); The agent tums right from facing the kitchen island to facing the coffee maker. Timestep 9 to 10 (RotateRight): The agent turs right from facing the coffee maker to facing the window. Timestep 10 to 11 SloveAhead) The agent moves closer to the window.
Timestep 11 to 12 (Rotatel oft): The agent tums left from facing the window to facing the coffee maker
Timestep 12 13 (PickupObject): The agent picks up the mug from the coffee maker.
Rating: ["Bad", 'Bad', "Bad', "Bad", "Bad", ; "Bad", "Bad", "Bad". "Bad", 'Bad", "Bad", "Bad", "Good"]
Task Instruction: "Pick up the mug in the coffee maker"
Prompt Table
Gemini
Analyzing:
Timestep to (RotateLeft): The agent rotates to the left, bringing counter into view. The bread is no longer visible.
Timestep to (MoveA The agent moves closer to the counter. The coffee maker and the mug larper the view.
larger, and the apple is now visible. Timestup 3 to heoveAhead): The agent moves closer to the counter, now very close to the coffee maker. The mug is now the primary ubject in view. Timestep 4 to 5 (RotateRight): The agent rotates to the right, bringing the sink into view. The mug is still visible, but the coffee makeris is mostly out of view.
Timestep 2 to (MoveA head): The agent moves closer to the counter, The coffee maker and mug are
Timestep to (LookDown); The agent down. The Timestep to (PickupObject); The agent picks up the bowl from the sink. The bowl large and the bottom of the view.
closer in the view,
mug is no longer visible.
Rating: ['Average", 'Average', "Average", "A 'Avergee" "Bad", "Bad", "Bad")
Figure 16. Examples of trajectories with different ratings in PickObject task in ALFRED. We use the prompt template from Table 7 to obtain ratings. The images shown were used to query the VL.M, and the textual outputs taken directly from the model's responses.
24

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
Task Instruction: "Pick up the tomato from the sink"
Prompt in Table 7
Gemini
Analyzing:
Timestep 0 to (RotateRight): The robot rotates to the right, the view changes from the counter to the side of the kitchen.
Timestep to 2 (MoveA head): The robot moves forward, the view changes to a closer view of the stove.
Timestep to (MoveA head): The robot moves forward, view changes to a closer view of the stove.
Timestep to heav:: The tobut moves forward, view changes a closer view the stove.
Timestep (MoveA head): The robot moves forward, the view changes
closer view of the stove.
Timestep 5 to (RotateRight): The robot rotates to the right, the view changes from the stove to the counter.
Timestep 6 to (RotateRight): The robot rotates to the right, the view changes from the countar to the side of the kitchon.
Timestep to 8 (MoveA head): The robot moves forward, the view changes to a closer view of the counter.
Timestep to (RotateLeft): The robot mtaaes to the left, the view changes from the side of the kitchen the counter.
sink. Timestep 10 11 (RolateRight): The robot rotates the right, the view changes from the sink to the side the kitchen.
Timestep 9 to 10 (MoveAbead); The robot moves forward, the view changes to a closer view
Timestep 11 to 12 heave: The robot moves forwart, the view changes to closer view of the counter.
Timestep 12 to 13 (RotateLeft) eft): The robot rolates to the left, the view changes from the side of the kitchen to the counter.
Timestep 13 to 14 (LookDown); The robut looks down, the view changes to a closer view of the sink.
Timestep 14 to 15 (PickupOhject): The rohot picks up the tomato, the tomato appears close to the bottom of the view.
Rating: ("Bad', 'Bad", "Bad', "Bad", "Bad", "Bad", 'Bad', "Bad", "Bad", "Bad', "Bad", 'Bad", "Bad", 'Average", "Good')
Task Instruction: "Pick up the tomato from the sink"
Timestap
Timeatep
Timestep 2
Timentep
Timentep
Timestop
Timestop
Timestop
Timestep
Timestep
Timestop
Timostop
Prompt in Table 7
Gemini
Analyzing:
Timestep 0 to (RotateRight): The view motats slightly to the right, now facing more of the floor and less of the counter.
Timestep (RotateLeftt) The view rotates back left, returning to roughly the same position in timestep 0.
Timestep (RotataRight): The view rotates to the right again, similar timestep
Timestep 3 to (MoveA head): The view moves forward, now showing wider view of the kitchen, including ladder and part of the fridge.
Timestep 4 to (Rotatelefft) The view rotates to the left, now facing the counter with the sink.
Timestep 5 to (RotateLeft): The view rutates further to the left, now showing the stove and window.
Timestep to (RotateRight): The view rotates back to the right, now facing the sink again.
Timestep (LookDownn): The tilts downwards, focusing
the sink and contents.
Timestep to (MoveA The view moves slightly closer to the sink.
Timestep 9 to 10 (MoveA bead): The view moves even closer to the sink.
Timestep 10 to 11 (PickupObject): The tometo, which was previously in the sink, is now at the bottom of the view, indicating it has been picked up.
Rating: ("Bad", 'Bad", "Bad', "Bad", "Bad", "Bad", "Bad", "Bad", "Bad", 'Average", 'Good")
Task Instruction: "Pick up the tomato from the sink"
Timestep
Esmentep
Tlawstep
Fimestep
Timextec
Tiaestep
Finastep
Tiaestep
Tinwstep
Prompt in T
Gemini
Analyzing:
Timestap 0 to :RotateLeft): The agent rotates to the left, now facing the sink more directly. The tomato and potato are more centered in the view.
Timestep 1 to :fot) The agent rotates further left, now facing away from the sink and towards the cabinets. The sink is partially out of view.
Timestop 2 to (MoveA The agent moves forward, closer to the cabinets. The sink no longer vistble.
Timestap 3 to 4 (MoveA head): The agent continues moving forward, still facing the cabinets.
Timostep 4 to 5 (MoveAhoad); The agent moves forward again, now close to the cabinets.
Timestep 5 to (RotateRight): The agent rotates right, now facing sink and counter again. The knife visible on the counter. Timestap 6 to (LookDown); The agent looks down, bringing the knife closer the bottom of the view. The sink is more visible. Timestep 7 to (PickupObject): The agent picks up the knife. It now large and at the bottom of the view.
Rating: ''Bad', "Bad", 'Bad', "Bad", "B Bad"" "Bad". "Bad", "Bad")
Figure/ 17. Examples of trajectories with different ratings in PickObject task in ALFRED. We use the prompt template from Table 7 to obtain ratings. The images shown were used to query the VL.M, and the textual oulputs are taken directly from the model's responses.
25

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Languaae Models
Task Instruction: "Open the fridge door, put the lomatio inside of the fridge. close the dour, open the dour. lake the tornato close the douee"
Prompt in Table 7
Gemini
A Analyzing:
Timestap 0 to (RotataRight); The robot
right, the view changes from the kitzhen counter the side of the fridge. Timestop 1 to 2 (MoveAhead); The robot moves forward, the view shows more of the fridge.
Timestep 2 to 3 (MoveA head): The rohot moves forward, the view shows more of the fridge.
Timestep 3 to
forwart, view shows Timestep 4 to 5 (MoveA head): The rohot moves forward, the view shows more of the fridge.
Moove The robot
of the fridge.
Timestep 5 to 6 MMovehhead); The rohot moves forward, the view shows more the fridge. forwari,
Timestep 6 to 7 (MoveA The robot
shows
the fridge.
Timeslep 7 tb 8 (MoveA head): The robol moves forward, the view shows more of the fridge.
Timestap 8 to 9 (RolataRight:: The robot rotates to the right, the view changes the front of fridge.
Timestep 9 to (OpenObjest): hee rboot opens the fridge door, the view shows the inside of the fridge.
Timestep 10 to 11 (PutOOjectt) The rohot puts the tomato inside the fridge, the tomato is now on the top shelf of the fridge.
Timestep to (Clos:Object): The robot closes the fridge door, view shows the front the fridge.
Timestep to (OpenObiect): The robol, opens the fridge dour, the view shows the inside of the fridge.
Timestap 13 to (PickupOhjecd;: The rboo picks up the tomato, the tomato is now close to the bottom of the view.
Timestep to (Cjose) robot closes the fridge door,
shows front the fridge.
Rating: ("Rad", "Bad", 'Bad', "Bad", "Bad", "Bad", "Bad", "Bad", "Bad", "Average", "Good", "Good", "Average", "Good", "Good")
Task Instruction: "Open the fridgo door, put the tomato inside of the fridge, close the door, open the doer. take the tomato out, close the door."
Prompt in Table
Gemini
A Analyzing:
Timestep 0 to 1 (RotateRight): The view rotates the right, the tomato is stationary.
Timestop 1 to 2 (MoveAhead): The moves forward slihtly, the tomato stationary.
Timestep 2 to 3 (MoveA head):
The view moves forward, the tomato is stationary.
Timestap 3 to 4 (MoveAhaad) The view moves forward, the tomato stationary.
Timestep 4 to 5 (MoveA head): The view moves forward, the tomato stationary.
Timestep 5 to 6 (MoveA The view moves forward, the tomato is stationary.
Timestep 6 to 7 (MoveA head): The
moves forward, the tomato stationary.
Timestep 7 to 8 (MaveA The view moves forward, the tomato is stationary.
Timestap 8 to 9 (RobataRight): The view rolates to the right, the tomato is stationary.
Timestep 9 to
(I))) The view looks up, the tomato stationary.
Timestep 10 to 11 (OpenObject): The fridge door opens, the tomato is stationary.
Timestep to (Putthject): The tomato is placed Inside the fridge.
Timestep to (CloseObject); The frädge door closed.
Timestap 13 to 14 (OpenObject): The fridge dooris opened.
Timestep 14 to (PickupObjert): (): The tomato picked up.
Timestep to bCjose The fridge dooris closed.
Rating: ("Dad", "Bad", 'Bad", "Bad", "Bad", "Bad", "Bad", "Bad", "Bad", "Bad", 'Average' "Good", "Good", "Average" "Good", "Good")
Task Instruction: "Open the fridge door, put the tomato inside of the fridge, close the door, open the door, take the tomato out, close the door."
Vimercap
Prompt in Table
Gemini
A nalyzing:
Timestep 0 to 1 (RotateRight) The agent rotates to the right, the view changes from the counter with mug, apple, golden statue, and bowl to view frtdge.
facing the stove
Timestep 1 to 2 (Movet The agent moves closer to the stove and fridge. The mug is now closer to the camera. Timestap 2 to 3 (MoveA The agent moves closer to the stove and fridge. The mug is even closer to the camera. mug is still held.
Timestep 3 to 4 (RolateRight): The agent rolates right, now facing the sink.
Timestap 4 to 5 (LookDown): The agent looks down, providing a better view of the mug and the sink.
Timestep 5 to 6 (LookUp); The agent looks up, providing wider view of the sink ares.
Timestep 6 to 7 (PutObject); The egent places the mug the sink.
Timestap 7 to 8 (ToggleObjectO0
The fauest is tumed on, filling the sink with water.
Timestep 8 to 9 (ToggloObjes:00), The faucel is turned off, the water stops filling tinsink. agent picks up mug from the sink. is close the bottom the view.
Timestep 9 to (PickupObject):
Rating: ("Dad"; "Bad', a "Bad' "Dad", 'Bad', 'Bad", " "Bad", "Bad", "Dad" "Bad")
Figure 18. Examples of trajectories with different ratings in CoolObject task in ALFRED. We use the prompt template from Table 7 obtain ratings. The images shown were used to query the VI.M, and the textual outputs are taken directly from the model's 's responses.
26

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
t-
t=10
t=15
t=20
t-25
t-30
t=35
Sweep Into
Sweep Into
Sweep Into
-1.0
1.0
1.0
ERL-VLM
RLVLMM.F
CLP Score
Leareed Rowordd
1.0
-0.8
GT Progress
Learned Reward
1.0
GT
-0.8
GT Task Progress
Learree Reweard
1.0
Task
0.88
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.0
0.0
0.0
20
30
Time stap along the expart trajectory
Time stop along the expert traj actory
Time stop along the axport trajectory
t-15
t=20
t-25
t-30
t=35
40
h45
t0
t-55
Drawer Open
Drawer Open
Drawer Open
1.0
1.0
ERL-ILM
RL-VLM-F
Score
Lesmedd Reward
GT Taik Progress
-0.8
Learned Beward
GT Task Progress
0.8
task Task Progress
Learnnd Reward
GT Task Progress
0..8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.6
0.6
GT Task
0.4
0.4
-0.4
0.4
0.4
0.2
0.2
0.2
0,2
0.2
0.0
0.0
0.0
0.0
0.0
10
20
30
40
50
60
10
20
30
40
50
60
10
20
30
40
50
60
Time stap along expart trajectory
Time
along expert traj actoory
step along expert trajectory
t=
t
t=10
t=15
t=20
t=25
t-30
t=35
t=0
Saccer
Soccer
Soccer
-1.0
1.0
ERL
RL Progress
CLP
Leained Ravard
1.0
Progress
-0.8
Tesk Pregress
Learned Reward
1.0
Task Progress
Learred Reword
1.0
Task Progress
0.8
0.8
0.8
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.0
10
20
40
60
10
30
50
60
20
30
40
50
60
Time step along the ort trajectory
Time stop along the expert traj ectory
Time Step along the expert trajectory
Figure 19. Comparison of reward outputs from FRI.-VIM, RL-VIM-F, and CLIP against ground-truth task progress in World tasks. The ground-truth task progress is computed following (Wang al., 2024a): for Sweep Into, is measured the negative distance between the cube and the hole; for Drawer Open, as the distance the drawer has been pulled out; and for Soccer, as the negative distance between the soccer ball and the goal. normalize both the reward values and ground truth task progress into the range of [0, 1] for a better comparison between them. The leameed rewards are averaged over three runs, with shaded regions indicating standard deviation. Images are rendered at corresponding timesteps from the expert trajectories.
27

---

Enhancing Rating- Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
t=O
t=
b=
t-12
t=15
t=17
b=18
t=19
Pickupōbject: "Pick up the mug in the coffee maker"
RLVEME
CLin
Leaned Rowara
Lo Roward
0.8
0.8
0.8
0.6
0.6
0.2
0.2
0.2
10
10
15
10
Time stop along the expert trajectory
Time step along the expert trajectory
Time step along the expert trajectory
t=0
b-1
tn2
tw3
bw4
t-5
PutObject: "Put the knife in the green the sink'
R.VEME
1.0
Loanee Rewara
Loarned Reward
0.8
0.8
0.8
0.6
0.6
0.6
0.2
0.2
0.2
Time stap dong the expert trapectory
Time slep along the espert trajectory
Time step eling the espert Trajectory
t=1
5=
t=16
t=17
t=18
t=19
t=20
t=21
t=22
CoolObjec:: "Open the fridge, put the cup in the fridge, dosee the fridge, wait, open the fridge, pick the cup, close the fridge"
VIME
uaaeed Now3ra
0.8
0.8
0.8
0.6
0.6
0.6
0.2
0.2
0.2
10
15
20
10
15
20
15
Time stop along the expert trajetoory
Time stop along the expert trajectory
Time stos along the expert trajectory
t=0
t=2
[=4]
t=
t=8
[=9
t=10
[=11
t=12
CleanOhject: "Put the coffee the sink. turn the water. turn off the water and pick the coffee
RLAMEF
CJP Stars
usanee Rov3ro
Leamer Bewwnd
0.8
0.8
0.8
0.6
0.2
0.2
0.2
10
12
10
12
10
12
Time stop along the expert trajectory
Time along trajectory
Time stop along the expert trajectory
Figure 20. Comparison of reward outputs from ERL-VL., RL-VIM-F, and CLIP across four AL.REDD tasks along expert trajectories. Ground-truth task progress is omitted is not available in ALFRED. We normalize the reward values into the range of [0. 1] for better comparison. Images are rendered at corresponding timesteps from the expert trajectories.
28


Reviews:
[{"review_id": "7iSiFTbKr4", "reviewer": "Reviewer_Rx7z", "summary": "This paper studies the problem of automated reward generation for RL policy training via VLMs. While prior work has shown that rewards can be extracted from VLMs either by preference (relative comparison) between two or more trajectory segments, or by using the representation itself as a distance metric, these approaches are often prone to instability (former) or local minima (latter). This paper proposes a framework for reward generation via VLMs based on a rating system (e.g. \"bad\", \"ok\", \"good\") rather than direct trajectory comparison, which (a) allows for more fine-grained quantitative feedback, and (b) circumvents the issue of training instability when sampled trajectories are highly similar. The VLM assigns ratings to trajectory segments (one or multiple image frames), which are then used to learn a reward model for downstream RL training. The authors find that naively training a reward model on the collected dataset leads to poor predictions due to its skewed data distribution (most early trajectories are expected to be bad in an online RL setting), and instead propose to use a stratified sampling as well as an MAE objective. Experiments are conducted on 3 tasks from Meta-World, as well as ALFRED. Results indicate that the proposed framework is effective at guiding online RL via learned rewards on the selected tasks, outperforming simpler baselines based on representation distance or pairwise preferences.\n\n## Post-rebuttal assessment\n\nI appreciate all the new qualitative analysis of the method provided in the rebuttal, and believe that it addresses my concerns. I have raised my score from Weak Accept -> Accept under the assumption that these new results will be included in the camera-ready version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and constructive feedback, and for recognizing the clear motivation behind ERL-VLM, the clarity of our writing, and the strength of our experimental results. We address your comments in detail below:\n\n- **Q1**. What do trajectories labeled with different ratings look like in practice?\n  \n  - **A1**. Sixteen examples of states/trajectories labeled with different ratings are shown as follows:\n    \n    - MetaWorld tasks (*Sweep Into*, *Drawer Open*, and *Soccer*), where each task includes two or three different states, each leading to different ratings assigned by the VLM. These examples are shown in Figure 1 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing).\n      \n    - ALFRED tasks with three different instructions, each paired with three trajectories leading to varying ratings. These are shown in Figures 2, 3, and 4 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing).\n      \n    \n   The reasoning behind the VLM’s ratings (obtained during the analysis stage of our prompts) is also illustrated in the figures. As shown, the VLM assigns appropriate ratings based on the status of the target object in relation to the task objectives specified in the task description.\n    \n- **Q2**. What is the typical distribution of ratings for the considered tasks?\n  \n  - **A2**. The typical rating distribution for the tasks is shown in Figure 5 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing). As shown, for the first three MetaWorld tasks, the percentage of *Good* ratings increases as the timestep progresses. For ALFRED tasks, the percentage of *Average* ratings increases over time, while the percentage of *Good* ratings remains almost constant. This is because, in ALFRED tasks, successful states occur at the end of the trajectory, so the VLM tends to assign *Good* ratings mostly at the final step, resulting in an increase in number of *Good* ratings, although the increase is relatively small compared to other ratings.\n- **Q3**. What does the output of the reward model look like in practice?\n  \n  - **A3**. The outputs of the reward models from ERL-VLM along expert trajectories in three MetaWorld tasks and four ALFRED tasks are shown in Figures 7 and 8 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing). We also include outputs from RL-VLM-F and CLIP for comparison. In MetaWorld tasks, CLIP rewards are generally noisy and poorly aligned with task progress. While both ERL-VLM and RL-VLM-F exhibit increasing reward trends along expert trajectories, ERL-VLM aligns more closely with the ground-truth task progress and shows significantly less noise compared to RL-VLM-F. In ALFRED, ERL-VLM produces smoother and more consistent reward signals along expert trajectories than the other methods.\n- **Q4**. The effect of #ratings.\n  \n  - **A4**. As mentioned in **Q1**, the VLM assigns ratings based on the status of the target object. For many tasks, the appropriate number of ratings can be easily determined (e.g., whether the task has been successfully completed is clearly observable from the visual state). However, for tasks where determining the optimal number of ratings is non-trivial, we use a trial-and-error approach. We demonstrate this process in Figure 6 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing), where we repeatedly query the VLM along an expert trajectory in the *Drawer Open* task.\n    \n    With $n=3$ ratings levels, the VLM produces more consistent and intuitive feedback, with ratings generally increasing as the trajectory progresses toward task completion. Small inconsistencies are mitigated through our MAE objective. However, with other values (e.g., $n=2$ or $n=4$), the VLM exhibits stronger inconsistency. For example, with n=2, the VLM sometimes assigns *Bad* ratings even when the state reflects successful task completion. Similarly, with n=4, both *Very Bad* and *Bad* ratings are occasionally assigned to successful states. We hypothesize that this behavior stems from inherent biases in the VLM. Based on this observation, we recommend performing a simple consistency check when designing prompts for new tasks to help select an appropriate number of rating levels.", "claims_and_evidence": "The main claims of this paper are (1) that their framework, ERL-VLM, is that it enables agents to learn new tasks using only a human-provided language task description, (2) that their proposed improvements to rating-based reward learning improve stability and thereby agent performance, (3) that their framework outperforms prior work on VLM-based reward generation, and finally (4) that their ablations identify key factors of their framework for good performance. I believe that all of these claims are reasonable and for the most part supported by convincing evidence. However, while it is true that their ablations identify key factors for performance, I find the analysis of *why* somewhat lacking; I go into more detail on this in the \"experimental designs and analysis\" section.", "methods_and_evaluation_criteria": "Yes. The improvements to rating-based reward generation are well motivated and backed by empirical evidence.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design appears fairly sound. While the number of simulated tasks is rather small, the authors validate their method across a combination of continuous (Meta-World) and multi-task language-conditioned discrete (ALFRED) control, as well as three simple manipulation tasks on a real robot. Experiments are conducted with 3 seeds; while additional seeds would make the experiments more sound, performance gains appear to be statistically significant as is. Both the VLM, reward model, and policy take RGB images as input which makes the framework broadly applicable.\n\nMy main concern with the paper in its current form is the lack of analysis on what exactly trajectories labelled different ratings look like, what the typical distribution of ratings is for the considered tasks, and what the reward model outputs for successful vs. unsuccessful trajectories. It would be informative to analyze this from both a quantitative and a more qualitative perspective. Based on the ablation on number of ratings in Figure 6, it appears to me that the ratings might actually correspond to task stages (e.g. reaching, grasping, and pulling for the Drawer Open task). This would explain why the number of ratings has a pretty drastic effect on policy learning, since many tabletop manipulation tasks like the ones studied here can be divided quite naturally into a small number of distinct stages. It feels important to analyze this in more detail, as it will provide insights into the rating-based framework proposed in the paper while it would also help inform readers how to select the number of ratings (task stages?) if applied to novel problems.", "relation_to_broader_scientific_literature": "The contributions of this paper, while somewhat incremental, are well motivated wrt prior work. The authors motivate their approach by providing insights into the limitations of current approaches for VLM-based reward generation, which is backed by empirical results.", "essential_references_not_discussed": "I believe that the discussion of related work is fairly comprehensive and I am not aware of any essential references that are not currently mentioned. It is however possible that I might have missed some.", "other_strengths_and_weaknesses": "**Strengths:** The paper is very well written and easy to follow. The discussion of related work appears to be thorough, and the experiments are fairly comprehensive and insightful. The problem is timely and likely to be of interest to the community.\n\n**Weaknesses:** There is limited quantitative and qualitative analysis wrt what exactly the ratings represent in the context of the chosen tasks; I believe that including some insights in this regard would greatly strengthen the paper. The technical content of the paper is somewhat incremental, so more analysis, insights, and takeaways would likely improve longevity and impact of the work.", "other_comments_or_suggestions": "It would be helpful to more explicitly mention that both VLMs, reward model, and policy takes RGB images as input. I had to read through the appendices to confirm that the policy was also image-based.", "questions_for_authors": "I would like the authors to address my comments in the \"experimental design and analysis\" and \"weaknesses\" sections above, especially my comments regarding the effect of # ratings, what the ratings and rewards look like in practice (quantitatively and qualitatively), and how they relate to task stages.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTKEUvA6Q8", "reviewer": "Reviewer_94ge", "summary": "The paper introduces ERL-VLM, a method that efficiently utilizes feedback from large VLMs like Gemini to generate reward functions for training RL agents. Instead of pairwise comparisons, it queries VLMs for absolute evaluations of individual trajectories on a Likert scale. This approach allows for more expressive feedback, reduces ambiguity, and ensures full utilization of queried samples. The authors also introduce enhancements to existing rating-based RL methods to address instability caused by data imbalance and noisy rating labels from VLMs.  Through extensive experiments across various vision-language navigation and robotic control tasks, ERL-VLM is shown to outperform prior VLM-based reward generation methods. Ablation studies are conducted to identify key performance factors and provide insights into its effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thoughtful review, and for recognizing the novelty of our method, the extensiveness of our experiments, and the strength of our results across a wide range of tasks and domains. We respond to your comments in detail below:\n\n- **Q1**. The method is highly dependent on the performance and accuracy of vision-language models. What if the vision-language model consistently exhibits understanding biases and inaccurate ratings in some complex scenarios?\n  \n  - **A1**. We acknowledge that biases and inaccuracies in VLM ratings can arise, particularly in complex scenarios. To mitigate these issues, we focus on designing effective prompts and selecting appropriate rating levels, which help guide the VLM’s output. Additionally, using higher-performing VLMs can further enhance the quality of the ratings. However, if a VLM consistently produces inaccurate ratings due to being undertrained on specific tasks or domains, the finetuning for the VLMs is required.\n- **Q2**. An accuracy metric for the reward function should be introduced to measure the degree of matching between the learned reward function and the real task objective. In environments with sparse rewards, what specific reward designs does ERL-VLM derive that lead to performance improvements?\n  \n  - **A2**. ERL-VLM performs better in sparse reward tasks because the reward model, learned from ratings, provides denser reward signals at meaningful states, rather than only sparse signals at the final state.\n\n    To measure the alignment between the learned reward function and the real task objective, we compare the reward outputs from ERL-VLM with the ground-truth task progress along expert trajectories. The results are shown in Figures 7 and 8 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing) . Outputs from RL-VLM-F and CLIP are also included for comparison. As shown, ERL-VLM rewards are more consistent and less noisy compared to the other methods. In sparse reward tasks, our approach enables reward shaping by assigning higher values to meaningful states. For instance, in the *CoolObject* task (Figure 8), ERL-VLM assigns higher rewards at timesteps when the agent interacts with the target object (e.g., the fridge), compared to intermediate navigation steps. This behavior arises from the VLM assigning higher ratings to critical subgoal states, as illustrated in Figures 2–4 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing).\n- **Q3**. ERL-VLM seems to rely on the task description. How should it handle multiple target tasks?\n  \n  - **A3**: ERL-VLM handles multiple target tasks (i.e., compositional task descriptions) by querying the VLM with a sequence of images rather than a single image, as illustrated in Figures 2-4 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing). We find that incorporating temporal context helps the VLM better understand the overall task structure, leading to more accurate ratings in compositional task descriptions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes. This paper conducts a series of experiments to evaluate the ERL-VLM method. In the simulation domain, it is compared with various baselines, showing obvious advantages in most tasks. Real-robot experiments demonstrate that this method can generate effective reward functions to facilitate policy learning. Ablation experiments analyze the contributions of various improvements to performance, revealing that the MAE loss, stratified sampling, etc., are effective, while label smoothing is ineffective. Experiments on different numbers of rating classes reveal the relationship between task characteristics and the optimal number of rating classes.", "relation_to_broader_scientific_literature": "1. Addressing the Challenge of Reward Design.\n2. Leveraging VLM Capabilities.\n3. Overcoming VLM-based Feedback Challenges.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths\n- A novel method, ERL-VLM, for learning reward functions from vision - language model feedback is proposed, providing new ideas and approaches for the design of reward functions in reinforcement learning.\n- Experiments are conducted not only on various types of tasks in simulated environments but also in real-world robot operation scenarios.\n- Detailed ablation experiments are carried out on each improved part of the method, clearly demonstrating the impact of each improvement measure on the overall performance.\n\n### Weaknesses\n- The method is highly dependent on the performance and accuracy of vision-language models. What if the vision-language model consistently exhibits understanding biases and inaccurate ratings in some complex scenarios?\n- An accuracy metric for the reward function should be introduced to measure the degree of matching between the learned reward function and the real task objective. In environments with sparse rewards, what specific reward designs does ERL-VLM derive that lead to performance improvements?\n- ERL-VLM seems to rely on the task description. How should it handle multiple target tasks?", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xThrjENpOV", "reviewer": "Reviewer_p9Em", "summary": "This paper introduces ERL-VLM, a straightforward yet effective method for learning reward functions by leveraging feedback from VLMs. By querying a VLM for absolute ratings of trajectory segments rather than relying on pairwise comparisons, the method aims to improve sample efficiency and expressiveness of the feedback signal. The authors augment this approach with stratified sampling and a mean absolute error (MAE) training objective to address issues related to data imbalance and noisy labels. Experimental results, spanning both simulated environments and real-world robotic tasks, demonstrate that ERL-VLM outperforms four baseline methods in six out of seven tasks, thereby examining the effectiveness of ERL-VLM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their in-depth review, and for recognizing the simplicity and effectiveness of our method, the strength of our results, and our efforts in real-world experiments. We address your comments in detail below:\n- **Q1**. Could you elaborate on the criteria for selecting trajectory segments from the replay buffer?\n  - **A1**. For MetaWorld, we randomly sample segments of length one (i.e., individual states) and use them to query the VLM. This approach is sufficient for the single-target nature of the tasks, as shown in Figure 1 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing). For ALFRED, since tasks are often compositional and critical information may appear at different points in the episode, we randomly sample an entire trajectory and use it to query the VLM to provide sufficient context for accurate ratings, as illustrated in Figures 2-4 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing).\n- **Q2**. Can you provide additional quantitative analysis on the consistency and reliability of the ratings across different tasks and scenarios?\n  - **A2**. The consistency of the VLM ratings is shown in Figures 9 and 10 at [Link](https://drive.google.com/file/d/19DtB_xCUhg8uTmTbE5bPDwXFoEn71PkL/view?usp=sharing). We repeatedly query the VLM along expert trajectories in different tasks. As pointed out by the reviewer, we also observe that identical segments may receive different labels. However, as shown in the results, the ratings along expert trajectories remain reasonable and consistent. For example, by calculating the accuracy of ratings based on task success/failure (for $n=3$ rating levels, we only count *Bad* and *Good*), we observe the following rating accuracies: for *Sweep Into*, $90\\pm4.6$; for *Drawer Open*, $65 \\pm 3.8$; and for *Soccer*, $66.8 \\pm 3.7$. The variance across trials is small, and these small inconsistencies can be mitigated through our MAE objective.\n- **Q3**. What is the rationale behind choosing a Likert scale for the ratings?\n  - **A3**. The rationale for selecting a Likert scale is as follows:\n  1. Human alignment: Since our reward labels are generated by a VLM designed to emulate human interpretation, a Likert scale aligns well with common human rating systems. This allows the VLM to map qualitative assessments (e.g., *Bad*, *Average*, *Good*) to structured numerical values, which would be more challenging to express meaningfully using an arbitrary continuous scale.\n  2. Stability and reliability: It is considerably more stable and reliable to prompt VLMs with structured classification tasks (e.g., “Rate the success of this grasp from 1 to 5”) than to request continuous scalar (e.g., “Give a score between 0 and 1”). Continuous scores often yield unstable or inconsistent outputs, as shown in prior work [1], whereas discrete classes reduce ambiguity, providing clearer grounding for ratings.\n  3. Scalability and prior work: Prior research in learning from evaluative feedback [2-5] often discretizes qualitative feedback into ordinal scales for reward modeling. These pipelines report that such formats are easier to scale, annotate, and integrate into training loops.\n- **Q4**: Discussion with some works use LLMs or VLMs as coding models to directly generate reward functions without computing similarity score.\n  - **A4**. The relevant works suggested by the reviewer utilize LLMs/VLMs as coding models that directly generate reward functions by accessing environment source code or privileged state information. In contrast, our method relies solely on visual observations and does not require access to internal environment representations. We will incorporate the works recommended by the reviewer ([6-8]) into our revised manuscript to enrich the related work section and more clearly position our method within the broader literature.\n\n[1] Wang, Y. et al. RL-VLM-F: Reinforcement Learning from Vision-Language Foundation Model Feedback, ICML 2024.\n\n[2] Knox, W.B. et al. Interactively shaping agents via human reinforcement: The tamer framework. 2019.\n\n[3] Warnell, G. et al. Deep tamer: Interactive agent shaping in highdimensional state spaces. 2018.\n\n[4] Arumugam, D. et al. Deep reinforcement learning from policy-dependent human feedback. 2019.\n\n[5] White, D. et al. Rating-based Reinforcement Learning, AAAI 2024.\n\n[6] Wu, Y. et al. Eureka: Human-Level Reward Design via Coding Large Language Models.\n\n[7] Li, X. et al. Self-refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics.\n\n[8] Chen, C. et al. AutoReward: Closed-Loop Reward Design with Large Language Models for Autonomous Driving.", "claims_and_evidence": "The claims are generally well supported by empirical results, as the improvements achieved over baselines are notable. However, some technical aspects, such as the details of the trajectory segment sampling and the validity of VLM ratings, could benefit from further clarification. In particular, it is unclear whether all sampled segments are informative, since early or mid-trajectory segments might not contain sufficient task-relevant information, and whether the observed teacher VLM accuracy (approximately 0.7 as shown in Figure 2(a)) adequately justifies its use.", "methods_and_evaluation_criteria": "The proposed method leverages a VLM for trajectory rating and two enhancement techniques to mitigate the intrinsic issues from the VLM (e.g., imbalance and imprecision). The workflow mainly resolves around how to enhance the feedback from the VLM, thereby optimizing the reward function. The major concerns from my end are about the segment sampling and the validity of the ratings generated by the VLM. \n\nWhile the approach is sound, there are two primary concerns:\n\n- Trajectory Segment Sampling: The use of random sampling without consideration of temporal dynamics might yield segments that do not adequately reflect the task’s critical moments. From Algorithm 1 Line 12, trajectory segments are randomly sampled from the replay buffer and fed to the VLM to acquire labels/rating. It is not clear whether the sampled segments are qualified to provide valid ratings. Namely, some segments from the beginning of the operation may not contain useful information for rating even from humans. Also, some tasks can be determined as good/succeed or bad/failed only at the very end of the operation, therefore, it is hard to justify a meaningful rating for the trajectory in the middle. It is not convincing that a random sampling approach without considering the temporal dependence and importance of the segments can collect trajectories efficiently and effectively.\n\n- VLM Rating Reliability: With a teacher VLM accuracy hovering around 0.7, further quantitative analysis is needed to explore the consistency and informativeness of these ratings across different scenarios.", "theoretical_claims": "This paper does not include theoretical proof, and the main contribution is not in such an aspect.", "experimental_designs_or_analyses": "- The authors conduct experiments across different environments and tasks. Success rate is used as the main metric to evaluate the quality of the reward function, which is sound. \n\n- As aforementioned, one concern is about the quality of the labels generated by the VLM. Since the output from generative models can be subjective and unstable due to the input prompt, namely, an identical segment may obtain different labels. \n\n- The authors conduct experiments in a real robot, which is commendable.", "relation_to_broader_scientific_literature": "The related work covers relevant methods and some of them are applied as baselines for comparison. It is worth noting that some works [1-3] use LLMs or VLMs as coding models to directly generate reward functions without computing similarity score. The authors may add some discussion regarding these approaches.\n\n[1] Eureka: Human-Level Reward Design via Coding Large Language Models\n\n[2] Self-refined large language model as automated reward function designer for deep reinforcement learning in robotics\n\n[3] AutoReward: Closed-Loop Reward Design with Large Language Models for Autonomous Driving", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The authors may justify more about using the Likert scale for rating, since, to the best of my knowledge, both floating numbers and Likert scale suffer a degree of uncertainty for evaluating/labelling.\n\n-  In section 5, the authors discuss the impact of the number of rating classes. From Figure 6, it seems the selection of the number of rating classes is somehow task-specific. The authors may elabrate more about this point.", "other_comments_or_suggestions": "- Some of the technical details in the Appendix regarding reward learning could be integrated into the main body to provide a clearer technical overview.\n\n- Although the experiments use only Gemini-1.5 Pro as the VLM, the choice is justified given its representative performance, and the authors acknowledge that extending to other VLMs is a potential future direction.", "questions_for_authors": "1. Could you elaborate on the criteria for selecting trajectory segments from the replay buffer? How do you ensure these segments provide sufficient context for accurate ratings, especially in cases where critical task information might appear only at the end of the trajectory?\n\n2. Can you provide additional quantitative analysis on the consistency and reliability of the ratings across different tasks and scenarios?\n\n3. What is the rationale behind choosing a Likert scale for the ratings, given that both discrete scales and continuous ratings can suffer from inherent uncertainties?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces a novel and effective method for learning reward functions in reinforcement learning (RL) using absolute ratings from large VLMs. The authors demonstrate ERL-VLM's superiority over existing VLM-based reward generation methods through comprehensive experiments across simulated (Meta-World, ALFRED) and real-world robotic tasks, achieving significant performance improvements in six out of seven tasks. The proposed enhancements, including stratified sampling and MAE loss, address critical issues like data imbalance and noisy labels, as shown in ablation studies. The rebuttal effectively addresses reviewer concerns, particularly regarding trajectory segment sampling and VLM rating reliability. The work is a meaningful contribution to the field.
Decision: Accept (poster)