GoalLadder: Incremental Goal Discovery with Vision-Language Models
Authors: Alexey Zakharov
Shimon Whiteson
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
vision-language models
Primary area: reinforcement_learning
TL;DR: Training reinforcement learning agents from a single language instruction using vision-language models.

Abstract: Natural language can offer a concise and human-interpretable means of specifying reinforcement learning (RL) tasks. The ability to extract rewards from a language instruction can enable the development of robotic systems that can learn from human guidance; however, it remains a challenging problem, especially in visual environments. Existing approaches that employ large, pretrained language models either rely on non‑visual environment representations, require prohibitively large amounts of feedback, or generate noisy, ill‑shaped reward functions. In this paper, we propose a novel method, GoalLadder, that leverages vision-language models (VLMs) to train RL agents from a single language instruction in visual environments. GoalLadder works by incrementally discovering states that bring the agent closer to completing a task specified in natural language. To do so, it queries a VLM to identify states that represent an improvement in agent's task progress and to rank them using pairwise comparisons. Unlike prior work, GoalLadder does not trust VLM's feedback completely; instead, it uses it to rank potential goal states using an ELO-based rating system, thus reducing the detrimental effects of noisy VLM feedback. Over the course of training, the agent is tasked with minimising the distance to the top-ranked goal in a learned embedding space, which is trained on unlabelled visual data. This key feature allows us to bypass the need for abundant and accurate feedback typically required to train a well-shaped reward function. We demonstrate that GoalLadder outperforms existing related methods on classic control and robotic manipulation environments with the average final success rate of $\sim$95\% compared to only $\sim$45\% of the best competitor.

GoalLadder: Incremental Goal Discovery with Vision-Language Models
Alexey Zakharov University of Oxford alexey. alexoy.zaktharovess wwkharovvcss.ooomooooooooo ox. ac uk
Shimon Whiteson University of Oxford shimon, whiteson@cs. ox ac uk
Abstract
Natural language can offer concise and luman-interpretien means of specifying reinforcement learning (RL) tasks. The ability extract rewards from language instruction can enable the development of robotic systems that can learn from human guidance: however, remains a challenging problem, especially in visual environments. Existing approaches that employ large, pretrained language models either rely on non-visual environment representations, require prohibitively large amounts of feedback, or generate noisy, ill ill-shaped reward functions. In this paper, we propose novel method, GoalLadder, that leverages vision-language models (VLMs) to train RL agents from single language instruction in visual environments. GoalLadder works by incrementally discovering states that bring the agent closer completing task specified in natural language. To do it queries a VLM to identify states that represent an improvement in agent's 's task progress and to rank them using pairwise comparisons. Unlike prior work, GoalLadder does not trust VLM's feedback completely; instead, it uses it to rank potential goal states using ELO-based rating system, thus reducing the detrimental effects of noisy VLM feedback. Over the course of training, the agent is tasked with minimising the distance the top-ranked goal in learned embedding space, which is trained unlabelled visual data. This key feature allows us to bypass the need for abundant and accurate feedback typically required train well-shaped reward function. We demonstrate that Goall adder outperforms existing related methods on classic control and robotic manipulation environments with the average final success rate of ~95% compared to only 455% of the best competitor.
Introduction
Reinforcement learning (RL) relies critically effective reward functions to guide agents toward desired behaviours. However, crafting reward functions by hand requires significant human labour and domain expertise. Fortunately, many RL tasks can be succinctly described in natural language (e.g., 'open the drawer", 'close the window"), from which humans can understand the task at hand and approximate reward function reach the goal state. In particular, extracting reward function from a language instruction requires understanding the semantics of the task description, associating its components with the given environment, and integrating this information with prior, common-sense knowledge. The rise of large language models (LLMs) brings hope of automating this process - given language instruction, can we automatically extract an effective reward function?
Indeed, LLMs can perform well on open-ended question answering, code generation, and be vast reservoirs of common-sense knowledge [ applicability to tasks involving images or other modalities [5] 7 g [O]. These properties make pretrained language models potentially suitable for providing learning signals reinforcement learning algorithms, such as by generating preference-based feedback [1] or outputting a reward
Moreover, multimodal models have even broader
function directly 12
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prior work attempts to use LLMs to generate reward functions given access to environment code or
interpretable state representations [3 12], limiting their applicability. Another line of work uses vision-language models MVL to define or learn a reward function given a language instruction and visual observations. In particular, two broad strategies have emerged: (i) embedding- -base 15 16 I7, which aligns visual observations with a language instruction in the embedding space a VLM; and (ii) preference-basee [Л]. which prompts a VLM to rank segments of an agent's s behaviour by how well they match textual specification, and trains reward function from the collected preference data.
Both approaches present certain challenges. Embedding- based methods employing CLIP [5] tend to yield noisy reward functions due to the mismatch between the CLIP training data and the observations from the tested environments [16]. Similarly, these methods require that every observation is embedded to produce rewards, making them expensive and inefficient in using large pretrained models. Preference-based methods like RL-VLMM-F [11 produce reward functions with less noise and better correlation with task progress but are nevertheless substantially noisy due to mistakes made by VLM when comparing trajectories. Erroneous feedback plagues the preference label dataset (at an unknown frequency) making it challenging to robustly train a well-shaped reward function without overfitting to mislabelled samples. Furthermore, although rrefreece-bbsse approaches are more sample-efficies efficient than embedding-bart -ased ones, they nevertheless require many VLM queries to train reward function that generalises well.
In this paper, we argue that, to be effective in practice, approaches employing VLMs for feedback must address two key issues: (a) robustness to noisy VLM feedback; and (b) query efficiency in using VLMs for feedback, given that repeated large-scale queries to VL.Ms can be prohibitively expensive.
To this end, we propose GoalLadder, an algorithm designed to address both of these challenges. Our method leverages VLM to incrementally discover environment states that bring the agent closer to completing a task. To avoid the problems of prior methods, GoalLadder performs repeated comparisons of a small subset of visual observations and maintains persistent, ELO-based utility measure (rating) over states. This process allows GoalLadder to progressively refine its estimates of state utilities without being severely affected by noisy VLM feedback. Furthermore, compared to previous methods, GoalLadder requires substantially fewer VLM queries to learn desired behaviours, since rewards are defined as distances between visual observations in learned embedding space that is trained on unlabelled data, allowing for reward generalisation to unseen states without explicit feedback. Using two classic control and five robotic manipulation tasks, we demonstrate that our method significantly outperforms prior work that utilises vision lision-lagguage models for RL, with the average final success rate of 995% compared to only ~45% of the best competitor. GoalLadder exhibits impressive performance even against the oracle agent that has access to ground-truth reward - -nealyy matching it across all tested tasks and convincingly surpassing it on one.
2 Related Work
Vision-language models Recent advances in large language models [88
19 20 demonstrate
impressive abilities for reasoning and common sense in text and other modalities such as vision
8 10]. Vision-language models trained on a joint visual and textual representation space, allowing the capabilities of LLMs to be applied within the visual domain. Nevertheless, existing VLMs suffer from poor spatial understanding (21). making it difficult reliably apply them to spatial reasoming tasks. Our approach takes this into account by design, using a robust pairwise evaluation strategy to reduce the effects of erroneous feedback on spatial tasks.
5 6
7
Language models in RL The use of language models in reinforcement learning has been pre- dominantly about making LLMs write code, particularly to design reward function [ 12). Although these approaches may improve as language models get better at code comprehension, it is unclear how they would perform in arbitrarily complex physical simulations or transfer to real-world environments. LLMs have also been used to provide reward signal in text-based environments
14
22 23
Vision- models in RL Several works explore the integration of VLMS in RL. One avenue using the embeddings of the VLMS directly the definition of the reward. For example, [16] use the CLIP model [5] to embed the task description specified in text, well as an environment's 's visual
2

---

observations, and define the reward as the cosine similarity between the image and text embedding. However, the resulting reward functions tend to noisy, which is often attributed to the mismatch between the training data of the VI.Ms and the environments they are applied to [16]. To this end, Baumli et al. [7 fine-tune CLIP model with contrastive learning to output the success of an agent in reaching text-based goal. Other papers use vision-languag models to align visual observations with language descriptions of a task (2 24 15 25 to perform incremental trajectory improvement from human feedback [26] or to elicit better task-relevant representations (27]
VLMs can also be used for direct feedback to learn reward function from VLM goal-conditioned preferences [TT]. In particular, Wang et al. [T] use VLMs to provide preference labels over states of visual observations conditioned on task description and use these labels learn a reward function with Reinforcement Learning from Human Preferences (RLHF) (28]. Although this work improves upon previous methods, noise in the learned reward function remains evident. Furthermore, for the experiments with robotic manipulation, the authors remove the robot from the image to improve the VLM's ability to identify goal states in object-oriented tasks. This trick, however, also reduces the amount of information VLM can use to identify incrementally better states. In contrast, GoalLadder solves the tasks without environment modifications and utilises an ELO-based rating system to reduce the effects of noisy VLM feedback.
3 GoalLadder
3.1 Preliminaries
Reinforcement learning We formulate the problem as a Markov Decision Process (MDP). defined as a tuple M A, R, 7. Here, S represents the state space of the environment, .A is the agent's 's action space, P : S x A X S + [0. defines the transition dynamics, R: S x A + [0, oc) is the reward function, and 7 E [o, 1) the discount factor. At each timestep I,, an agent takes an action a,, transitions to state s1+1, and receives reward rt. In our setup, we assume that the agent similarly receives an image observation o that represents the state of the environment. The goal of the agent to maximise the sum of discounted rewards in the environment, defined G = Ex kmo 7 r((s,, az).
Soft-Actor Critic We use soft-actor critic (SAC) (29 as the RL backbone of our agent. However, in GoalLadder, the reward function changes periodically new targets are being set over the course of training. The vanilla implementation of SAC can struggle with non-stationary reward given that it reuses past data from the replay buffer, which may not be up to date with respect to the latest reward function. As such, similar to [30], we periodically relabel all of the agent's 's trajectories with the latest reward function.
Assumptions We assume access language instruction, that succinetly describes the task of the agent in any given environment (e.g., "open the drawer"). Furthermore, we consider tasks whose goal can be represented by visual observation (not necessarily unique).
3.2 Method
Overview We propose GoalLadder, which uses dision-language model incrementally discover states that bring the agent closer to completing a task specified in natural language. GoalLadder uses VLM in two ways: (i) to identify candidate goals, denoted as g. to be put into a ranking buffer B, = .2....? for further evaluation, and (ii) to rate the candidate goals, with rating denoted as e, based on their proximity to the goal state specified in language. The candidate goals are discovered and ranked online, as the RL. agent trains and collects new observations according to its latest SAC policy. Over the course of training, the top-ranked goal serves as the target state that the RL agent is tasked with achieving.
GoalLadder involves several stages that are expanded upon in Sections 3.2.1
1. Collection: the RL agent collects an episode by interacting with the environment according to its current SAC policy.
2. Discovery: a VLM is queried to determine whether any of the collected observations represent improvement over the current top-rated candidate goal.
3

---

Collect trajectory
(2) VLM compares trajectory against top-rated goal
(3) Place goal buffer
User: which
image the
Rating:
agent closer goal?
Rating: 620
X
Ratting 800
Trajectory
New
Current best
Ratingg: 850 -> 860
R₂ d(s((o+1)) ((0")
0²
Rating: 820
810
X
(6) Update reward butfer
(5) Select best candidate target
(4) Compare pairs of candidate goals
Figure 1: Summary of GoalLadder. (1-2) An agent collects a trajectory using the current SAC policy, Trajectory observations are uniformly sampled and compared against the current best candidate goal in the buffer. (3) If successful, they are placed into the candidate goal buffer. (4) Pairs of candidate goals are sampled and compared against each other using a VLM. Their ratings are updated after each comparison. (5-6) Periodically, the target state is updated with the current top-rated goal from the candidate buffer. Reward function is defined as the negative distance to the current best candidate goal in learned embedding space; the RL agent is then trained with this reward. The process returns to step and repeats.
3. Ranking: a VLM is queried with sampled pairs of existing candidate goals from the buffer for comparison and their ratings are updated accordingly.
4. Training: The agent is trained to minimise the distance to the top-ranked candidate goal in a learned embedding space.
Together, the stages represent repeating process, in which the RL agent trains and collects new observations. while candidate goals are being discovered and ranked. visual illustration of GoalLadder operations is provided in Figure I
3.2.1 Discovery of candidate goals
To succeed, GoalLadder must address the question: Which states are worth evaluating as candidate goal states? Since capacity query the VLM is limited, need to filter out irrelevant states before they are entered into the goal buffer for detailed evaluation. This ensures that the VLM focuses on the most promising states, avoids wasting compute obvious or trivial cases, and more quickly converges on the final goal. To this end, we maintain a buffer of candidate goals, B,, where each candidate goal gi consists of image o; representing the state and the goal's rating e; such that (os, ex). Given the current top-rated candidate goal,
g" arg max Ci where g" (o" e*), g:EB,
and randomly sampled state image 0j from the agent's 's latest collected trajectory, the VLM is asked decide which of the two images, o" or oj. represents a state that closer to the language-specifer task goal, 1. We denote the VLM output by
y = VLM "),,)
E 1-1,0,1),
where y 1 indicates that there is no decision the images equivalent, y O means is deemed a better goal than oj. and y = means o; deemed a better goal than g"
If 1, the newly sampled state 0j is inserted into B, as a fresh candidate goal with standard initial rating. Conversely, y or y 1, 0j disregarded in order keep the goal buffer focused on higher-quality candidates.

---

3.2.2 Rating of candidate goals
Once the goal buffer is large enough, GoalLadder begins querying a VLM to compare the candidate goal states and get a better estimate of their true rating. Specifically, a pair of candidate goals is sampled, (gi, 91 gg. and the same querying process, as in Section 3.2.1 is used.
Ideally, our goal ratings would have two main properties: (1) be robust to noisy VLM outputs and (2) allow for quick updates to a candidate goal's rating when new and better goals are discovered. In GoalLadder, we draw inspiration from the ELO chess rating system [31] to maintain a robust, adaptive measure of each candidate goal's utility. In chess, players' ratings are updated after every match based on two factors: the observed outcome (win, lose, or draw) and the expected outcome given the players' current ratings.
In ELO, the expected score for candidate goal Ii against 9j is given by the logistic function:
F 1+ 10/e--///'
where e, and ej are the current ratings of and gj. respectively, and C' is constant controlling how sensitive the expected scores are to rating differences.
Given an observed result S, E (-1,1,0) ("win," " "loss," or "draw"), we update the rating of gi as:
Gg + e + T(S - F.):
and similarly for 9j- Here, T' is constant that governs how quickly ratings are adjusted.
The ELO rating system incrementally absorbs noisy VLM comparisons, adaptively adjusts ratings when new evidence shows a goal is better or worse, and continuously refines its estimates to converge on a stable hierarchy of candidate goals aligned with the iinguage-specffiee target.
3.2.3 Defining the reward function
Once a candidate goal 9² E By emerges as the highest-rated goal, we treat its image o" as the agent's current best guess for the true task objective. As such, we wish to encourage the agent bring its state as close as possible to o* in an appropriate metric space. However, defining a distance directly in an environment's s intrinsic state-space may be unreliable given patirooment-tt-evviioee
differences.
To address this, GodlLadder learns a visual feature extractor to produce a compact latent representation of environment observations. We achieve this using variational autoencoder training objective [32
C = (z)lo))))) lo 2x) + DRL. (va ox) || [(=))
where the encoder (-) is trained to produce a latent vector 2, representing observation og. We implement the feature extractor using simple convolutional neural network architecture, the details of which can be found in Appendix AA
By training on a diverse set of observations the agent encounters, we obtain a latent space that captures salient visual (and potentially semantic) features of the environment, Finally, we define the agent's 's reward at time step to be:
R(s: a-11) = d(2,,²)) =))
where 2, (o:).. : (bo)), and d(-.-) -) the Euclidean distance. As such, the agent incentivised to minimise the distance to the top-rated goal g"
As the candidate goals are being discovered and rated during training, we periodically update our reward function and relabel all stored transitions (30) with respect to the top-rated candidate goal. Periodicity ensures that sudden or erroneous changes in the top-rated goal do not significantly destabilise the training process, while also keeping the best goal estimate up-to-date.
3.2.4 Implementation details
We Gemini 2.0 Flash
VLM backbone. The top-rated candidate goal is selected the new target every L = 5000 environment steps. The goal buffer size is capped at |B, = 10, as the lowest
/ woa...goolle... http//////aaaaa//////////////oaa

---

Algorithm 1: GOALLADDER: Pseudo algorithm
Input
Task description 1, experience buffer D, SAC, VIM,, reward update schedule T. Initialise candidate goal buffer B, with random observations
while training do
Collect episode & ~ and store in D // Discovery and Ranking
if mod K 0 then
Sample M trajectory observations for, (01....,..1 oorr
Compare each o; (oi...., 0Mr against top-rated candidate using VLM and /
Update goal buffer B based on the feedback
Compare M pairs of (gi: (i) Bg using VLM and
10
Update candidate goal ratings, eg)
end
12
// Update reward function if mod then
13
14
Select the top-rated goal from the buffer, g" = arg max (o C Update rewards in experience buffer 'D using ||<((0) tb(0") )|2
15
16
end
// Train
18
if mod 1 = 0 then
19
Update SAC
20
Update & using a minibatch of observations from D
21
end
22 end
rated candidate goals removed every L steps. This ensures the comparisons remain focused on the most promising states. VLM queries are performed every environment steps with queries per feedback session (see Section 4 SAC gradient update is performed after every environment step. Finally, the VLM prompts are standardised using a single template to ensure consistency and ease of use Algorithm shows the stepwise, high-level operations of GoalLadder. See Appendix A for further implementation details, including hyperparameters, architectural details, and used computational resources.
4 Experiments
We analyse the performance of our algorithm in the scope of continuous control tasks, from classic control more complex robotic manipulation. Specifically, we aim to answer the following questions:
1. Can GoalLadder effectively discover the best goal over the course of training?
2. Can GoalLadder solve continuous-control tasks?
3. Is GoalLadder more sample-efficient with respect to VLM queries than related methods?
We run variety of centinuous-contrr experiments investigate GoalLadder's performance. In particular, we use two classic control environments (CartPole, MountainCar) from OpenAl Gym [33 and five robotic manipulation environments (Drawer Close, Drawer Open, Sweep Into, Window Open, Button Press) from the Metaworld suite (34. Importantly, unlike prior works 16 11I, we do not perform any special environment modifications help VLMs discem task progress. We use feedback rates of = 2000 with M = for OpenAl Gym environments and K 500 with M for the Metaworld environments.
Environments

---

Mounlain Car
Drawer Open
stepo
abter viepa
steps
stepe
stepa
stopa
sike stepa
F
ofter 2k stops
afte Scetppe
afse strp
aftr Sk step
aher 28 strpe
afee 50k steps
ohher 88k eteps
Drawer Close
Window Open
Figure 2: Top-rated candidate goals during training. Top-rated candidate goals in the Mountain Car environment follow a natural progression of the car getting closer the top of the hill. Similarly, in the Metaworld tasks, GoalLadder rapidly discovers states that bring the robotic arm closer to the true task objective, until the best goal discovered.
Baselines
We compare GoalLadder to the following baselines:
Ground-truth reward (Oracle): A strong baseline in which the RL algorithm is trained with the environment's intrinsic reward. This benchmark acts the oracle and should serve as an upper bound on the performance of our method.
Uses CLIP [S] to embed the task description and an image observation and defines the reward as the cosine-similarte between the two embeddings.
VLM-RM 16:
RoboCLIP 13] Employs a pre-trained video-language model, S3D [35 to compute rewards based on the similarity between an agent's 's video trajectory and either reference video or a text description. To ensure fair comparison, we use the text version of RoboCI.IP.
RL-VLM-F (ЛI) Uses a VLM to compare pairs of images with respect to language instruction. Unlike GoalLadder, RL-VLM-F utilises VLM feedback to train a reward function from the collected labels. For this baseline, apply the same feedback rate used in GoalLadder and use the same vision-language model as backbone Gemini 2.0 Flash
4.1 Can GoalLadderr discover the best goal?
Figure shows the evolution of the top-rated candidate goals the buffer over the course of the training for the MountainCar and Metaworld environments. Despite the VLM's tendency make mistakes in its comparisons, the best goal consistently rises to the top of the ranking. Based on our experiments, we confirm that GoalLadder can effectively discover the target goal given the language instruction for all tested environments.
goss
ELO
2420
Similarly, Figure 3 shows the evolution of the candidate goal ratings in the buffer for the Metaworld Window Open environment. During the initial discovery stage (until 50k steps), there is no clear winner and the ratings tend similar. Upon the discovery of a clear winner (at around the best
120
Figure 3: Candidate goal ratings over time in Window Open. Different colours indicate different candidate goals present in the buffer. GoalLadder quickly singles out the best goal once it is discovered.
50k steps), GoalLadder quickly singles goal for the agent to pursue. Figure 5 provides further insight by visualising the entire goal buffer over the course of training for the Drawer Open task. Two trends can be observed: (i) the buffer approximately ordered by how close the agent is to achieving the task in each image, and (ii) the buffer becomes progressively filled with better states as training goes on. We observe similar behaviour for all other environments.

---

Ground-ruth reward (Orazie)
VUX-RM
RoboCLIP
GoelLarder (Ours)
CariPole
MountainCar
DrawerClose
1.0
40000
0.8
Success Rate
Reward
-1000
0.6
-2000
1000
0.2
-3000
0.5
1.0
1.5
2.0
2.0
0.2
0.4
0.6
0.8
0.0
0.2
04
0.6
0.8
100
Training Steps
Training Staps
x10
Training Steps
x10
WindowOpen
DraverOper
ButtonPress
Sugges Rate
0.8
0.0
0.8
0.5-
0.6
0.4
0.4
0.4
M
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.4
1.0
0.2
0.6
0.0
0.0
0.4
Training Staps
Triining Staps
Training Steps
Suroess Rate
Training
environments
Figure 4: GoalLadder performance against baselines on OpenAl Gym and Metaworld environments. Shaded regions represent the standard deviation over seeds. Averaged across all tasks, GoalLadder achieves a mean success rate ~95%, compared to the next best competitor performance of ~.55% achieved by RL-VLM-F.
4.2 Can GoalLadder solve the tasks?
Figure 4 shows that GoalLadder effectively solve continuous control tasks given only language instruction, reaching mean success rate of ~955% averaged across all tasks. Curiously, our method nearly matches the performance of the ground-truth reward baseline and convincingly surpasses for the Drawer Open task. Empirically, we find that sometimes the oracle agent learns to reach the drawer, but does not learn to pull the handle to open We believe this once again highlights the difficulty of manually designing effective reward functions an issue that was also previously discussed in Wang al. .
The preference-basee and MountainCar from the OpenAl Gym suite, well as Drawer Close from Metaworld. However, it begins to struggle with the more complicated Metaworld tasks. We attribute this poor performance of RL-VLM-F to the fact that it requires more feedback to learn an effective reward function, as well as ways to mitigate the effects of noisy labels in the preference data. Similarly, VLM-RM and RoboCLIP struggle to learn in most of the tasks, which is in line with previously observed limitations of these methods .
baseline, RL-VLM-R, manages to solve the relatively easy tasks - Cartpole
4.3 How sample-efficient is GoalLadder?
Using large pretrained models is expensive, so methods that rely on them for feedback must do so efficiently. While embedding-basee approaches, like VLM-RM [16], all environment observations for reward calculation, preference-batte efficiency by training reward function from limited amount of feedback, in the hope that the learned reward function generalises to unseen states. By contrast, GoalLadder produces reward signals that generalise to unseen states by defining the reward in terms of distances in an embedding space (Section 3.2.3) trained with unsupervised learning. Utilising the full extent of agent's experiences, rather than just the labelled observations, for reward definition allows CoallLaddes to approximate state utility without collecting large amounts of feedback. Averaged across all Metaworld tasks, GoalLadder
methods, like RL-VLM-F [1I], can offer better sample

---

training
SK d4
19t
20th depa
Lowest
Figure 5: Visualisation of the goal buffer in GoalLadder over the course training. Each line demonstrates the candidate goals present in the buffer after an indicated number of environment steps. The candidate goals are ordered by ELO rating, from left to right.
preference- based learns to solve them after only ~4500 queries. For comparison, PEBBLE 30
method that uses ground-rruth reward preferences, achieves the same performance after attaining average ~~1000 preference labels on the same tasks. Indeed, we observe that GoalLadder boasts superior sample efficiency compared to the tested baselines. Performance curves in Figure[Jillustrr iillustrate that RL-VLM-F struggles to learn the tasks using the feedback rate of GoalL while VLM-RM, that embeds all collected observations, fails almost entirely.
Discussion
5.1 Limitations
As mentioned in Section 3.2, we assume that the progress or success task can be identified from a single image, limiting the application of method to environments with static goals. Nevertheless, we believe that future work could extend GoalLadder to video- bascd settings. GoalLadder also largely relies visual feature similarity for reward definition. In some settings, visual similarity between observations could be a limiting proxy for the underlying state similarity or task progress. Here, more advanced visual embedding techniques could be used instead. Lastly, while method could always benefit from evaluation on more env environments, the costs of using VLMs limit what is feasible.
5.2 Broader impacts
Reinforcement learning from language instructions opens up the door for cumman-ineerpretbbl ways to interact with robotic systems the real world. At the same time, robotic systems trained with the assistance of large pretrained language models have the potential inherit any existing biases of these models. Though the scope of this paper is limited to object-oriented environments, we would caution against the application of such models in settings with real humans, until this potential issue is thoroughly investigated.
5.3 Conclusion
We introduced Goall-adder, method for training reinforcement learning agents from vision-language model feedback using a single language instruction. GoalLadder systematically discovers new and better states that take the agent closer the final task goal over the course of training. Our method demonstrated impressive performance gains against prior work, while nearly matching the performance of the oracle agent with access ground-truth reward. We argued that implementing more comprehensive pairwise comparison technique and using a learned embedding space for reward definition allowed GoalLadder be robust to noisy feedback and significantly more sample-efficient compared to prior methods.
GodlLa offers several interesting directions of future research. Firstly, believe that method could be extended to video-based settings. depending on the capabilities of the existing vision-language models for video comprehension. Secondly, believe GoalLadder would benefit from more advanced visual feature extraction techniques or from building a more meaningful latent space, for example using self-supervised learning.

---

References
[I] Qiyuan He, Yizhong Wang, and Wenya Wang. Can Language Models Act as Knowledge Bases Scale?, February 2024, URL http://arxiv. http///xiiv..rrr/a/////2//22 14273 arXiv:2402.14273 [es].
[2] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? In Pro- ceedings of The 4th Annual Learning for Dynamics and Control Conference, pages 893-905. PMLR, May 2022. httph:tps/////m hptpps///pocceeingg.. mlr. press/ ///p//vvvv//////e cui Maaiiaaa.. html ISSN: 2640-3498.
[3] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language Models as Knowledge Bases?, September 2019. URL http://arxiv.o org/ abs/ / 1909. 010066 arXiv:1909.0106) [es].
[4] Anastasia Kritharoula, Maria Lymperaiou, and Giorgos Stamou. Language Models as Knowl- edge Bases for Visual Word Sense Disambiguation, October 2023. URL http::/araxxi...oo org/
abs/2310.. 01960, arXiv:2310.0199 [cs].
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference Machine Learning, pages PMLR, July 2021. URL htttps:/pproceedingg mlr. (heess/v139/radffrdrd html ISSN: 2640-3498.
[6] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 4904-4916. PMLR, July 2021. URL UtL/https///prrceeeiii mir
aress/v139/53 /1ll//////111bbbb ia21b. html
ISSN: 2640-3498.
[7] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained Interactive Language- Image Pre- Training, November 2021. URL http://arxiv wtgg/as//aaa 07783 arXiv:2111.07783
[es].
[8] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to Prompt for Vision- Language Models. International Journal of Computer Vision, 130(9):2337-2348 September 2022. ISSN 405. doi: 10.1007/s1 111007/s11223-022--000 1263-022-01653--11 URL 1007/ 2007/s11263-1 022-
[9] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. CLIP-Adapter: Better Vision-Language Models with Feature Adapters. Inter- national Journal of Computer Vision, 132(2):581-595 February 2024. ISSN 1573-1405. doi: 10.1007/s1
2223226--00891111111133313339911 0rt8://2.00000///...a//aaaa//a/0 /ttp:/ppp///a.....//001100/01000 1007/ /1107//11133... 023- 01891x
[10] Florian Bordes, Richard Yuanzhe Pang. Amurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie. Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri. Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, and Vikas Chandra. An Introduction to Vision Lision-Language Modeling, May 2024. arXiv:2405. 17247 [cs].
URL htttp://arxiv.. www.arwww.....aaaaa...a.eeoooeee /wt.///...... 17247
[11] Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RI--VI--F: Reinforcement Learning from Vision Language Foundation Model Feedback, March 2024. http:tt://arxe wwwppxwwww.wr/////////22// 03681 arXiv:2402.0368 [es].
10

---

[12] Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models, April 2024. URL http://arxiv... orgg abs/ 2310. 12931 arXiv:2310.1291) [cs].
[13] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang- Lee, Montse Gonzalez Arenas, Hao- Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng. Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to Rewards for Robotic Skill Synthesis, June 2023. URL http://arxiv. wrw.:/..rgg/..... 2306 08647 arXiv:2306.0866 [es].
[14] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning, May 2024. URL http:h/tt////xi. org/ wwwp:xi...ooo//aaae
11489. arXiv:2309.1 11489 [cs].
[15] Sumedh A. Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem BIyIk, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. RoboCLIP: One Demonstration is Enough to Learn Robot Policies, October 2023. URL arXiv:2300007899 [cs].
wwwgg/bb//aa 2310. 07899
[16] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- Language Models are Shoo-Shot Reward Models for Reinforcement Learning, October 2023. URL
12921 arXiv:2310. 12921 [cs].
[17] Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebas- tian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, Clare Lyle, Hussain Masoom, Kay McKinney, Volodymyr Mnih, Alexander Neitz, Fabio Pardo, Jack Parker-Holder, John Quan, Tim Rocktäschel, Himanshu Sahni, Tom Schaul, Yannick Schroecker, Stephen Spencer, Richie Steigerwald, Luyu Wang, and Lei Zhang. Vision-Language Mod- els as a Source of Rewards, February 2024. URL http::ttttax:tt httt:://arxiv.or//bbb//222 2312. 01111.1..1777 arXiv:2312.0918' [es].
[18] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. ISSN 1533-7928. URL http: wwwo/////g../eoe http//pprss///// v24/ 22 h1m4.html
[19] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023. UtRLhhtpp:/aare //tp:iv/////vvv/a org/ htee/ab//220 abt///ass///.//////////////////a 2302. 13971 arXiv:2302.13977 13977 [cs].
[20] OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Bernadett-Shapiro Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory
11

---

Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti. Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider. Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Knueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu. Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano. Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Gi- ambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Paviov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, C. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, March 2024. http://p///artt wrgg harg//bb//a/ 2303 08774 arXiv:2303.0877. [cs].
[21] Amita Kamath, Jack Hessel, and Kai-Wer Chang. What's "up" with vision-language models? modelss Investigating their struggle with spatial reasoning, October 2023. utLhhtt:::// org/ abs/2310. 19785, arXiv:2310.1978 [cs].
[22] Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Lue Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic Motivation from Artificial Intelligence Feedback, September 2023. URL ptttp://arxiv: org/ abs 2alps///310..00 00166 arXiv:2310.00166 [cs].
[23] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward Design with Language Models, February 2023. URL http://arxiv http:///xxii.org/bbbba. 2/////33330000 arXiv:2303.00001 [cs].
[24] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge, November 2022. URL (RI./ttp::///// //atii org org/abs/2206 08853 arXiv:2206.08853 [cs].
12

---

[25] Taewook Nam, Juyong Lee, Jesse Zhang, Sung Ju Hwang. Joseph J. Lim, and Karl Pertsch. LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers, December 2023. UtI.htpp://arxii org/ abs/ 2312. 08958 arXiv:2312.0899 [es].
[26] Zhaojing Yang, Miru Jun, Jeremy Tien, Stuart J. Russell, Anea Dragan, and Erdem Bryık. Trajectory Improvement and Reward Learning from Comparative Language Feedback, October
2024. URL (RLttp////axxx org/ abs/2410 06401 arXiv:2410.06401.
[27] William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-Language Models Provide Promptable Representations for Reinforcement Learning, May 2024. URL httpp:////rx
org/abs htg/abb/2402..000 02651 arXiv:2402.02651 [cs].
[28] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc..
2017. URL https://procceedings neurips. cc/ wwtpsscrcpppppppeeeeeeeeeee files/lleppppe pappr__fllesppperr///a200 pttss/pppa//////ss/// (b220calaa000e11199444440ddeeee Abstsaat.tttttttttamaaaa
[29] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-
Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, August 2018. arXiv:1801.01290 [cs].
URLptttp///axxi wwtpoo///a//..o1//1111111
[30] Kimin Lee, Laura Smith, and Pieter Abbeel. PEBBLE: Feedback-Effiiien Interactive Rein- forcement Learning via Relabeling Experience and Unsupervised Pre-training, June 2021. URL
httt::/arxiv.co org hbse//bs///a 2206. 05091 arXiv:2106.0555 [cs].
[31] Arpad E. Elo and Sam Sloan. The rating of chessplayers past and present. Ishi Press International, 2008. URL https: http::://o...ii....ppp///o////// ptt.aacpppcrrdd11102220111333331
www.
[32] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, May 2014. URL
6114 arXiv:1312.6114 [cs, stat].
[33] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym, June 2016. URL http::/arxiv. .rg. h/tg//bb//// abs/1606 1606 01540 arXiv:1606.015400 [cs].
[34] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A Bench- mark and Evaluation for Multi-Task and Meta Reinforcement Learning, June 2021. URL
attp://arxiv. www....org/bb///11i.. 10897 arXiv:1910. [cs].
[35] Wei Zhong and Manasa Bharadwaj. S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for ow-Memory GPUs, June 2024.
ab4/
20314 arXiv:2405.2031 [cs].
[36] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. URL http:://rxivy. org/ abs//412......... 1412. 6980
arXiv: 1efi::141..00 1412.6980 [cs].
[37] Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. February 2017. URL https:: /toppse//openrvvew net/
forum?id=Sy2fz29gg]
[38] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Marc Szafranicc, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Ass- ran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang-Wer Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, February 2024. http://:///arx: org/abs/ 2304 07193 arXiv:2304.07193.
13

---

A
Implementation details
A.1 Soft-Actor Critic
The implementation of the Soft-Actor Critic (SAC) 22 used in the paper follows the implementation of SAC in PEBBLE [30], similar to Wang et al. [ to ensure a fair comparison. As such, the used hyperparameters, including the used optimisers, the same and can be found in the original paper
30.
A.2 Feature extractor
As mentioned in the paper, we train visual feature extractor (i(-) using a variational autoencoder (VAE) training objective. In particular, the VAE is implemented using simple convolutional neural network architecture, the details of which are shown Table [ The dimensionality of latent states |=| was chosen to be 16, thus ensuring high informational bottleneck to encourage feature disentanglement. The feature extractor trained using Adam optimiser [36] with a learning rate of 0.0001. The batch size is 128. A gradient step is applied after each environment step, and the training continues throughout RI training. We set the beta parameter / = 0.1, which is common way to encourage better image reconstructions in VAEs 37 and use MSE reconstruction loss.
To ensure training stability with respect to the RL target (top-rated candidate goal), we fix the weights of the VAE for calculating visual embeddings and update them every L. = 5000 steps at the same rate as top-rated candidate goals are updated as RL target states.
Table 1: Architecture of the convolutional feature extractor.
Component
Layer
Output Shape
Conv2D (3, 16, 4x4, stride=2, pad=1) + ReLU
128x128x16
Conv2D (16, 32, 4x4, stride=2, pad=1) + ReLU
64x64x32
Conv2D (32, 64, 4x4, stride=2, pad=1) + ReLU
32x32x64
Encoder
Conv2D (64, 128, 4x4, stride=2, pad=1) ReLU
16x16x128
Conv2D (128, 256, 4x4, stride=2, pad=1) ReLU
8x8x256
Conv2D (256, 256, 4x4, stride=2, pad=1) ReLU
4x4x256
Flatten
4096
Encoder (FC)
Linear (4096 16)
16
Decoder (FC)
Linear (16 4096)
4096
Conv 256, 4x4, stride=2, pad=1) + ReLU
8x8x256
ConvT2D 128, 4x4, stride=2, pad=1) + ReLU
16x16x128
ConvT2D((28,, 64, 4x4, stride=2, pad=1) ReLU
32x32x64
Decoder
ConvT2D(64, 32, 4x4, stride=2, pad=1) ReLU 64x64x32
(32 16, 4x4, stride=2, pad=1) ReLU 128x128x16
ConvT2D(16, 3, 4x4, stride=2, pad=1) + Sigmoid 256x256x3
A.3
GoalLadder
A.3.1
Candidate goal rating
When candidate goal gets added to a goal buffer By, it gets assigned an initial rating, é. The initial rating is calculated by taking the mean rating of all existing goals in the buffer, é ē= =e,e ei/N. This ensures that new goals are able to quickly catch up with top-rated goals, while not being immediately considered superior upon initial discovery.
We also set the parameters of ELO rating updates: C = 400 (constant controlling how sensitive the expected scores are to rating differences) and = 32 (rating update speed). These were chosen to match the commonly used values in chess rating. In practice, we find that these default parameters were satisfactory for effectively absorbing noisy VL.M outputs, while bringing the best goal to the top of the buffer.
14

---

A.3.2 Reward definition
Rewards are calculated using Euclidean distance between visual embeddings, I1<b()) ~5(0") (6(0))l2: In practice, we find that it is useful normalise rewards within bounds [0, 1], using a simple max max-min min normalisation, and then apply non-linear transformation, such that the rewards used during training, are calculated as = r", where 7 = 20. This ensures that reward scaling does not significantly impact the learning process of the RL agent, the visual feature extractor changes over the course of training. The non-linear transformation ensures that the agent accrues proportionally larger amounts of reward the closer it gets to the target state.
A.4 Computational resources
For training GoalLadder, we use Tesla V100 16GB GPU and 2x Intel Xeon E5-2698 v4 CPUs, each with 20 cores and 2 hardware threads core. The training time of a single GoalLadder agent took -45 hours. Initial experimentation included identifying the abilities of the used vision- language model, Gemini 2.0 Flash, to compare pairs of visual observations with respect to language instruction.
B
Task descriptions and prompts
Below is the prompt template used to get feedback from vision-language model. The formatting instructions of the expected response can be adjusted depending on the implementation.
Further, Table Table@ishon shows the used language instructions for each task in GoalLadder. The same template used for RL-F for controlled comparison. Table 3 shows language instructions used for reward calculation in VLM-RM. Finally, Table 4 shows language instructions used in RoboCLIP.
VLM prompt template in GoalLadder
Image 1: {IMAGE 1}
Image 2: {IMAGE 2}
The goal (LANGUAGE INSTRUCTION).
Answer the following questions:
1. What is shown in Image 1?
2. What shown in Image 2?
3. Is there any difference between Image and Image 2 in terms of achieving the goal?
4. Is the goal better achieved in Image 1 or in Image 2? Explain your reasoning.
If the goal is better achieved in Image than it is in Image 1, (FORMATTING INSTRUCTIONS).
15

---

Table GoalLadder language instructions for each task
Task
LANGUAGE INSTRUCTION
cartpole
"is pole balanced upright"
mountaincar
"is car at the peak of the mountain, to the right of the yellow flag"
opaner-open-v2
"of the robotic arm is to open the green drawer"
drawer-close-v2
"of the robotic arm close the green drawer"
sweep-into-v2
"of the robotic arm is to sweep the green cube into the hole"
vindow-open-v2
"of the robotic arm to open the window"
buttor-press-toplom---2
"of the robotic arm is to press the red button into the orange box"
Table 3: VLM-RM language instructions for each task
Task
LANGUAGE INSTRUCTION
cartpole
"pole vertically upright top of the cart"
mountaincar
"car at the peak of the mountain, to the right of the yellow flag"
drawer-open-v2
"open drawer"
drawer-close-12
"closed drawer"
sweep-into-v2
"green cube in hole"
vindow-open-v2
open window"
batton-prres--toploww--
"pressed button"
Table 4: RoboCLIP language instructions for each task
Task
LANGUAGE INSTRUCTION
drawer- open-v2
"robot opening drawer"
drawer-close-v2
"robot closing drawer"
sweep-into-v2
"robot pushing green cube into hole"
vindow-open-v2
"robot opening window"
"robot pressing red button"
C
Additional ablations
C.1 ELO rating
To disentangle the contributions of the ELO ranking more thoroughly, we have run a small ablation study of GoalLadder's performance without the ELO system, i.e. the 'greedy' rating system where always trust the VLM's decision to replace the top-rated goal. Table[5]shows the results on the Drawer Open task, confirming the importance of the ELO rating system in achieving good performance.
Table 5: ELO ablation.
GoalLadder
Final success rate
with ELO
0.97 + 0.11
without ELO
0.20 = 0.35
Qualitatively, we observe the expected suboptimal behaviour in GoalLadder without ELO: signifi- cantly better goals are periodically replaced by significantly worse goals as targets (i.e. VLM makes mistakes). This destabilises the training process and results in rapid performance drop.
16

---

C.2
Buffer size
In our initial experimentation, we have found that the buffer size should be big enough allow for diversity, but small enough to allow for frequent pairwise comparisons between the candidate goals.
Table 6: Success rates by buffer size.
Buffer size
Final success rate
Average success rate
Max success rate
I (No ELO)
0.20 + 0.35
0.14 + 0.23
0.25 + 0,45
5
0.83 + 0.13
0,48 + 0.13
1.00 + 0.00
25
0.80 + 0.11
0.47 + 0.14
= 1.00
50
1.00 + 0.00
0.61 + 0.11
1.00 = 0.00
10000
0.00 + 0.00
0.01 + 0.01
0.13 = 0.27
To confirm this intuition, we have run a small ablation of GoalLadder with varying buffer sizes. initial hypothesis: (a) smaller buffer sizes hinder
The results shown in Table [ confirmed performance as the diversity of candidate goals in the buffer suffers; (b) large buffer sizes require an exponentially larger number of pairwise comparisons arrive converged ratings, thus also hindering the performance of the model. Overall, we find that GoalLadder is robust to buffer sizes within the reasonable ranges.
C.3
Visual encoders
We have similarly tested the performance of GoalLaadder with two off-the-shelf visual encoders DINOv2 38 and CLIP [5] shown in Table These results indicate that small VAE trained environment observations outperforms large, pre-trained encoders. We hypothesise that the VAE allows the model better distinguish between fine-grained state differences, which are not well captured by the pre-trained models.
Table 7: Success rate by vision encoder.
Model
Max success rate
GoalLadder + VAE
1.00 + 0.00
Goall + DINOv2
0.25 0.21
GoalLadder + CLIP
0.38 + 0.17
17

---

D
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate that GoalLadder is able solve continuous control tasks, surpassing the performance of competitor methods and almost matching the oracle. By running competitor methods under the same feedback rate, we also showcase adder's sample efficiency with respect to the required amount of feedback. Lastly, we present ample evidence the fact that GoalLadder's rating system consistently brings the best goal the environment to the top of the ranking, thus reducing the effects of noisy VLM feedback. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the fact that GoalLadder currently only applicable tasks with static goals. Furthermore, we discuss the potential limitation of using visual similarity measure for calculating rewards.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
18

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Full implementation details included, both in the main body and the appendix of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
Depending on the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Specified in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
24


Reviews:
[{"review_id": "bGcjEOsyYH", "reviewer": "Reviewer_zeqD", "summary": "1. This paper introduces GoalLadder, a novel method for training RL agents in visual environments from a single natural language instruction. The core idea is to leverage a VLM to incrementally discover and rank goal states that represent progress towards the final task. \n2. This approach avoids the need for a large volume of VLM queries and the direct generation of a reward function, making it more query-efficient and robust. \n3. The authors demonstrate through experiments on classic control and robotic manipulation tasks that GoalLadder significantly outperforms existing methods and nearly matches, or surpasses, the performance of an oracle agent with access to the ground-truth reward.", "questions": "1. The paper's strongest component seems to be its robustness to noisy VLM feedback, which the authors attribute to the ELO rating system. Could you provide an ablation study or further analysis comparing the ELO system to a simpler ranking mechanism (e.g., a \"greedy\" approach that always replaces the current best goal with any new goal favored by the VLM)? This would help quantify the direct impact of the ELO system.\n2. The result where GoalLadder surpasses the oracle agent on the \"Drawer Open\" task is fascinating. The paper suggests this is due to the difficulty of manual reward engineering. Could the authors provide more insight into this? For example, what does the discovered goal image ($o^*$) look like compared to the state that would yield the maximum ground-truth reward? Does GoalLadder discover a reward function that provides a better-shaped gradient for the policy to learn?\n3. The reward function is defined as the negative Euclidean distance in the latent space of a VAE trained on collected observations. What is the intuition behind why this is a reasonable approach? This implicitly assumes that Euclidean distance in this learned space is a meaningful proxy for task progress. Have you considered or experimented with alternative distance metrics or using a different unsupervised representation learning objective (e.g., contrastive learning) instead of a VAE?\n4. How did you choose the key hyperparameters like the goal buffer size, query frequency (K), and reward update frequency (L)? A brief discussion on the sensitivity to these parameters would be valuable for others seeking to reproduce or build upon your work. For instance, is there a trade-off between VLM query cost and performance based on the choice of K and M?\n5. Could you discuss the potential for extending this method to more complex tasks, for instance, tasks where it is notoriously difficult to engineer a reward function? Furthermore, could you add an analysis and discussion of the method's failure cases? Clarifying the boundaries of this method will help the community understand the problem's difficulties and the paper's contribution.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The primary contribution is a novel approach to learn behaviors from a natural language instruction by leveraging VLM. The use of an ELO rating system to filter out noise from VLM judgments is a significant and original idea in this context. This addresses a well-known challenge of VLM-based feedback systems. Furthermore, the method decouples VLM feedback from the dense reward signal by using the VLM for goal-ranking and a separately trained autoencoder for defining a distance-based reward metric.\n2. The paper demonstrates exceptional performance. GoalLadder achieves an average success rate of \\~95% across all tasks, dramatically outperforming the next best baseline (\\~45%).\n3. Its high sample efficiency with respect to expensive VLM queries makes the approach more practical and scalable than many existing alternatives.\n4. The paper is well-written, clearly structured, and easy to follow. The method is explained logically, and the inclusion of Figure 1 and Algorithm 1 provides an excellent high-level overview of the system's components and data flow. The motivation and problem statement are well-established in the introduction.\n\n**Weaknesses:**\n\n1. The paper is missing ablation studies that would help disentangle the contributions of its key components. For example:\n   - How critical is the ELO rating system? How does it compare to a simpler approach, such as always trusting the VLM's judgment or averaging scores? An ablation here would strengthen the claim about robustness to noise.\n   - How important is learning a visual representation via a VAE? How would the system perform if the reward was calculated using distances in the VLM's own embedding space (e.g., CLIP's), or another off-the-shelf visual encoder?\n2. The method introduces several new hyperparameters, including the goal buffer size (∣Bg∣=10), the VLM query frequency (K), the number of queries (M), and the reward update schedule (L). The paper does not include a sensitivity analysis, making it unclear how difficult the method might be to tune for new environments.\n3. While the experiments are strong, they are confined to simulation environments with relatively clear visual progress. The authors acknowledge that the method assumes static goals identifiable from a single image. It is unclear how well the VAE-based visual similarity metric would work in tasks where semantic understanding is more critical than visual feature matching (e.g., \"put the red block on the blue block\" if there are other visually similar blocks).", "was_revised": true, "final_justification": "I sincerely thank the authors for their detailed response and the effort made during the rebuttal period. All of my concerns have now been addressed. I have also read the discussions between the authors and other reviewers.\n\nWhile I believe this work makes a valuable contribution to better utilizing pre-trained knowledge for reward modeling, I have some reservations about its broader impact. The method's applicability does not seem to extend beyond the scope of prior work (e.g., RoboCLIP), and its underlying mechanism remains unclear. As the authors themselves note, the use of the L2 norm is an empirical finding rather than a theoretically motivated choice. For these reasons, the work provides a useful result but lacks deeper, more generalizable principles.\n\nTherefore, I recognize the value of this specific work. I hope the paper is accepted by the community. I also encourage the authors to explore more principled methods for leveraging pre-trained knowledge in future iterations. I will maintain my positive evaluation.", "rebuttal": "Dear Reviewer zeqD,\n\nThank you for your positive review of our paper. In what follows, we clarify a few points and present additional ablation results you recommended to include. \n\n.\n\n# TL;DR:\n---\n- We ran ablations to confirm that ELO ranking significantly improves robustness over naive ranking, thus disentangling the contribution of the ELO rating system (W1 / Q1). **Table 2**.\n- We ran ablations showing that GoalLadder + VAE outperforms off-the-shelf encoders  like CLIP/DINOv2, which fail to capture fine visual differences and produce embedding spaces with weaker task alignment (W1). **Table 3.**\n- We explain why GoalLadder can sometimes outperform the oracle agent (Q2): ground-truth reward functions include implicit constraints on *how* tasks are expected to be solved; GoalLadder does not constrain the space of solutions. \n- On reward shaping (Q3), we clarify on our initial experimentation with the different reward metrics (L1, L2, cosine similarity). We also agree that contrastive learning is a promising future direction. \n- On hyperparameters (Q4), we present a buffer size ablation showing clear performance trade-offs. Other hyperparameters were selected to ensure a controlled comparison with the baselines. **Table 1.**\n- Finally, we reiterate our method’s limitations (Q5) and outline future work, such as extending to dynamic goals and improving embeddings via self-supervision. \n\nWe hope our clarifications and new experiments have addressed your questions. In light of these updates, we would be grateful if you would consider increasing your evaluation of our paper.\n\n.\n\n# New results\n---\nAblations tested on the `Drawer Open` task with 3 random seeds.\n\n---\n\n>**Table 1**: Buffer size.\n\n| Buffer size | Final success rate | Average success rate | Max success rate |\n| :---------: | :----------------: | :------------------: | :--------------: |\n| 1 (No ELO)  |    0.20 ± 0.35     |     0.14 ± 0.23      |       0.25 ± 0.45       |\n|      5      |    0.83 ± 0.13     |     0.48 ± 0.13      |       1.00 ± 0.00     |\n|     25      |    0.80 ± 0.11     |     0.47 ± 0.14      |       1.00 ± 0.00      |\n|     50      |    1.00 ± 0.00     |     0.61 ± 0.11      |       1.00 ± 0.00      |\n|     500     |    0.75 ± 0.24     |     0.56 ± 0.06      |       1.00 ± 0.00      |\n|    10000    |    0.00 ± 0.00     |     0.01 ± 0.01      |       0.13 ± 0.27      |\n\n---\n\n>\n>**Table 2**: ELO rating system.\n\n| GoalLadder  | Final success rate |\n| :---------: | :----------------: |\n|  with ELO   |    0.97 ± 0.11     |\n| without ELO |    0.20 ± 0.35     |\n\n---\n\n>**Table 3**: Visual encoders.\n\n|        Model        | Max success rate (%) |\n| :-----------------: | :------------------: |\n|  GoalLadder + VAE   |     1.00 ± 0.00      |\n| GoalLadder + DINOv2 |     0.25 ± 0.21      |\n|  GoalLadder + CLIP  |     0.38 ± 0.17      |\n\n.\n\n# Detailed response\n---\n\n- **W1 / Q1: \"How critical is the ELO rating system? How does it compare to a simpler approach, such as always trusting the VLM's judgment or averaging scores?\"**\n\nThis is a great question. In the paper, we address this point by having a controlled comparison with RL-VLM-F. As mentioned, in RL-VLM-F, inconsistent or noisy feedback is not explicitly dealt with. Since the results indicate that RL-VLM-F fails to solve most of the tasks, we believe it is a reasonable indicator of said issue.\n\nNevertheless, as you suggested, we have run a small ablation of GoalLadder's performance _without_ the ELO system, i.e. the 'greedy' rating system where the VLM's decision to replace the top-rated goal is always trusted. Ablation results on the `Drawer Open` task confirm our hypothesis -- **Table 2**.\n\nQualitatively, we observe the expected suboptimal behaviour: better goals are periodically replaced with worse goals as targets (i.e. VLM makes mistakes). This destabilises the training process and results in a rapid performance drop.  \n\n\n- **W1: \"How important is learning a visual representation via a VAE?\"**\n\nThank you for this question. In GoalLadder, we attempted to use one of the simplest examples of a visual feature extractor fine-tuned to the environment -- in the form of a VAE. Still, as we mention in the discussion section, we suggest future work to investigate self-supervised learning techniques to potentially improve the embedding space further.\n\nFollowing your suggestion, we have tested the performance of GoalLadder with two visual encoders: DINOv2 (Oquab et al., 2023) and CLIP (Radford et al., 2021) on the `Drawer Open` task -- **Table 3**. Further, we visualised the embedding space produced by DINOv2 / CLIP using t-SNE and overlayed the ground-truth rewards as a colour map, and compared it to the embedding space produced by our trained VAE model.  \n\nWhat we found is in line with the initial hypothesis. Firstly, GoalLadder + DINOv2 / CLIP backbone performs worse on the tested task. The plotted visual embedding space of DINOv2 and CLIP with ground-truth reward colour mapping indicates poor clustering: (a) there exist a large number of visual states that produce a low Euclidean distance (high GoalLadder reward) to the top-rated goal, and (b) high ground-truth reward states are significantly more spread out compared to the embedding space produced by the VAE. \n\nNevertheless, as you point out, we think that the use of off-the-shelf visual encoders may be useful in environments where fine visual differences are not crucial *and/or* where semantic differences are of particular importance. We leave this investigation for future work.\n\n\n- **Q2: \"The result where GoalLadder surpasses the oracle agent on the \"Drawer Open\" task is fascinating. Could the authors provide more insight into this?\"**\n\nThank you for acknowledging this, we also think this result is particularly interesting. \n\nWhat we believe is crucial is *how* the agent gets to the success state. Manually-designed Metaworld reward functions contain some constraints on how the task is expected to be solved. For instance, in the `Drawer Open` task, the agent is rewarded for grasping the handle; however, in many cases, the agent is able to open the drawer using the *side* of the handle. As such, with GoalLadder, the agent has more options on how to solve the task. Similarly, in the `Sweep Into` task, the reward function rewards the grasping of the cube; however, this is not necessary to solve the task, since pushing the cube can be as effective. \n\n\n- **Q3: \"The reward function is defined as the negative Euclidean distance in the latent space of a VAE trained on collected observations. What is the intuition behind why this is a reasonable approach?\"**\n\nThank you for this point. We have indeed experimented with (a) L1 distance, (b) L2 distance, and (c) cosine similarity. Upon initial experimentation L2 distance worked best -- hence our choice. We were unable to run large-scale ablation experiments due to financial constraints in using commercial VLMs. With regard to alternative learning objectives, we agree that it is a promising direction of improving GoalLadder even further. At this point, we leave this direction for future work. \n\n\n- **Q4: \"How did you choose the key hyperparameters like the goal buffer size, query frequency (K), and reward update frequency (L)?\"**\n\nIn order to disentangle the contribution of the novel components in GoalLadder against its best competitor model (RL-VLM-F), we wanted to ensure a controlled comparison. For this, we used the same hyperparameters and feedback scheduling format, where possible. As such, we fixed the reward function updates to L = 5,000 (used in PEBBLE (Lee et al., 2021) and RL-VLM-F). For the same reason, we used the standard RLHF feedback scheduling format by fixing K (query frequency) and M (feedback size) over the course of training. This allowed for a more substantiated empirical comparison of the two models. \n\nThe feedback rate was chosen in line with the rate of experience accumulation (i.e. episode length, which was 500 for Metaworld), making K = 500. Given the design of the system, we expect it to work better with higher feedback rates; however, as demonstrated, the current feedback rate is: (a) already significantly lower than those of the baseline models; (b) leads to a clearly superior performance against the baselines. \n\nWith regard to the buffer size, we have now run an ablation study with varying buffer sizes. Please see the following results for the `Drawer Open` task -- **Table 1.** These results confirmed our initial hypothesis: (a) smaller buffer sizes hinder performance as the diversity of candidate goals suffers; (b) larger buffer sizes require an exponentially larger number of pairwise comparisons for ratings to converge, thus also hindering the performance of the model.\n\n\n- **Q5: \"Could you discuss the potential for extending this method to more complex tasks? Furthermore, could you add an analysis and discussion of the method's failure cases?\"**\n\nThe limitations of our method are discussed in Section 5.1. To reiterate: (a) we assume that the progress or success of a task can be identified from a single image, limiting the application of our method to environments with static goals, and (b) GoalLadder relies on visual feature similarity for reward definition. In some settings, visual similarity between observations could be a limiting proxy for the underlying state similarity or task progress. Here, more advanced visual embedding techniques could be used instead.\n\nPossible extensions are mentioned in Section 5.3. In short, we think that there are two major directions of improvement: (a) video-based evaluation of task progress, where a VLM could be used to point out improving sequences of actions; and (b) improving the visual embedding space with self-supervised learning techniques. The former would allow GoalLadder to work with dynamic goals, while the latter could improve the learning signal from the distance minimisation objective."}, {"review_id": "zndxCVMbGa", "reviewer": "Reviewer_sgKz", "summary": "This paper introduces GoalLadder, a reinforcement learning (RL) framework that learns from natural language task descriptions using vision-language models (VLMs). Rather than relying on dense reward signals or extensive VLM queries, GoalLadder incrementally builds a ranked buffer of goal candidate states using pairwise VLM comparisons guided by an ELO rating system. The agent is trained to minimize the distance in a learned embedding space to the top-rated goal. This enables sample-efficient, noise-tolerant goal discovery without explicit supervision. The method outperforms prior VLM-based baselines and achieves performance close to or better than oracle reward baselines on standard continuous control and robotic manipulation tasks.", "questions": "1. What was the prompt used for the VLM during pairwise comparisons in the discovery phase (i.e., for $\\text{VLM}(o^*, o_j, 1)$)?\n\n2. When a new observation $o_j$ is inserted into the goal buffer after outperforming the current best candidate, how is its initial ELO rating determined? i.e., what is $e_j$ if $\\text{VLM}(o^*, o_j, 1) = 1$ in the discovery phase?\n\n3. How does the size of the goal buffer affect the performance?\n\n4. The paper uses Gemini 2.0 Flash as the VLM backbone. Have the authors considered how GoalLadder would perform with more capable or recent models, such as GPT-4, Claude, or Gemini 2.5?\n\n5. The use of ELO to refine candidate goal rankings is a core component of GoalLadder, but its added value is not explicitly demonstrated. For instance, have the authors measured the correlation (e.g., Spearman rank correlation) between initial VLM comparisons and final ELO rankings? If the initial discovery decisions are highly predictive of final rankings, this might suggest that the ELO system introduces limited additional robustness.\n\n6. Have the authors observed any failure cases where the ELO system converges to suboptimal goals due to VLM inconsistencies?\n\n7. The method uses a variational autoencoder (VAE) to learn the latent space in which rewards are defined. Have the authors experimented with alternative approaches such as contrastive learning?\n\n8. Since the reward function depends on distances in the learned latent space, it would be helpful to see an analysis of whether this space meaningfully reflects task progress. Have the authors performed any sanity checks such as t-SNE or PCA visualizations of the latent representations?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* Achieves strong empirical results: ~95% average success rate, outperforming all prior VLM-based baselines by a wide margin.\n\n* Significantly more sample-efficient in terms of VLM queries (∼4500 vs. ∼15000 for PEBBLE).\n\n* Clear, well-organized presentation with helpful diagrams and detailed algorithm description.\n\n* Leverages a learned embedding space to define rewards, enabling generalization to unseen states without requiring dense supervision.\n\n\n**Weaknesses**\n\n* No appendix is given in the paper. Should be fixed.\n\n* Embedding space is learned via VAE, but no comparison to alternative methods (e.g., contrastive learning).\n\n* Uses only Gemini 2.0 Flash as the VLM backbone; lacks evaluation on robustness across different models.\n\n* No analysis of robustness under adversarial or inconsistent VLM feedback.\n\n* The current evaluation is limited to five Metaworld tasks. In contrast, RL-VLM-F evaluated on a wider set, including Soccer, Cloth Fold, Straighten Rope, and Pass Water.", "was_revised": true, "final_justification": "NA", "rebuttal": "Dear Reviewer sgKz, \n\nThank you for your review!\n\n.\n\n# TL;DR: \n---\n- We clarify that the appendix was included in the supplementary zip and already addresses several of your questions, including full prompt details (Q1) and ELO initialisation (Q2). \n- We've now run ablations on goal buffer size (Q3). GoalLadder is robust to reasonably sized buffer sizes (5-500). Small (<5) and large (>500) buffer sizes deteriorate performance in line with the intuition. **See Table 1**.\n- We've now run ablations to confirm that ELO ranking significantly improves robustness  over naive ranking, thus disentangling the contribution of the ELO rating system (Q5). **See Table 2.**\n- We have observed no notable failure cases from VLM inconsistencies *after* adding the ELO rating system (Q6). \n- While we used Gemini 2.0 Flash due to financial constraints, GoalLadder is designed to  perform better with better VLMs (Q4). In our opinion, using the less performative VLM adds to the significance of our results. \n- Regarding VAE embedding space visualisation (Q7–8), we observed meaningful clustering of the latent space, supported by t-SNE visualisations and a strong negative correlation between latent distances and ground-truth rewards. We welcome future work attempting to use contrastive learning to improve the embedding space further. \n\nWe hope our clarifications and new experiments have addressed your questions. In light of these updates, we would be very grateful if you would consider updating your evaluation. We believe the revised version merits a higher score, especially considering the fact that the initial review did not take into account the appendix provided. We think it may have affected your scores under \"*quality*\" and \"*clarity*\", given that the other reviewers provided scores of at least 3 for both.\n\n.\n\n# New results\n---\nTested on the `Drawer Open` task with 3 random seeds.\n\n---\n\n>**Table 1**: Buffer size.\n\n| Buffer size | Final success rate | Average success rate | Max success rate |\n| :---------: | :----------------: | :------------------: | :--------------: |\n| 1 (No ELO)  |    0.20 ± 0.35     |     0.14 ± 0.23      |   0.25 ± 0.45    |\n|      5      |    0.83 ± 0.13     |     0.48 ± 0.13      |   1.00 ± 0.00    |\n|     25      |    0.80 ± 0.11     |     0.47 ± 0.14      |   1.00 ± 0.00    |\n|     50      |    1.00 ± 0.00     |     0.61 ± 0.11      |   1.00 ± 0.00    |\n|     500     |    0.75 ± 0.24     |     0.56 ± 0.06      |   1.00 ± 0.00    |\n|    10000    |    0.00 ± 0.00     |     0.01 ± 0.01      |   0.13 ± 0.27    |\n\n---\n\n>**Table 2**: ELO rating system.\n\n| GoalLadder  | Final success rate |\n| :---------: | :----------------: |\n|  with ELO   |    0.97 ± 0.11     |\n| without ELO |    0.20 ± 0.35     |\n\n\n.\n\n# Detailed response\n---\n\n- **\"No appendix is given in the paper. Should be fixed.\"**\n\nWe apologise for the confusion -- we did in fact provide the appendix; however, it was attached separately in the supplementary materials zip file. Indeed, several of the questions you raised in the review are answered in the appendix. It includes: implementational details, computational resources, prompts. \n\n\n- **Q1: \"What was the prompt used for the VLM during pairwise comparisons in the discovery phase?\"**\n\nThe full prompt can be found in the appendix. In short, we ask the model to compare two images with respect to a specified text instruction. \n\n\n- **Q2: \"How is its initial ELO rating determined?\"**\n\nThis is similarly clarified in the appendix. In short, the initial rating of a newly-added candidate goal is computed to be the average of ELO ratings of other goals in the buffer (i.e., the goal is inserted in the middle of the buffer). This ensures that newly-discovered goals are not immediately considered to be superior to the top-rated goal. \n\n\n- **Q3: \"How does the size of the goal buffer affect the performance?\"**\n\nThis is a great question. In our initial experimentation, we have found that the buffer size should be big enough to allow for diversity, but small enough to allow for frequent pairwise comparisons between the candidate goals.\n\nFollowing your suggestion, we have now run an ablation study with varying buffer sizes. The results confirmed our initial hypothesis: (a) smaller buffer sizes hinder performance as the diversity of candidate goals in the buffer suffers; (b) large buffer sizes require an exponentially larger number of pairwise comparisons to arrive at converged ratings, thus also hindering the performance of the model -- **see Table 1.**\n\n\n- **Q4: \"Have the authors considered how GoalLadder would perform with more capable or recent models, such as GPT-4, Claude, or Gemini 2.5?\"**\n\nIt is unfortunate that experiments with commercial VLMs require substantial financial resources. As such, we were unable to evaluate the performance of GoalLadder using the more expensive models. We hope for your understanding in this matter. \n\nHowever, we would like to stress two points here: (1) the existing VLMs still struggle with spatial understanding (e.g., [1]). In our small-scale experimentations, we have found that they fall prey to the same problems as Gemini 2.0 Flash. (2) Perhaps more importantly, GoalLadder is designed to perform better with better VLMs. This is because more accurate VLM labels would result in the faster convergence of goal ratings to their 'true' ratings. GoalLadder will retain its capability to solve the tasks studied in the paper, while also maintaining its superior sample efficiency against the competitor methods. Even in its current form, GoalLadder produces *better sample efficiency than PEBBLE*, which uses ground-truth preference labels -- *effectively amounting to a VLM producing labels with 100% accuracy*. We believe this result substantiates the above claim.  \n\n[1] SpatialVLM, Chen et al., 2024\n\n\n- **Q5: \"The use of ELO to refine candidate goal rankings is a core component of GoalLadder, but its added value is not explicitly demonstrated.\"**\n\nThis is a great question -- we appreciate that you brought it up. In the paper, we address this point by having a controlled comparison with RL-VLM-F, in which inconsistent or noisy feedback is not explicitly dealt with. Since it failed to solve most of the tasks, we believe it is a reasonable indicator of said issue.\n\nNevertheless, to address your question more thoroughly, we have run a small ablation study of GoalLadder's performance *without* the ELO system, i.e. the 'greedy' rating system where we always trust the VLM's decision to replace the top-rated goal. As another reviewer pointed out, this would effectively showcase the contribution of the ELO system to our model. \n\nHaving run this ablation on the Drawer Open task, we can confirm that the ELO rating system is a crucial component of GoalLadder -- **see Table 2.**\n\nQualitatively, we observe the expected suboptimal behaviour: significantly better goals are periodically replaced by significantly worse goals as targets (i.e. VLM makes mistakes). This destabilises the training process and results in a rapid performance drop. \n\n\n- **Q6: \"Have the authors observed any failure cases where the ELO system converges to suboptimal goals due to VLM inconsistencies?\"**\n\nIn our initial experimentations, particularly with RL-VLM-F, we have observed that VLMs can make frequent and often consistent mistakes in some pairwise comparisons. We therefore noted that trusting VLMs completely may result in such 'suboptimal' convergence points. Based on the experiments we ran since the incorporation of ELO, we have not observed notable failure cases, as diverse pairwise comparisons allow VLMs to get out of these  suboptimal loops, when and if they appeared. This is further evidenced by the ablation study we ran in regard to your Q5. \n\n\n- **Q7: \"Have the authors experimented with alternative approaches such as contrastive learning?\" *AND* \"Embedding space is learned via VAE, but no comparison to alternative methods (e.g., contrastive learning).\"**\n\nIn the paper, we acknowledge that more sophisticated methods for representation learning, such as contrastive learning, would be interesting to investigate in future work. \n\nHowever, the results show clear evidence that a simple visual feature extractor suffices to produce state-of-the-art results in this area. Self-supervised learning techniques may be used to *extend* our method by considering temporal ordering or other learning objectives to improve the landscape of the latent space. Still, we believe that the use of a VAE does not diminish the significance of our results. \n\n\n- **Q8: \"It would be helpful to see an analysis of whether this space meaningfully reflects task progress.\"**\n\nThis is a useful suggestion. In the paper, we implicitly address this question by demonstrating the superior performance of GoalLadder against the baselines, showing that the *resultant rewards are conducive to solving the tasks*. \n\nNevertheless, we have (1) made a t-SNE visualisation of the learned latent space while overlaying a colour map of the ground-truth reward and the L2 distance to the top-rated goal; (2) calculated the correlation between the latent L2 distances and the ground-truth reward using subsets of observations from the replay buffer. \n\nFor (1) we find that the visualised latent space contains meaningful clustering of the state space. For example, for the `Drawer Open` task, states in which the drawer is open are clustered together and have both high ground-truth reward and low L2 distance to the top-rated goal. \n\nFor (2), we find a moderate-to-strong negative Pearson correlation of −0.55 ± 0.007 (r² ≈ 0.30) and a Spearman rank correlation of −0.43 ± 0.01 (p < 1e−16). This shows that the resultant latent distance is a meaningful (though not perfect) proxy for task progress. \n\nWe believe both of these results substantiate the fact that GoalLadder produces a useful reward landscape, which is further evidenced by its superior performance against baseline methods."}, {"review_id": "K8BIVj3xiG", "reviewer": "Reviewer_Y7Jk", "summary": "This paper presents GoalLadder, a RL framework for solving robotics control tasks from a single natural language intruction (without using the dense reward function). The method incrementally discovers goal states using pairwise comparisons from a vision-language model (VLM) and ranks these states using an Elo style rating system. The top-ranked goals are then used to define a reward function based on euclidean distance in a learned latent space. Experimental results show good performance on classical control and and simple manipulation tasks (drawer open, window close etc).", "questions": "- How did you robustly verify that the latent space aligns with actual task progress? Did you test whether visually close but semantically different states are rewarded the same? as in doing any edge-case testing.\n- What are the compute/runtime implications of GoalLadder vs. preference-based methods? Can the method run efficiently in real-time or on physical robots?", "limitations": "- This method only works on tasks where success is clearly apparent from a single image frame.\n- Visual similarity is not always equivalent to task completion, this assumption can fail in tasks with subtle state changes.\n- This method's robustness is also tied to the accuracy of the VAE learned representation.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n- No handcrafted or dense reward required: As far as I understood, this approach does not require a dense reward, not requires training a reward model\n- The writing is clear and the method seems sound.\n- Good performance results.\n\n## Weaknesses:\n- I'm not fully up-to-date with the SOT in this line of work, but I know some existing baselines in language-conditioned RL such as SayCan [1], Code-as-Policies [2], Language-to-Reward [3]. My general feeling is that these are strongly related work and that should be included as a comparison.\n- Text version of RoboCLIP is used even though it also provides a video-level alignment capability.\n- This method assues the task completion can be inferred from a single image (it may not work for more complex tasks).\n- Reward is defined as euclidean distance in the latent space. This assumes that visual proximity in this space corresponds to task progress and this can be a weak reward proxy in more complex environments.\n- No real world experiments.\n\n\n\n\n[1] - https://say-can.github.io/\n\n[2] - https://arxiv.org/pdf/2209.07753\n\n[3] - https://arxiv.org/pdf/2306.08647", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Y7jk, \n\nThank you for your positive review of our paper!\n\n.\n# TL;DR: \n---\n- We acknowledge the mentioned works [1, 2, 3]. We clarify that the suggested baselines assume access to control APIs, pre-trained skills, or expert demonstrations. This significantly diverges from our setting where agents learn low-level behaviours directly from VLM feedback in visual environments. We do cite [3] and will cite [1,2] in the final version of the paper.\n- Regarding RoboCLIP, we used the text version to ensure fair comparison, since GoalLadder and its baselines operate *without expert demonstrations* and learn to solve tasks *using only a single language instruction*.\n- Regarding VAE embedding space visualisation, we observed meaningful clustering of the latent space, supported by t-SNE visualisations and a strong negative correlation between latent distances and ground-truth rewards.\n- While both GoalLadder and preference-based methods are similar in runtime inference cost, GoalLadder requires significantly fewer VLM queries, thus making it more scalable and practical for real-world robotics.\n\nWe hope our clarifications have addressed your questions. If our response has helped to resolve any of your doubts regarding the paper, we would be very grateful if you would consider increasing your confidence score.\n\n.\n\n# Detailed response\n---\n\n- **W1: \"... some existing baselines in language-conditioned RL such as SayCan [1], Code-as-Policies [2], Language-to-Reward [3]. My general feeling is that these are strongly related work and that should be included as a comparison.\"**\n\nThank you for these pointers. While these works deal with language-conditioned RL, their problem settings fundamentally differ from ours. For our baselines, we have selected methods that match the crucial assumptions / problem setting used in GoalLadder; namely, (a) visual environments, (b) access to a language instruction that describes the task at hand, (c) learning without any access to agent's action spaces (i.e. no grounding) or to environment simulation. \n\nAllow us to briefly expand on why we believe it would not be appropriate to use the aforementioned methods as baselines. \n\nIn SayCan [1], the agent is assumed to have access to a set of low-level skills, while the LLM is used to select high-level actions (skills) to be executed. In contrast, GoalLadder and its baselines do not assume access to pre-trained behaviours and are concerned with learning low-level behaviours from VLM feedback, instead.\n\nCode-as-Policies [2] does not address the learning of a low-level RL policy and assumes access to control APIs/primitives. It generates code that parametrises pre‑existing perception and control APIs (scripted primitives or motion controllers). In contrast, GoalLadder and its baselines learn behaviours directly in the environment’s intrinsic action space.\n\nLanguage‑to‑Rewards [3] similarly does not train a low‑level RL policy. Instead, it maps language into a predefined reward library and uses MuJoCo MPC with privileged state / dynamics to optimise actions.\n\nWe want to note that we did cite [3] in the Related Work section, pointing out the differences stated above. We would be happy to cite [1] and [2] in the final version of the paper, as well. \n\n\n\n- **W2: \"Text version of RoboCLIP is used even though it also provides a video-level alignment capability.\"**\n\nThis choice is justified in the paper (see Experiments: Baselines). To reiterate, in order to ensure a fair comparison (namely, GoalLadder and its baselines must be able to learn using *only a single language instruction*), the text version of RoboCLIP was used. \n\nUsing the video version of RoboCLIP would imply access to an expert demonstration, which GoalLadder and its baselines do not have or need by design. \n\n\n- **Q1: \"How did you robustly verify that the latent space aligns with actual task progress?\"**\n\nThis is a great question. In the paper, we implicitly address this question by demonstrating the superior performance of GoalLadder against the baselines, showing that the *resultant rewards are conducive to solving the tasks*. \n\nNevertheless, to show this more explicitly, we have now (1) made a t-SNE visualisation of the learned latent space while overlaying a colour map of the ground-truth reward and the L2 distance to the top-rated goal; (2) calculated the correlation between the latent L2 distances and the ground-truth reward using subsets of observations from the replay buffer. \n\nFor (1) we find that the visualised latent space contains meaningful clustering of the state space. For example, for the `Drawer Open` task, states in which the drawer is open are clustered together and have both high ground-truth reward and low L2 distance to the top-rated goal. \n\nFor (2), we find a moderate-to-strong negative Pearson correlation of −0.55 ± 0.007 (r² ≈ 0.30) and a Spearman rank correlation of −0.43 ± 0.01 (p < 1e−16). This shows that the resultant latent distance is a meaningful (though not perfect) proxy for task progress. \n\nWe believe both of these results substantiate the fact that GoalLadder produces a useful reward landscape, which is further evidenced by its superior performance against baseline methods.  \n\n\n- **Q2: \"What are the compute/runtime implications of GoalLadder vs. preference-based methods? Can the method run efficiently in real-time or on physical robots?\"**\n\nThank you for this question. \n\nBoth GoalLadder and preference-based methods require the use of a neural network to compute the reward. In the case of GoalLadder -- it is the visual feature extractor and the subsequent calculation of the distance to the top-rated goal. In the case of preference-based methods -- it is the trained reward function. In this sense, there is no substantial difference between the two approaches. The computed rewards are used to train a light-weight SAC policy, which can be deployed on a physical robot.\n\nWhere important savings come into the picture is during the *acquisition of the expensive VLM feedback*. As demonstrated, GoalLadder requires substantially fewer VLM queries to solve the tasks. We find this result particularly important if GoalLadder were to be scaled or applied for training physical robots. Preference-based methods would likely require significantly more feedback during the training phase."}]

Author rebuttal:

Meta-review: Authors present a method that uses a VLM to rank subgoals for completing a task using an ELO rating system to 1) reduce the number of queries made to the VLM and 2) avoid learning a parameterized reward function. They also use a pretrained embedding space as a dense reward to gauge progress towards these subgoals. There are strong empirical results that significantly outperform similar methods and significantly reduces the number of required queries to the VLM. 

Strengths: Strong empirical performance and improved efficiency in terms of the number of needed queries of the VLM compared to the next best prior method. The paper is clearly written, and the high-level method well laid out. 

Weaknesses: There is a lack of ablations testing each component of the method, i.e. the representation used as dense reward for reaching subgoals, the importance of the ELO rating system, and there was a lack of sensitivity analysis to the many new hyperparameters introduced as part of the method.

Most important reasons: There is significant improvement over prior work on multiple fronts (computational needs and overall performance) and one of the main contributions seems to be the use of the ELO rating system, which is an interesting insight. 

Reviewer discussion: The weaknesses that were brought up in the reviews were broadly addressed in the rebuttal phase. The authors provided extensive additional ablations and hyperparameter sweeps to demonstrate how their method is affected by various changes. There were concerns brought up about assumptions about the environment that are necessary for this method to work, i.e. obvious visual progress from the observation space and missing environments that were present in prior work. The prior point was part of the authors' discussion about limitations which I commend, but the latter point was not addressed.
Decision: Accept (poster)