VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation
Authors: Kuo-Han Hung
Pang-Chi Lo
Jia-Fong Yeh
Han-Yuan Hsu
Yi-Ting Chen
Winston H. Hsu
Venue: ICLR 2025 Poster (2025)
Keywords: reward learning
reinforcement learning
long-horizon robot learning
vision-language
Primary area: reinforcement learning
TL;DR: Existing Vision-Instruction Correlation (VIC) reward models struggle with training for long-horizon tasks. We propose VICtoR, a new reward model for long-horizon robotic reinforcement learning that assigns rewards hierarchically.

Abstract: We study reward models for long-horizon manipulation by learning from action-free videos and language instructions, which we term the visual-instruction correlation (VIC) problem. Existing VIC methods face challenges in learning rewards for long-horizon tasks due to their lack of sub-stage awareness, difficulty in modeling task complexities, and inadequate object state estimation. To address these challenges,
we introduce VICtoR, a novel hierarchical VIC reward model capable of providing effective reward signals for long-horizon manipulation tasks. Trained solely on primitive motion demonstrations, VICtoR effectively provides precise reward signals for long-horizon tasks by assessing task progress at various stages using a novel stage detector and motion progress evaluator. We conducted extensive experiments in both simulated and real-world datasets. The results suggest that VICtoR outperformed the best existing methods, achieving a 43% improvement in success rates for long-horizon tasks. Our project page can be found at https://cmlab-victor.github.io/cmlab-vicotor.github.io/.

Published as a conference paper at ICLR 2025
VICTOR: LEARNING HIERARCHICAL VISION- INSTRUCTION CORRELATION REWARDS FOR LONG-
HORIZON MANIPULATION
Kuo-Han Hung' National Taiwan University
Pang-Chi Lo National Taiwan University
Jia-Fong Yeh National Taiwan University
Man-Yuan Hsu National Taiwan University
Yi-Ting Chen
Winston H. Hsu National Taiwan University
National Yang Ming
Chiao Tung University
ABSTRACT
We study reward models for long-horizon manipulation by learning from action-free videos and language instructions, which we term the risual-intrruction correlation (VIC) problem. Existing VIC methods face challenges in learning rewards for long- horizon tasks due to their lack of sub-stage awareness, difficulty in modeling task complexities, and inadequate object state estimation. To address these challenges, we introduce VICtoR, a novel hierarchical VIC reward model capable providing effective reward signals for long-horizon manipulation tasks. Trained solely on primitive motion demonstrations, VICtoR effectively provides precise reward signals for long- long-horizon tasks by assessing task progress various stages using a novel stage detector and motion progress evaluator. We conducted extensive experiments in both simulated and real-world datasets. The results suggest that VICtoR outperformed the best existing methods, achieving a 43% improvement in success rates for long-horizon tasks. Our project page can be found at https 7/ 7/cmlab allaaabvviioo github. ho//mmlabb vicotor. wit.ub. 1o/
INTRODUCTION
Reinforcement learning (RL) has been extensively studied for long-horizon manipulation 1 2 However, crafting reward functions for these tasks is complex, as it typically requires access to true states significant domain expertise. Consequently, there is urgent need for robust and accurate reward model for these tasks. Previous research has explored modeling reward functions via robotic expert demonstrations [4, goal-imager (6; 7]. and human demonstrations [ 9.. Unfortunately, the required task specification materials, such as goal-images, remain costly and impractical.
Recently, methods that consider the vision-instructionn correlation (VIC) reward signals have emerged, providing a more accessible way for task specification through language. Specifically, these VIC methods view the reward modeling as a regression classification problem and train the reward model on the given action-free demos and instructions. Pioneer approaches [10] 11 12: 13 [4] studied using pre-trained Vision Language Models (VLMs) or Video Language Models, like CLIP
15] to generate rewards by assessing the similarity between visual observations and language goals. Additionally. Ma al. [16] explore the possibility of using human video and language data to pre-train reward model, later fine-tuned with in-domain data. However, regardless of whether diverse data is used for pre-training, existing VIC studies still require full-length data to retrain the model. Notably, these methods are constrained to short-horizon tasks, such as "picking up block".
Figure I Jillustrates three challenges we observe when applying existing VIC methods to long-horizon manipulation tasks: (1) No awareness of task decomposition: Failing to divide complex tasks into manageable parts limits adaptability.
(2) Confusion from variance task difficulties: Training reward model long-ho horizon tasks impairs the learning of reward signals and fails to generate suitable progressive rewards. (3) Ambiguity from lacking explicit object state estimates: Relying on whole-scale image observations overlook critical environmental changes. For instance, when

---

Published as a conference paper at ICLR 2025
Training long-horizon task with existing Vision-Instruction Correlation methods suffer from
VICtoR Solution
Instruction:
block the closed
No awareness of task decomposition
Task Knowiedde
Confusion from variance in task difficulties
Stage
open the crower
Tove the Black teto the crewer
Ambiguity from lacking explicit object state estimates
Potdon
slack
lift Block
Task:
then
Observation
Agent
VzCcoR
Reward
The light la coeed
opered
Figure I: Problems in existing VIC methods and VICtoR's solution. Training long-horizon task with existing VIC methods commonly suffer from the listed problems. To address these problems, we propose VICtoR, a hierarchical reward model that can decompose long- long-horizon tasks and assign rewards by identifying the stage, motion, and progress of the agent from visual observations.
training for the task "move the block into the closed drawer", previous VIC models would assign high rewards for moving the block even the drawer is closed, misleading the learning process.
We propose VICtoR, a hierarchical VIC reward model for long-horizon manipulation. Figures ] and depiet its concept and architecture. VICtoR learns effective rewards for long-horizon tasks by hierarchically assessing overall progress. Specifically, it decomposes long- oong-horizon tasks (high-level) into stages (mid-level) and motions and progress (low-level, robot primitives). determining the reward by considering task progress at different granularities. Meanwhile, the hierarchical feature sets VICtoR free from the requirements of full-length demonstrations and further grants better generalizability. By treating each long-horizon task as an arbitrary permutation of available motions, VICtoR, trained only on action-free motion videos, can be adapted to unseen long-horizon tasks.
To minimize human effort, we leverage GPT-4's semantic understanding extract task knowledge (expected object statuses and required motions). Then, a stage detector retrieves the current object statuses from the observation and verifies if they match the condition of specific stage in the task knowledge (Challenge #1). Next, from a list of required motions to complete the stage, a motion progress evaluator determines the current motion and assesses the in-motion progress (Challenges #2 and #33) by evaluating the correlations between visual observations and each motion instruction. Consequently, VICtoR can generate informative rewards complete long-horizon tasks.
We conducted extensive experiments in both simulated and real-world settings. In the simulation, we designed ten tasks of varying horizons, including those where each step's success depends on the success of the previous stage. For the real-world experiments, we used the dataset collected from XSkill [17] and visualized the trained reward curves in comparison to other baselines. Compared to the best prior VIC reward model, VICtoR achieved a 25% improvement in the average success rate across all tasks and a 43% improvement in the more challenging tasks (tasks which consisted of more than 4 motions). Additionally, our detailed ablation studies emphasized the importance of our hicrarchical architecture and training objectives. Visualizations of the learned embedding space and rewards demonstrated that VICtoR effectively assesses task progress.
We summarize our contributions as follows: (I) We are the first to explore the potential VIC rewards for corizoorizzon manipulation tasks. (2) We introduce VICtoR, novel hierarchical VIC reward model that assesses task progress by decomposing it into various levels. (3) We present extensive results from simulated and real-world experiments, ablations, and visualizations to demonstrate VICtoR's ss superiority, outperforming the best prior method by 43% on more challenging tasks.
RELATED WORK
Large Pre-trained Models for Robotics With great ability of reasoning, Large Language Models (LL.Ms) have been utilized for robotics in navigation [18., task planning [19 20 code-completion
2

---

Published as a conference paper at ICLR 2025
for policies 21 22], and manipulation (33 24 26 27). These works leverage LLMs' ability to translate high-level instructions into actionable sequences and break them down into sub-orders for precise agent control. Additionally, recent advancements have leveraged Vision-Language Models (VLMs) to enhance environmental understanding for agents. With profound understanding of the physical world, VLMMs facilitate the works in navigation, robotic control, and monitoring (28] 29 30 31 32]. Unlike previous methods, for reward modeling, VICtoR uses LLMs to break down tasks and VLMs to assess visual observation with motion descriptions, allowing our method to better assess the task's progress and distinguish it with different granularity.
Addressing Long-Horizon Tasks. Long-horizon manipulation is Long-standing problem. Tradi- tional Hierarchical Reinforcement Learning (HRL) methods [33] 34 35 36 address this challenge
by decomposing complex tasks into hierarchical levels or sequential orders, where higher-level policies make decisions that guide the selection of parameters for lower-level policies or pre-trained actions. Task and Motion Planning (TAMP) methods (37 38
39 attempt to assemble and sched- ule well-trained motions to accomplish long-horizon tasks. Additionally, Long-horizon Imitation Learning [40 [1] has been proposed to learn long-horizon tasks from expert demonstrations. Unlike these approaches, VICtoR aims to learn the reward model for long-hone manghorizon tasks from action-free demonstrations and task instructions. Our reward model is designed to be independent of policy design and can be applied to RI. algorithms learn long-horizon manipulation tasks.
PRELIMINARIES
Problem Statements. For a reinforcement learning task μ,, we assume the environment follows partially observable Markov Decision Process (POMDP), which can be described with the tuple M₁ = P,
in which the observation o E Q is derived from the observation function (o 8) conditioned on the current state E S. A is the space of actions; P(s' | a) denotes the dynamics of the environment; R and y are the environment's reward function and discount factor, respectively. The objective aims seek a policy x =a o) that assigns probability to action o, given the observation o, which maximizes the accumulated discounted rewards in a rollout.
VIC Reward Model. This work aims to develop a Vision-Instruction Correlation (VIC) reward model to generate accurate rewards from visual observations and language instructions. We assume an instruction L for the task y is accessible, as the instruction is a low-cost resource to collect. Specif- ically, we seek reward model R(o,, or-1, I.²) computes the reward by evaluating the difference in correlation between the image observation o and the instruction Lt. Intuitively, R(o;, ot-1, I") should generate higher reward the or moving closer to the task's goal compared to Oz-1-
Stages and Motions. In the context of long-horizon manipulation, we define motion the primitive movement that a robot can make, such as reaching or grasping an object. In addition, stage refers to high-level interaction with object in the environment, requiring the consideration of complex objects' status, and can encompass multiple motions. For instance, the stage "open the drawer" involves the motions "reach the drawer handle" and "pull out the drawer." A comprehensive list of motions and stages is detailed in Appendix C.I
4 METHOD
Overview. VICtoR is reward model for long-horizon manipulation, relying on visual observations and language instructions. As depicted in Figure 2 VICtoR employs hierarchical approach to assess task progress at various levels, including stage, motion, and motion progress. It consists of three main components: (1) a Task Knowledge Generator that decomposes the task into stages and identifies the necessary object states and motions for each stage (Section 4. : (2) Stage Detector that detects object states to determine the current stage based on the generated knowledge (Section
4.2) (3) Motion Progress Evaluator that assesses motion completion within stages (Section 4.3). With this information, VICtoR then transforms it into rewards (Section 4.4). Both the Stage Detector and Motion Progress Evaluator trained motion-level videos labeled with object states, which are autonomously annotated during video collection. This setup enables VICtoR to deliver precise reward signals for complex, unseen long-horizon tasks composed of these motions in any sequence.
3

---

Published as a conference paper at ICLR 2025
VICteR Training
VICtoR Inference (Reward Model)
Motionievel Videoa with Language Iratruction mach dlaver hands
Gurrent
Progreax
Stage Detector
Votie Raward
Object
Bower iboue
Pegres NA
Candidatas Mationa
Progress
Whet the
reict the drowe
Fifl
Toxt Knowleddee Ganeration
Srice
Task Instruction
Object State Labels
clived
"love
mndare tnach Saver tands,
Stage
closed drower"
Curmen CoRvEz
Stage Rlward
Figure 2: Training and inference pipeline of VICtoR. VICtoR is trained using motion-level videos with language annotations and object state labels. It first decomposes the task into task knowledge for decomposed stages, conditional object states, and motions. Then, it uses Stage Detector to identify the stage, and Motion Progress Evaluator (VLM) to detect the motion and in-motion progress.
4.1 TASK KNOWLEDGE GENERATION
To make long-horizon tasks more achievable and manageable, we leverage the reasoning ability in Large Language Models (LLMs) to decompose and analyze the tasks. Previous research [19] 20] introduced LLMs into policy learning by planning sub-goals or actions for explicit guidance to enhance long- morizooioon capabilities. Building upon this foundation, we explore more nuanced usage. decompose given training task into distinct sub-stages, define the specific conditions or states of each object at these stages, and identify the motions required by the agent to advance to the next stage. To achieve this, we employ GPT-4 to analyze and construct task knowledge. example of task knowledge generation for the task "move the block into the
Our goal
Figure Saarre@@llusstrtee closed drawer." " Details of our prompt design and implementation are provided in Appendix G and additional experiments for robustness described in Appendix B.2
4.2
STAGE DETECTION
Traditional VIC models, which encode entire visual observations assess task progress, often overlook subtle yet critical changes in object states. To address this, we differentiate between detecting local object state changes and global agent movements (discussed
next section). In the previous section, Task Knowledge Generator decomposed long-horizon tasks into sub-stages, each conditioned on different object states. To detect these stages, we designed Stage Detector, which detects stage by identifying each object's state in the environment and comparing these with the conditioned states of each stage as defined in the task knowledge.
Object State Detection. To detect object states, our Stage Detector first leverages a language- conditioned object detector D, such as MDETR [42), to identify objects in the image observation o. Essentially, we crop out the bounding box D(o, c) generated from the detector based on the current observation and the queried object c. Subsequently, these cropped images fed into object state classifier P. The classifier predicts the likelihood of categorical object state for each cropped image as P(T.5) P(Ts)D((,c)) c)), where I represents the language description of possible object states for object c. We summarize the training details in Appendix
Stage Detection. After extracting all the object states in the environment, our Stage Detector can then determine the current stage by comparing these states with the task knowledge. A stage is identified if, and only if, all object states align with the task knowledge for a specific stage. For example, illustrated in Figure 2] if the drawer is currently closed, compare this condition with the states specified in each stage of the task knowledge. Upon finding that the drawer is expected to be closed only the first stage, conclude that the agent is the first stage. This comparison
4

---

Published as a conference paper at ICLR 2025
should be applied to every conditioned object for the tasks. If the extracted set of object states does not match any stage, the agent will considered to be in the initial stage.
4.3 MOTION PROGRESS EVALUATOR
Since stage may require multiple motions to complete, relying merely on the stage reward insufficient for learning complex tasks. To this end, we introduce the Motion Progress Evaluator (MPE) to provide the reward signals within the stage. As discussed in Section 4.1 each detected stage comes with list of motion descriptions in the task knowledge. Building this, the MPE assesses the agent's s current motion and progress within the motion, taking into account these descriptions. This MPE model yields more detailed reward signal that accurately reflects progress within stages.
Architecture. Our MPE model employs the vision encoder EV, and language encoder E. from CLIPP[5], with two additional heads (fully-connected layers) attached FV. Initially, the vision encoder E'V8 generates the base embedding f. which is then transformed into motion-specific embedding m and progress-specific embedding through separate heads. As for the language encoder ELg. employ it to generate the language embedding (The of motion without updating its weights during training. Our goal is to determine the motion class and motion progress by evaluating the similarity between the embeddings and p and the language description 1mo respectively.
Objectives. To effectively evaluate progress within a stage, expect three key capabilities in the MPE model: (1) the ability to discern the temporal difference of frames extracted from the same video, (2) the capacity to identify the motion in which the agent is engaged, and (3) the capability assess the progress motion completion. To this end, we introduce objectives based on InfoNCE 43 to guide MPE model, focusing time contrast, motion contrast, and language-frane frame contrast. The functions the three objectives are illustrated in Figure Zin Appendix.
For implementation, utilize the negative L2 distance the similarity function S. During each training iteration, we sample B videos focusing on specific motions and N videos featuring arbitrary agent movements, from which we randomly select batch of frame sequences [F,, Fjzi, Fkzz]-
Time Contrastive Loss. To capture features relevant to physical interaction and sequential decision- making, we apply the time contrastive loss Lton the base embeddings, as inspired by 44 45) Essentially, images that are temporally closer should have more similar representations (embeddings) than those that are temporally distant or from different videos. The loss Leen can be formulated by
e SU
Lion = Σ
Σ
log e SU2 - S(.
(I)
be
(a,y)e
((1)).(1.)
where f is the base embedding of the x' 'tth frame.
Motion Contrastive Loss. To differentiate between various motions, we introduce the motion contrastive loss Lmon. which aligns each motion's embedding with its relevant language embedding and separates it from unrelated language embeddings. Additionally, to reduce reward hacking and enhance the robustness of our MPE model, task it with distinguishing frames from motion videos or meaningless videos that contain arbitrary movements. The loss Lmen is formulated by
e ())
Emon
Σ
log
Smm ()) + 139)
BE (B.)
JE (t.jkk)
where b is the motion embedding of th frame, 2 is the language annotation of motion in the video b, and 1 is the instruction for other motion.
During RL training, agents may explore areas not covered in demonstrations and exploit reward model vulnerabilittie, leading high rewards for incorrect actions.
5

---

Published as a conference paper at ICLR 2025
Language-Frane Contrastive Loss. To assess progress within the motion, employ the language-fran frrmme contrastive loss Lifen. The loss Lifen aims to bring the progress embeddings of nearly completed steps closer to the instruction embedding of the motion while distancing the progress embeddings of frames from earlier steps, which can be stated by
Lifen
log e S(r + S03 + (
(3)
DEB
where ph is the progress embedding of th frame, and 1 is the motion instruction for the video
Total Loss. Finally, we employ the Adam optimizer and the total loss Liotat to guide our MPE
model, where Leatal a weighted combination three objectives mentioned above:
Liotal = Ai + A2emen + XzCifen
(4)
Inference. During inference, along with the stage detected the Stage Detector, the MPE model evaluates the current motion from the motion candidates L within the stage, provided by the task knowledge from Section 4.1 Specifically, it calculates the similarity scores between the motion embedding m of current observation and language embeddings mo of each motion candidate mo E L M, then selects the motion with the highest score, as formulated by
motion
argmax (m,,"mo) mog 1,M
To determine the confidence level of the MPE model, further calculate the confidence score by comparing the similarity difference between the motion embedding m and the language embedding pmotione of the selected motion, and the language embedding (" of arbitrary agent movements. We decide whether to choose the selected motion motions or the first motion by
motion* confidence > d
motion
where S(m, mootions
(6)
0.
otherwise
confidence 5mm, /motion+) + 1(m,11")
(7)
Finally, given the determined motion, the progress within the motion by calculating the similarity score between the progress embedding p and the motion's 's language embedding motion,
[p(p, mmotion), if confidence > A
progress
10
otherwise
(8)
4.4 REWARD FORMULATION & POLICY LEARNING
Considering the information acquired from previous sections, we now have knowledge regarding the current stage of the agent, its ongoing motion within the stage, and the progress within that motion. We aggregate this information and the task knowledge into a measure of the task's 's potential, representing the overall progress within the task:
sstage
(00)) = Am
#motions; + motion
+ progress
==0
in-motion progress
preceding motions
where (0o;) denotes a potential function conditioned on the image observation at timestep t, stage is the number of the detected stage in the task. iont is an integer indicating the total number of

---

Published as a conference paper at ICLR 2025
different motions within stage i, motion is the integer representing the number of the detected motion in the stage, and Xm is a constant near the maximum of progress, ensuring ((oz) increases across motions and stages. Additionally, progress is float determined by the MPE model. Note that the number of stage, #motions; motion correspond to the sequence provided in the task knowledge. Using the potential, we implemented potential-based shaping rewards R(o,, ot-1) = x(or) ((0-1) from [46] to guide the agent's s behavior based on potential changes.
EXPERIMENTS
Our experiments aim to answer the following questions: (1) Does VICtoR provide effective rewards for long-horizon tasks? (2) Is VICtoR able generate dense rewards from real-world videos? (3) Are all reward signals for stage, motion, and progress indispensable for learning effective rewards in long-horizon manipulation tasks, and do they capture accurate information?
5.1 EXPERIMENT SETTINGS
(A) Stanlizedd Emirooment
(9) Real World Envinement
Evaluation Settings.
Most existing manipulation bench- marks 47 48 49] do not support long-horizon tasks
lynt
StWAR Idft pean
deverr
where the success of each step depends the outcomes of previous steps. While Calvin [50] addresses the aforemen- tioned issue, the tasks and action spaces (e.g., rotate) are too challenging for VIC reward models trained on action- free videos. Consequently, ment to test different VIC reward models for long-horizon manipulation. For policy training in this environment, we use a ground-trutt object detector to speed up the process, as we need run multiple tasks with multiple seeds. For real-world experiments, we train and test reward models using the XSkill [7 dataset. Details on the environment, tasks, and data are in Appendix C.T and Figure 3
box
developed new environ-
Essizvesa
Teai
Datavt
Sindllact
Nachere Dtano
Figure 3: Environment information. Tasks are generated from permutations of actions on interactable objects shown in the figure.
Baselines. We compare VICtoR with the following baselines: (1) Sparse Reward: A binary reward function assigns a reward only when the task succeeds. (2) Stage Reward: A reward function that assigns reward equal to the stage number when the agent reaches new stage. (3) LOReL [51]: A language- conaitiioood reward model that learns classifier fo oz. 1) to evaluate whether the progression between the On and o, aligns with the task instruction Note that, we replaced the backbone of LOReL to CLIP for fair comparison with VICtoR and LIV. (4) LIV [I6]: A vision-language representation for robotics that can be utilized as a reward model by finetuning on target-domain data. We apply the same reward shaping method for LIV and LOReL discussed in Section 4..4 (5) VICtoR (task): A baseline for VICtoR trains on task-level data, using the same MPE objective but with demonstrations of tasks not divided into stages motions. Note that the training demos for VICtoR are action-free free videos with text instructions; therefore, we do not compare method with other works requiring action data, such contuagie-ondiioneeed imitation learning.
5.2 POLICY LEARNING WITH VIC REWARDS
In this section, we train the Proximal Policy Optimization algorithm (52 using various reward functions on the simulated environment. We utilize a sparse reward as signal for task success and shape the reward function using different reward models. Our experiments are divided into two sets: initially, we train on single-stage tasks with varying numbers of motions, and subsequently, on tasks with varying numbers of stages. As shown in [[ VICtoR consistently outperforms current baselines, achieving a 25% average performance gain. This advantage increases to 43% in harder tasks with more than three motions, highlighting VICtoR's 's effectiveness in training for long-hoe horizon tasks. The significant benefits of the motion design are also evident when compared to the task- baseline, especially in more complex tasks. Furthermore, as Table I demonstrates, at the task level. VICtoR outperforms other baselines, showcasing the effectiveness of our training objective detailed in Section 4.3 Training details and curves are provided in Appendix C.4
7

---

Published as a conference paper at ICLR 2025
Table 1: Experiment of tasks with one stage but different numbers of motions. The table presents the success rate [1]] of the learned policy using different reward functions across three tasks that have one stage but vary the number of motions. It shows that the differences between VICtoR and other baselines become larger as the number of motions in the tasks increases. This indicates that when stage requires more detailed actions to complete, becomes necessary to decompose the task to assess progress. Results are averaged over three different seeds.
#motions
2
4
Task
reach block
pick block
move block
Sparse
99.8%
0.0%
0.0%
LIV [6
99.9%
98.6%
0.0%
LOReL [5]
100.0%
88.8%
0.0%
VICtoR(task) 99.9%
94.2%
36.8%
VICtoR 99.9%
98.8%
58.9%
Table 2: Experiment of tasks with different numbers of stages and motions. Extending from Table [ this table includes seven additional tasks with varying numbers of stages and motions. It shows that VICtoR outperforms prior methods, and the differences become larger as the complexity of the tasks increases, demonstrating the effectiveness of our design of stage and motion rewards. Results are averaged over three different seeds.
#sstgge
3
3
#motions
Task
open box
open drawer
open drawer open light
open box pick block
open drawer + move
open light + open drawer reach block
open open light +
pick block
Sparse
100.0%
79.5%
0.0%
0.0%
0.0%
0.0%
0.0%
Stage
100.0% %
79.5%
26.0%
0.0%
0.0%
LIV [6]
99.9%
66.7%
0.0%
0.0%
0.0%
0.0%
0.0%
LOReL [S]
100.0%
100.0%
33.1%
35.1%
0.0%
35.0%
28.9%%
VICIoR
99.9%
100.0%
66.1%
96.4%
30.6%
70.9%
57.7%
5.3
ABLATION STUDIES
To better understand how each reward signal in VICtoR contributes to policy learning, we conducted an ablation study on the "open box then pick blue block" task evaluate the effectiveness of VICtoR's stage determination, motion determination, and progress assessment as introduced in Section Based on the findings presented in Table we observe that injecting reward signals at each level proves beneficial for policy learning. Furthermore, our analysis indicates that reward signals at the "progress" level notably enhance performance. This underscores the importance of nuanced reward signals, particularly in tasks with longer horizons, thereby supporting our claims. Ultimately, the combination of these three levels of reward signals enables the policy to achieve peak performance.
Table 3: Ablation study. This table illustrates the effectiveness each level of reward signals in VICtoR for policy learning. Results are averaged over three different seeds.
Ablation Settings
Success Rate
Stage
Motion
Progress
0.0%
22.8%
/
v
29.7%
/
/
v
96.4% %
5.4
QUALITATIVE ANALYSIS
We plot the potential curves from three different reward models across four tasks with varying complexities (with longer horizons on the right). The results Figure demonstatte that VICtoR effectively generates reward curves for complex, composed tasks, while other baseline models fail to
8

---

Published as a conference paper at ICLR 2025
provide progressive signals task horizons grow. Notably, while prior reward models trained on full-length, munghorizon demonstrations, VICtoR was only trained on motion-level data yet still achieved the most optimal results. We also put the analysis for misordered action in Appendix B.1
drnwer
(b) hiw block
open box open then pick
open spen denver blue
L.ORel.
open
block
(c) open box open light then pick blue bleck
drower then reach bluee block
(4) open light then open
VICtoR
drawer
(b) bls block
(c) oper bos then open light then
(d) open light thet open
pick blue bacck
drawer then reuch bloc
Figure 4: Potential comparison across different tasks: different reward models. In these comparisons, we can see that VICtoR provides the most progressive and near-strictly increasing potential function, especially as the horizon increases. This demonstrates its ability provide fine-grane fine-grained rewards for long-hoo hong-horizon tasks.
We compare the potential generated by
5.5 REAL WORLD EXPERIMENTS
To demonstrate VICtoR's practical efficacy, we visualized the potential curves Φ(ox) for two scenarios: one with videos that match the corresponding instructions, and another with incorrect videos using the same instructions. As illustrated in Figure [ for the correct actions, the potential curves show that VICtoR effectively identifies task progress in real-world scenarios by increasing the potential as the agent completes tasks, capability not matched by previous VIC reward models. For the incorrect actions, as agent moves from the right to the left side to close the drawer, VICtoR's visualization shows initial increase in potential. This increase is logical as these movements approach the light, aligning with the first stage of the instruction, "open the light." However, as the agent continues toward the drawer, VICtoR recognizes the incorrect task and begins to decrease the reward, effectively deterring the agent's 's movement. This highlights VICtoR's s ability to analyze agent movement and task progress accurately. Additional visualizations for various tasks are provided in Appendix
5.6 VISUALIZATIONS
Embedding Visualization. To verify the effectiveness of the contrastive objectives designed to boost understanding of task progress, we present a t-SNE analysis on motion embeddings m across every motion in simulated environment. From Figure 6 observe that the learned embeddings

---

Published as a conference paper at ICLR 2025
With the right actions
With the wrong actions
cel
VICioR
VICIoR
Time
Time
lested open light upen oven Instraction: opan light then open ovan door then close the drewar
video:
drawer
instruction open light then open oven doee then close the drawer
Figure 5: VICtoR on real world data. This figure displays LIV 16 and VICtoR's potential
visualizations for bong-horizon tasks from XSkill with correct and incorrect videos.
of identical motion are clustered, indicating high discriminative capability. This attribute is crucial for enhancing the accuracy and efficiency of our reward model, particularly long-horizon tasks.
Reward Visualization. To assess VICtoR's s ability to differentiate motions and assess progress within a motion, we visualize motion determination per timestep and measure the embedding z's distance between motion descriptions and frame embeddings. As shown in Figure (6 VICtoR accurately switches motions at the appropriate timestep and decreases the embedding distance the determined motion as the agent approaches each motion's 's goal.
(b) Polentia
candidates (Tack: move drswer)
Limbedding Visulization
Roach the Suoe Blook
lhe bius boock
Move slse brock top of the thinerr
Place the blue block
-50
-100
100
Time
Figure 6: Visualization. (a) t-SNE analysis for different motion frames, and (b) analysis of negative embedding distances and determined motions in a demonstration video.
6 CONCLUSION
We have presented VICtoR, a Hierarchical VistonnInstrruction Correlation Reward Model for Long- horizon Robotic Manipulation. Using only language instructions, VICtoR is able to provide effective rewards to the reinforcement learning agent. Having been trained solely modular short-horizon demonstration videos, VICtoR outperforms state-of-the-art approaches trained on targeted long- horizon tasks and successfully operates in both simulated and real-world environments.
Future works and limitations VICtoR can provide rewards for unseen long- long-horizon tasks com- posed of seen motions. However, similar to previous works, the limitation of VICtoR is that it cannot be applied to tasks involving unseen motions. Therefore, future direction for this project will be training with diverse range of motion data explore its zero-shot capabilities unseen motions.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by the National Science and Technology Council, Taiwan, under Grant NSTC 113 2--226633-00--0000 and NSTC and area chairs for their valuable comments and positive recognition of our work during the review process.
We also thank all reviewers
REFERENCES
[1] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbecl. Overcoming exploration in reinforeement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292 6299. JEEE, 2018.
[2] Abhishek Gupta. Vikash Kumar. Corey Lynch. Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1910. 1956, 2019.
[3] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning:
A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-33 35,,, 2021.
[4] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICMI. '00, page 663-670, San Francisco, CA,, USA, 2000. Morgan Kaufmann Publishers Inc.
[5] Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement leaming. Proceedings, Twenty-Firg First International Conference on Machine Learning, ICML 2004, 09 2004.
[6] Avi Singh, Larry Yang. Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-on to-ett- robotic reinforcement learning without reward engineering. arXiv preprint arXiv: 1904.07854, 2019.
[7] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via cillecimplicit pre- pre-training. arXiv preprint arXiv:2210.0003: 10000000,,0000 2022.
[8] Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from "in-the-wild" human videos. ArXiv, thss/2003.1681 2021.
191 Minttu Alakuijala, Gabriel Dulac-Amold, Julien Mairal, Jean Ponce, and Cordelia Schmid. Learning reward functions for robotic manipulation by observing humans, 2023.
[10] Xuzhe Dang. Stefan Edelkamp, and Nicolas Ribault. Clip-motion: Learning reward functions for robotic actions using consecutive observations, 2023.
[11] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification grounded natural language. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14743 -14422. PMLR, 17-23 Jul 2022.
[12] Sumedh A Sontakke, Jesse Zhang, Sébastien MR Arnold, Karl Pertsch, Erdem Biylk. Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: one demonstration is enough learn robot policies. arXiv preprint 1rx::23110007899, 2023.
[13] Jingyun Yang, Max Sobol Mark, Brandon Vu, Archit Sharma, Jeannette Bohg, and Chelsea Finn. Robot fine-tuning made casy: Pre-training rewards and policies for autonomous real-world reinforcement learning. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023.
[14] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision languael models are zero-shot reward models for reinforcement learning, 2023.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[16] Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. arXiv preprint mmXiv:2306.00956 2023.
11

---

Published as a conference paper at ICLR 2025
[17] Mengda Xu. Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. XSkill: Cross embodiment skill discovery. In 7th Annual Conference on Robot Learning, 2023.
[18] Dhruv Shah, Blažej Osiński, Sergey Levine, al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. Conference on Robot Learning, pages 492-504, PMLR, 2023.
[19] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. LIm-planner Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.
[20] Wenlong Huang, Picter Abhcel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 91 18-9147. PMLR. 2022.
[21] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for emhodied control, 2023.
[22] Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for roboties: Design principles and model abilitics. IEEE Access, 2024.
[23] Wenlong Huang, Fei Xia. Ted Xiao, Harris Chan, Jacky Liang, Pete Florence. Andy Zeng. Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian lchter. Inner monologue: Embodied reasoning through planning with language models, 2022.
[24] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Texi2motion: From natural language instructions to feasible plans. arXiv preprint 2r311:2303.12153 2023.
[25] Anthony Broham, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[26] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference Robot Learning. volume 164 of Proceedings of Machine Learning Research, pages 10949906.777 PMLR. 08-11 Nov 2022.
[27] Vivek Myers, Andre He, Kuan Fang. Homer Walke, Philippe Hansen-Estruuhh, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolohov, Anca Dragan, and Sergey Levine. Goal representations for instruction following: A semi-supervised language interface to control. arXiv preprint arXiv:2300.000 1.000000011,1,11 17, 2023.
[28] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and and-aanguagg navigation: Interpreting visually-grounde navigation instructions in real environments, 2018
[29] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action, 2022.
[30] Danny Driess, Fei Xia, Mchdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth. Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023.
[31] Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: zero-ffew-shot anomaly classification and segmentation, 2023.
[32] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chehotar, Xi Chen, Krzysztof Choromanski, Tianli Ding. Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan. Kehang Han, Karol Hausman. Alexander Herzog. Jasmine Hsu. Brian Ichter, Alex Irpan. Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang. Isabel Leal, Lisa Lce, Tsang-We Wei Edward Lee, Sergey Levine, Yao La, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker. Paul Wohlhart, Jialin Wu, Fei Xia. Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RI-2: Iision-lnguaage-attion models transfer web knowledge to robotic control, 2023.
[33] Pierre- Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, page 1726-1734, 2017.
12

---

Published as a conference paper at ICLR 2025
[34] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning, 2017.
[35] Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In International Conference on Learning Representations, 2019.
[36] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pages 1430-1440. PMLR, 2021.
[37] Siddharth Srivastava, Eugene Fang. Lorenzo Riano, Rohan Chitmis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible plamner-independent interface layer. In 2014 IEEF International Conference on Robotics and Automation (ICRA), pages 639-646, 2014.
[38] Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. 2020.
[39] Shuo Cheng and Danfei Xu. League: Guided skill learning and abstraction for honghooizoon manipulation. 2023.
[40] Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar, Mimicplay: Long-horizon imitation learning by watching human play, 2023.
[41] Zhiwei Jia, Vineet Thumuluri, Fangchen Liu, Linghao Chen, Zhiao Huang, and Hao Su. Chain-of-thouglle predictive control. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 21768-21790 PMLR, 21-27 Jul 2024.
[42] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[43] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0311 2018
[44] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang. Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141, 2017.
[45] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[46] A. Ng. Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999.
[47] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
[48] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 -026,, 2020.
[49] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. VImbench: A compositional benchmark for lannauugg manipulation. Advances in Neural Information Processing Systems, 35:665 1166656788.20 2022.
[50] Oier Mees, Lukas Hermann, Erick Rosete-Beas and Wolfram Burgard. Calvin: benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks, 2022.
[51] Suraj Nair, Eric Mitchell, Kevin Chen, brian ichter, Silvio Savarese, and Chelsea Finn. Learning language- conditioned robot hehavior from offline data and crowd-sourced annotation. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors. Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1303-1315. PMLR, 08-11 Nov 2022.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint a7xiv:1707.063447, 2017.
[53] Coppelia Rohoties. Coppeliasim software.
cc..ccc..cooo comm
13

---

Published as a conference paper at ICLR 2025
[54] Stephen James, Mare Freese, and Andrew J. Davison. Pyrep: Bringing v-rep deep robot learning. arXiv preprint arXiv:/906.1 1176,, 2019.
[55] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[56] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-haselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021.
[57] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[58] Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, and Luca Weihs. Universal visual decomposer; Long-horizen manipulation made easy, 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A ILLUSTRATIONS FOR LOSS OBJECTIVES
In-motion contrastive
Crose-motions contrastive
Longugge contrastive leaning
Notion contractios learning
Time costraative leamiin
Figure 7: In-motion and cross-motion Contrastive Objectives: We illustrate how the three contrastive loss functions we applied manipulate the distances and relationships between instruction representations and encoded visual observations within the embedding space.
B ADDITIONAL EXPERIMENTS
B.I QUALITATIVE ANALYSIS MISORDERED ACTION
We visualize the reward by giving text goals for long-horizon tasks (open light then open drawer then reach blue block) misordered action (reach blue block) videos the difference in reward models. The result in Figure X indicates that VICtoR is capable of determining the current state of environmental objects decide the level of potential while LIV simply generates progressive signals on each task.
(8) LIV
(b) VICtoR
(a) LIV
(b) VICIoR
Figure 8: Potential Visualization Misordered Action: We visualize the rewards given for long-horizon tasks (open light, then open drawer, then reach blue block) in videos with misordered actions (reach blue block) compare different reward models. The figure shows that VICtoR can determine the current state of environmental objects to decide the level of potential, while LIV simply generates progressive signals for each task without penalizing incorrect actions, which may lead to poor learning outcomes.
15

---

Published as a conference paper at ICLR 2025
B.2 ACCURACY OF TASK KNOWLEDGE GENERATION
To validate the robustness of LLMs for task knowledge generation, conducted additional experiment testing the accuracy LLM outputs. We tested our model by randomly selecting tasks, which could involve 1 to 4 stages. We then used GPT-4 to translate these tasks into human-like queries. With these queries, we tested the task knowledge generation model by comparing its output against the ground truth. After randomly sampling 50 different tasks with varying free-ffrm instructions, we observed that GPT-4 achieved an accuracy of 96% in decomposing tasks into stages, motions, and conditional object states, confirming its robustness for this application.
B.3 ABLATION STUDY ON THREE CONTRASTIVE LOSSES
To examine the importance and effectiveness of the three proposed contrastive objectives in Eq. 4 we train the Motion Progress Evaluator under three conditionsIn each experiment, remove one contrastive loss, as illustrated in Table 4 This study conducted on the "open box then pick blue block" task. The results clearly demonstrate that each objective plays significant role, particularly the Lifen objective. This objective effectively differentiates the progress in motion, providing nuanced guidance for long-horizon tasks.
Table 4: Ablation Study the proposed three contrastive losses
Ablation Settings
Success Rate
Lien
Lyen
Lifen
0.0%
V
61.8%
V
V
94.0%
V
96.4%
B.4 ABLATION STUDY ON AUTOMATIC ANNOTATION BY LVLMS
Since object status labels are required deploy VICtoR a new environment, we conduct an ablation study to automatically annotate these labels using modern LVLMs. The study is performed on the XSkill dataset, focusing on four interactable objects: cloth, light, oven, and drawer. Specifically, for each video where the robot interacts with one of these objects, first use MDETR (without fine-tuning) to obtain cropped images of the target object. These cropped images are then provided to GPT-4V, which annotates the object's status. Finally, we compare the consistency between human-annotated and LVLM- M-----tnttteed labels, as summarized in Figure
The results demonstrate high level of consistency between the two sets of labels, indicating that deploying VICtoR in new environments is highly feasible. Most inconsistent labels arise during fuzzy phases when switching motions. These inconsistencies are easily corrected or even negligible.
C DETAILED SETTINGS FOR THE EXPERIMENT
C.I DETAILS ABOUT OUR SIMULATED ENVIRONMENT AND TASKS
Details about the environment Our environment was developed using Coppeliasim [53] with Pyrep (54) serving as the coding interface, as shown in Figure In this environment, there are four interactable objects: a light, a drawer, blue block, and box, each associated with different stages The tested tasks can be any permutation of these stages.
as detailed in Section C.I
List of Stages and Motions. We treated the motions the action primitives for agents to implement. Definitions of the motions in each stage in our experiments are detailed as shown in Table 5] The definition of motions of stages could be initiated by either humans or GPT.
16

---

Published as a conference paper at ICLR 2025
label consistency
Cloth
96.88%
The dish hung en the Saucet
The cloth s grasped by robot
The clthh inside the sink
Light
94.87%
The light off
TTe fight on
90.63%
Clssed
opened
opered
Drawer 88.10%
The drower is opened
The drawer half cificann closed by robot
The drawer is closed
Figure 9: Automatic Annotation by LVLMs: We leverage LVLMs (i.e., GPT-4V) to automatically annotate the status of objects the XSkill dataset. The results are then compared with human- annotated labels to evaluate label consistency. The findings indicate that modern LVLMs already possess the capability to handle this task effectively. Most inconsistencies occur during the fuzzy phases when switching motions.
Table 5: Description of Separated Motions for Each Stage
Stage
Motion
Motion 2
Motion
Motion
reach blue block
reach the blue block
pick blue block
reach blue block
grasped
blue block
move blue block to drawer
reach the blue
lift the grasped
move the blue
place the blue block down the drawer
block
blue black
black the top of the drawer
open drawer
reach the closed drawer handle top reach the opened drawer handle top
pull the drawer out
close drawer
push the drawer forward slide the box
open box
reach the box holder back reach the box holder front reach and push down the button
holder forward
close box
slide the box
holder backward
open light
close light
reach and push down the button
Training Tasks Details. The detailed success conditions, training timesteps, and the running machines for each task are shown in T
Training Data Details. To train the reward models for VICtoR and other baselines, we collected a dataset in our environment. Note that VICtoR requires only motion-level data for training, while other baselines need full-length videos. Therefore, the data used train VICtoR and other baselines differ type but are comparable total amount per motion or stage. For VICtoR, we provide only the motion-level demos, with each motion accompanied by 150 demos and corresponding motion
17

---

Published as a conference paper at ICLR 2025
Table 6: Training Details for each task
Reach blue block
Task Goal
Reach the blue block
Success Condition Max Training Step GPU
The distance between the end-effector and the blue block
30000
NVIDIA RTX 4090
Pick blue block
Task Goal
Grasp the blue block and lift up
Success Condition
The bluue block is lifted higher than 0.62
Max Training Step
100000
GPU
NVIDIA RTX 4090
Move blue block drawer
Task Goal
Pick up the blue block and place into the druwer
Success Condition
The position of the blue hlock (if is in drawer or not)
Max Training Step GPU
300000
NVIDIA RTX-4990
Open drawer
Task Goul
Pull out the drawer
Success Condition
The position of the drawer
Max Training Step
50000 NVIDIA RTX 4090
GPU
Open box
Task Goal
Push the lid of the box open the box
Success Condition
The position of the box lid 50000
Max Training Step
GPU
NVIIJA RTX 4090
Open drawer then open light
Task Goal
Pull out the drawer and then press the button open the light
Success Condition
The position of the drawer and the color of the light
Max Training Step
200000
GPU
NVIDIA RTX 3090
Open box then pick blue block
Task Goal
Push the lid of the box to open it and pick up the blue block The position of the box lid and the height of the blue block
Success Condition
Max Training Step
200000
GPU
NVIDIA RTX 3090
Open drawer then move blue block to drawer
Task Goal Success Condition Max Training Step
Pull out the drawer and then pick up the blue block and place into the drawer The position of the blue block (if it in drawer or not) 400000
GPU
NVIDIA RTX 3090
Open light then open drawer then reach blue block
Task Goal Success Condition
Press the button to open the light, then pull out the drawer. and reach the blue block
The color of the light, the position of the drawer, and the distance between the end-erfector and the blue block
Max Training Step
400000
GPU
NVIDIA RTX 3090
Open box then open light then pick the blue block
Task Goul
Push the lid of the box to open it and press the button
to the light then pick up the blue black
Success Condition Max Truining Step
The position of the box lid, the color of the light, and the height of the blue block 400000
GPU
NVIIIA RTX 3090
Open box then open light then open drawer then then reach the blue block
Task Goal
Push the lid of the box to open it and press the button to turn on the light and open the drawer then reach the blue block
Success Condition
position box lid, the color and the distance between the end effeectr and the blue block
light, the position
drawer,
Max Training Step GPU
500000
NVIDIA RTX 3090
descriptions. For training other baseline models, such as LIV [6] and LOReL [51], we provide the full-length demo videos, each paired with task descriptions and consisting of 150 demos per task.
18

---

Published as a conference paper at ICLR 2025
C.2 DETAILS OF STAGE DETECTOR AND TRAINING DETAILS OF MOTION PROGRESS EVALUATOR
The structure of classifier P for Stage Detector includes CLIP Text Encoder for processing object state descriptions and CLIP Image Encoder for processing cropped images. These encoders are followed by downstream MLP classifier that scores relevance.
We provide more details of the Stage Detector and Training Details of the Motion Progress Evaluator in Table 7 and Table 8 For the hyperparameter selection of 21,2,3, tried various combinations but found that the model is not sensitive to these hyperparameters. Therefore, we have chosen to set them all to the same values.
Table 7: MDETR configuration VICtoR
Object Detector .DDTRR [42]
Backbone Resnet101
Table 8: Details of VICtoR Motion Reward Model and Stage Determinator Training
Motion Progress Evaluator
Language Encoder
CLIPTextModel CLIP ModionModel
Image Encoder
CLIP Model
openai/clip- vittbbase-atth33
Learning Rate
0.000001
Batch Size
32 20000
Training Steps
Optimizer
Adam (55
Loss Function
Equation
Parameters in Equation
\1 1, A2 A3
Stage Determinator
Language Encoder
CLIPTextModel Codd
Image Encoder
CLIP Model
openai/clip-vite vit-base-patchh3
Learning Rate
0.00001
Batch Size
32
Training Steps
40000 Adam Binary Cross Entropy Loss
Oprimizer
Loss Function
C.3 DETAILS OF MPE'S INFERENCE AND REWARD FORMULATION
We organize our reward function parameters Section 4.3 in Table 9 For the selection Ams we tried find the biggest range of the progress (embedding distance from language to motion), in order to make the proceeding motion's 's potential higher than the previous one. For the selection of Xc, we have tested with different numbers and found out that when the A is in the range of 0.1 to 0.4, the model has similar results, and if it is less than 0.1, it cannot avoid reward hacking. Once it is larger than 0.4, the model cannot provide enough signal for the agent to learn.
Table 9: Reward formulation and MPE model parameters of VICtoR
A
Am
0.2
36
19

---

Published as a conference paper at ICLR 2025
C.4 TRAINING DETAILS OF POLICY TRAINING
Training details. We trained the policy under the Subproc from stable-baseline3 [56] with reward normalization by VecNormalize. For online interactable environment, we built simulation environment with Coppeliasim [53] and Pyrep [54] and wrapped the environment for policy learning under the framework of OpenAl Gym [57]. Training time is about 3 hours for 10000 steps. Training details for each task and policy are summarized in Tables and 10 respectively.
Table 10: Details of policy training
Policy
PPO [52]
Learning Rate
0.003
Policy Model
MipPolicy
Discount factor (?)
0.99
Reward Normalization
True
C.5 COMPUTATIONAL COST AND RESOURCE REQUIREMENTS
The computational cost in our framework is 2 GPU hours for the training of the progress evaluator. For the policy learning, training with our reward model for 20000 steps required approximately 12 GPU hours. Both of the stats above are running on RTX 4090.
Regarding the inference time of the current implementation, the object detector takes approximately 51.5ms, and the reward model accounts for 24ms average over 1000 steps (tested CPU Intel(R) Core(TM) i7-13700 and a GPU RTX 3090). Considering that end effector action latency is typically long in both simulation and real-world scenarios, believe that a 0.05s inference time would not significantly impact the agent's efficiency, even in real-world environment.
C.6 TRAINING DETAILS OF BASELINE MODELS
We provide details of other baselines in Table
Table 11: Details Baseline Reward Models Training
LIV
CLIP Backbone
Resnet50 pretrain
Learning Config
Batch Size
32
Train Steps
70000
Learning Rate
0.00001
Weight Decay
0.001
LOReL
Language Encoder
CLIPTextModel
Image Encoder
CLIP Model
Batch Size
32
Train Steps
20000
Learning Rate
0.00001
Flipped Negative
True
alpha
0.25
20

---

Published as a conference paper at ICLR 2025
C.7 DETAILS OF EXPERIMENTS ON XSKILL
For the visualization experiments on real-world dataset XSkill [17 in Appendix FF we train VICtoR on 120 demonstrations with each breaking down into 3 different motion-level videos while baseline models are trained with the full-length demonstrations. Details about the 4 motion level skills and the test tasks in the visualization experiments are listed in Table 12
Table 12: Text description for motions and test tasks in XSkill
Motion Name
Description
close the drawer open the oven door
push the drawer so that it is closed open the oven door push the light on grab the cloth and release it into the sink
open the light move the cloth into the sink
Test Tasks
close the drawer then move the cloth into the sink then open the light close the drawer then move the cloth into the sink then open the oven door close the drawer then open the light then move the cloth into the sink close the drawer then open the light then open the oven door open the light then move the cloth into the sink then open the oven door open the light then close the drawer then move the cloth into the sink open the light then open the oven door then close the drawer open the oven door then open the light then move the cloth into the sink
D LEARNING CURVES OF POLICY LEARNING
We evaluate the policies trained from different reward functions with the success rate in Figure 10 the shadows show the standard deviation.
E ADDITIONAL COMPARISONS BETWEEN VICTOR AND OTHER SOLUTIONS TO ROBOTIC POLICY LEARNING
Imitation Learning. Prior works have attempted to solve the robot manipulation tasks via imitation learning from either robot demonstrations human vidcos. However, in our setting, do not use action sequences from demonstrations to train VICtoR. Thus, the line of imitation learning works is not considered as baseline. Our objective is to learn reward model solely from vision and language inputs, without relying on action or observation vectors. This distinction sets our approach apart from imitation learning and other similar methods, which typically require action information to clone behavior. Therefore, our method addresses a unique problem space that is not directly comparable imitation learning.
Motion Decomposition Previous studies 41 58 have utilized motion decomposition to facilitate learning in long-horizon manipulation tasks. By breaking down these tasks into subgoals, they demonstrated that imitation policy networks could be trained more effectively. While we also use the concept of task decomposition, our method introduces tailored framework for VIC reward models, as shown in Figure[] We segment tasks into three levels: stage, motion, and progress, providing finer- sree-ggaiinddd approach to handle task complexities. Our experiments show that this structured method significantly improves the effectiveness of VIC reward models in complex scenarios.
Task and Motion Planning (TAMP). Prior works investigated through accomplishing long-horizon robot manipulation tasks via arranging trained skills. While TAMP methods require a predefined and well-trained skill set and put more emphasis on planning for the execution sequence of these trained skills, our model aims to provide reward signals directly toward policy learning to guide the agent to learn a oonghoorzon policy from scratch with reinforcement algorithms. Therefore, we do not consider TAMP methods baselines for VICtoR.
21

---

Published as a conference paper at ICLR 2025
(6) neach block
(b) openn box
(4) open drawer
rellout (wu.us/ssccsssss. pabs
rellostaucees_sse rabe
rolloutaacoes_.m r8te
dhiulin Frim
Chon F01 1u0t Woil U0tts
ViD0R 108t) State
Stage
0.55
0.5
3.5
23
gotbl
poba Lap
pibbll ptop
pick blue block
(e) open box then pick blue block
blue block drowor
rolloutissscosss
Show L0Pt
0.8
3.5
04
0.4
0.
02
Edba
254
444
50t
10k
23c
1500
150%
2004
2036
(g) open drower then open light
(h) open light then open drawer then reach blue block
rollost/succes..o
Shiving
Slowing
0.5
0.4
04
#obal sto
gloual Stop
50k
33%
1534
185x
2006
2004
(6 open box than open light then pick blue block
() open drower then move block to drow
To mllost/uccess.ooo ra8e Slowing
2.
0.6
3.4
2006
2004
Figure 10: Training Curve Visualization: We visualize the success rate across various training episodes and display the success rate curves for different methods in different tasks. The figure demonstrates that VICtoR effectively guides the RL. agent to learn quickly and achieve the highest success rate in the fewest episodes, highlighting the effectiveness of the rewards generated by VICtoR.
22

---

Published as a conference paper at ICLR 2025
F
MORE POTENTIAL VISUALIZATION FOR COMPOSED TASKS IN XSKILL 17
The visualization results are shown in Figure 11 and Figure
(a) close the drawer then move the cloth into the sink then open light
(b) close the drawer then move the cloth into sink then open the oven door
close the drawer then open the light then cloth into the sink
(d) open the light open the oven door
Figure 11: Potential Visualization on XSkilI: Evaluation of the potential curve across multiple test videos on different tasks for XSkill.
23

---

Published as a conference paper at ICLR 2025
(o) open the light then close the drawer then move the cloth into the sink
open the light then cloth into the sink then open door
!
(g) open the light then open the oven door then close the drawer
(h) open the oven door then open the light then move the cloth into the sink
Figure 12: Potential Visualization on XSkill: Evaluation of the potential curve across multiple test videos on different tasks for XSkill.
24

---

Published as a conference paper at ICLR 2025
G PROMPTS FOR TASK KNOWLEDGE GENERATION
You are task splitter. Given task, you should split it into stages and motions. Also, during transitions between stages, you must provide the environment status.
# Possible Motions
+++ drawer
reach the open drawer holder top
push the drawer forward
reach the closed drawer holder top pull the drawer out
44 box (slide forward to open, alide backward to close)
reach the box holder back
slide the box holder forward
reach the box holder front
slide the box holder backward
#* Light
reach and push down the button
#+ blue_block
reach the blue block
lift the grasped blue block
the blue block the top the drawer
nove the blue block to the top the table
place the blue block down to the drawer place the blue block down to the table
#* Envizonment Objects and States
The environment contains these objects: ["blue_block", "box", "drawer",
4 "light"]
#+# Possible Environment Status Sets for Each Object
drawer: ["The drawer is closed", "The drawer is open"]
box: ["The box is closed", "The box is open"]
light: ["The light is closed", "The light is open"]
["The blue block is in the drawer", "The blue block is the table", "The blue block
the box"]
# Output Format
output should this JSON format:
''jsonn
["", ...l,
"stages":
"name": ""
"environment": "interaacted_obeectt"
// list all the interacted object environment statuses "motions".
b,
// should be the motion of the interact_oojects
"name": ""
25

---

Published as a conference paper at ICLR 2025
"environment": Linteracte_oobeet ""
// 1ist all the interacted object environment statuses
b,
"environment";
// list all the interacted object environment statuses
b
"mot ions" ;
b
# Guidelines
For every task, you should split the task into stages, motions, and alse provide the environment status during stages there are two or more stages.
#+ Notices
This task involves a robot arm that is initially far from all the + objects.
Therefore, unless mentioned in the task goal, the first phase should
4 be "reach xxx.
The definition of a stage is: Complete interaction with object.
If the task has interacted with multiple objects, the stage will
④ only change during the transition between interacted objects Only perform behaviors that are ment mentioned in the task.
In the environment for each stage, list the initial environment status for each object in that stage.
The listed of "environment" object for each stage should be all of the
4 objects
#interact_objecte should only contain the object that will be interacted
The adjacent interact enteract_object should be different
The motion in each stage can only be subset of the iiteracted_object
4 motion list
Example:
Task: open the drawer then open the box then move the blue block to the 4 table
Initial environment:
drawer: "The drawer is closed"
box: "The box is closed"
Light: "The light is closed"
bluc_block: "The blue block is in the drawer"
Output:
'jjson
"interact "interact_objeats". ["drawer", "box", "blue_blocks "stages":
"name"; "open the drawer", "Lntoratee_oobectt" "drawer" ":
"drawer": "The drawer is closed",
"box": "The box is closed", "light": "The light closed",
"blue block"; "The blue block is in the S draver"
26

---

Published as a conference paper at ICLR 2025
"motions":
"reach the closed drawer holder top", "pull the drawer out"
1,
"name": "open the box", "Interattedoojeett" "box" "environment:;
"drawer": "The arawer is open", "box"; "The box closed", "light"; "The light closed" "blue_block": "The blue block is in the S drawer"
"motions":
"reach the box holder back", "slide the box holder forward"
"name": "move the blue block to the table", "interated_obfer"" "blue_ block" "environment";
"drawer": "The orawer is open", "box": "The box is open", "light": "The light is closed", "blus_block" "The blue block is in the drawer"
"motions":
"reach the blue block",
"lift the grasped blue block",
"nove the blue block to the top of the 4 table"
"place the blue block down to the table"
Task: move the blue block
table
Initial environment:
drawer: "The drawer is open"
box: "The box is open"
light:
"The light is open"
blue_block: "The blue block
drawer"
output:
"jjonn
"interact_objeats": ("blue_block"), "stages";
"name": "nove the blue block to the table", " "interated object": "blue_blod block" " "environment";
"drawer"; "The drawer is open", "box": "The box is open", "light": "The light is open", "blue_block": "The blue block is in the 9 drawer"
1, "motions":
"reach the blue block",
27

---

Published as a conference paper at ICLR 2025
"lift the grasped blue block",
"nove the blue block to the top the 4 table"
"place the blue block down to the table"
Task: [(task_name)]
Initial environment:
drawer: (finitialddraeee
box: (initial_box]]
Light: (finitialllghtt
blue_block: [1initial_block]
Output:
Listing I: Prompt for Task Knowledge Generation
BROADER IMPACT
Our research is focused on developing reward model for manipulation tasks, aiming to accelerate the adoption of robotic applications in complex settings. While we haven't 't identified any immediate negative impacts or ethical concerns, it's crucial that we remain vigilant and continuously assess any potential societal implications as our work evolves.
28


Reviews:
[{"review_id": "CyIekrnIXr", "reviewer": "Reviewer_D5gq", "summary": "The authors propose a hierarchical reward model that provides reward signals for long-horizon manipulation tasks. First, It proposes using a Task Knowledge Generator to decompose a task into stages and identify the necessary object states and motions for each stage. It then adopts a Stage Detector to detect the stage and object states given the visual observations. Finally, with a Motion Progress Evaluator that assesses motion completion within stages, the framework gives the overall reward for the long-horizon task. While the task knowledge generator leverages LLMs, the other parts are trained solely on primitive motion demonstrations. The paper shows that their proposed framework produces better rewards for long-horizon manipulation tasks, thus facilitating policy learning with methods such as PPO.", "questions": "+ I didn't find ablation studies of the three contrastive losses used in training the motion progress evaluator. It would be better to know their contributions to the overall quality of the generated reward.\n+ The proposed approach utilizes motion decomposition of subtasks, a concept similar to the ones used in [2]. It would be helpful to add a discussion about it. In particular, I wonder will the proposed approach also works well at a potentially more granular motion level (e.g., a subskill in the Move Chair task in [2]).\n\n[2] Chain-of-Thought Predictive Control", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The approach explores how to leverage LLMs for providing task decomposition for long-horizon manipulation tasks by defining stages and motions as the task hierarchy. And it seems that LLMs such as GPT-4 work well for this setup.\n+ The design of the modules sounds reasonable, i.e., leverage MDETR for stage and object status detection and leverage CLIP for motion progress evaluator. The training losses are well-motivated. The results are good.", "weaknesses": "- The long-horizon tasks used to evaluate the proposed framework are a bit limited, e.g., even compared to a popular benchmark [1] proposed in 2020. A more diverse set of tasks would help to fully verify the usefulness and adaptability of this approach. \n\n[1] ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B6f1Py18kJ", "reviewer": "Reviewer_sb4X", "summary": "This paper proposes a method called VICtoR that utilizes LLM (GPT-4) and VLM (CLIP) to calculate the task rewards for long-horizon robotic manipulation. Specifically, it decomposes the language instructions to different stages with the help of GPT-4, and uses MDETR to detect objects and classify their states. Comparing the states with the desired states from GPT-4, the algorithm can determine if the stage has been completed. For the motions inside each stage, VICtoR trains the CLIP encoders with multiple contrastive losses to learn the task progress corresponding to the instruction. Combined with the above reward, a model is trained with self-constructed benchmarks and shows better performance compared to multiple baselines.", "questions": "1. Line 160, how are the objects states autonomously annotated during real-world experiments?\n1. Line 205-208, What if the object detector fails? Will using cropped images for object states classification have certain problems? For example, some object states are positional relationships that need a larger visual field to accurately determine them. What is the model structure for the classifier P?\n1. Line 471, why the curves are called ``potential curves''? Are the rewards calculated with collected videos? Do you use the rewards for action rollouts?\n1. How do you deal with the long inference time of LLM, when you are training the policy with the proposed design?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is intuitive. Decomposing a long-horizon task with LLM and utilizing VLM for the short-term progress evaluator are both reasonable and have been validated in previous works.\n1. The visualizations are intuitive. I can easily get the idea of most figures and they help to understand the paper. The reward curves and t-SNE graph prove the effectiveness of the proposed reward design and contrastive loss designs.\n1. The results shown in the paper are good, compared to other baseline methods.", "weaknesses": "1. The technical novelty is relatively limited. The proposed method mainly has two parts, LLM for decomposing the long-horizon task and CLIP for the short-term progress evaluator. These two parts have both been extensively explored. There have been multiple related works mentioned in the paper. Even in Line 186, the authors acknowledged that the differences are relatively nuanced.\n1. The experiments are conducted on a self-collected dataset, not the widely adopted CALVIN benchmark. As mentioned in Line 343-346, the proposed method has critical limitations when dealing with some meticulous actions like rotations.\n1. Some technical details are confusing and need further clarification. Please see the questions part below.\n1. Minors. Typo in Line 143 (as as)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xFiIDDzWrb", "reviewer": "Reviewer_R2Xa", "summary": "This paper introduces a hierarchical vision-instruction correlation method to estimate rewards for long-horizon robotic manipulation tasks. Specifically, the approach initially identifies task-aware objects and employs GPT-4V to associate the object status with the corresponding motion stage. Additionally, this paper proposes a contrastive learning-based method that evaluates the reward of visual observations by calculating the similarity between visual observations and language instructions. Experimental results validate the efficacy of this method.", "questions": "Please refer to the weakness.\n\nMy questions are:\n1. Could this method generalize to novel tasks?\n2. Could other methods achieve comparable results when fine-tuned on the target task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a hierarchical reward estimation method to provide vision-based reward for long-horizon manipulation tasks.\n2. This paper describes this method clearly.\n3. The simulation and real-world experiments prove the effectiveness of this method.", "weaknesses": "1. The method requires lots of relevant data for estimating visual object information, which makes it struggle to generalize to new task scenarios due to reliance on the object status classifier. Could this method generalize to any novel task? \n\n2. The comparison results are unfair. This method utilizes the robotic manipulation videos of the same tasks to train the contrastive learning method, while the compared methods such as LIV are pretrained encoders.  Would these vision-language correlation methods achieve comparable results when the encoder is fine-tuned on the target tasks?\n\n3. The conservative learning method for vision-language reward estimation is common. The stage and motion detection method seems to simply achieve state-based rewards through visual models by overfitting on specific manipulation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DqHBDA2qTh", "reviewer": "Reviewer_Hvoi", "summary": "This paper proposes VICtoR, a hierarchical reward model for long-horizon manipulation tasks that learns from action-free videos and language instructions. The proposed approach addresses challenges in task complexity, sub-stage awareness, and object state estimation through a stage detector and motion progress evaluator. VICtoR outperforms existing models by using a novel stage detector and motion progress evaluator, achieving a 43% improvement in success rates for complex tasks.", "questions": "Refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper effectively advances research by integrating LLMs for high-level hierarchical planning, which allows the model to assess the robot’s progress at each step.\n- It incorporates an innovative approach to extract relevant information from short video clips, enabling the system to understand key aspects of the task without viewing the entire sequence.\n- The methodology section is well-structured, clearly outlining the three core components of the proposed approach, aiding reader comprehension.\n- Results in the experiments section demonstrate substantial improvements over previous methods in complex task settings, showcasing the effectiveness of the hierarchical framework.\n- Reward visualizations effectively illustrate the approach’s ability to accurately assess performance.", "weaknesses": "- A more thorough evaluation of each individual component would clarify its contribution to the overall system’s performance.\n- Additional evaluations on zero-shot capabilities, specifically with unseen trajectories, would be valuable in understanding the approach’s generalization.\n- The paper lacks an assessment of the computational cost and resource requirements of the proposed method.\n- The reliance on GPT-4 could pose potential challenges, which are not thoroughly addressed; analyzing GPT-4 outputs in greater detail could reveal possible limitations.\n- A comparative discussion of other methods for long-horizon manipulation, such as hierarchical reinforcement learning (HRL) and task and motion planning (TAMP), is missing and would have enriched the analysis.\n- While the authors mention using data from XSkill3 for real-world experiments, they do not provide sufficient details on the dataset’s size, diversity, or the types of tasks involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)