VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation

## Abstract

Abstract We study reward models for long-horizon manipulation tasks by learning from action-free videos and language instructions, which we term the visual-instruction correlation (VIC) problem. Recent advancements in cross-modality modeling have highlighted the potential of reward modeling through visual and language correlations. However, existing VIC methods face challenges in learning rewards for long-horizon tasks due to their lack of sub-stage awareness, difficulty in modeling task complexities, and inadequate object state estimation. To address these challenges, we introduce VICtoR, a novel hierarchical VIC reward model capable of providing effective reward signals for long-horizon manipulation tasks. VICtoR precisely assesses task progress at various levels through a novel stage detector and motion progress evaluator, offering insightful guidance for agents learning the task effectively. To validate the effectiveness of VICtoR, we conducted extensive experiments in both simulated and real-world environments. The results suggest that VICtoR outperformed the best existing VIC methods, achieving a 43% improvement in success rates for long-horizon tasks.

## 1 Introduction

Reinforcement learning (RL) has been extensively studied for long-horizon manipulation [, , ]. However, crafting reward functions for these tasks is complex, as it typically requires access to true states or significant domain expertise. Consequently, there is an urgent need for a robust and accurate reward model for these tasks. Previous research has explored modeling reward functions via robotic expert demonstrations [, ], goal-images [, ], and human demonstrations [, , ]. Unfortunately, the required task specification materials, such as goal-images, remain costly and impractical.

Recently, methods that consider the vision-instruction correlation (VIC) as reward signals have emerged, providing a more accessible way for task specification through language. Specifically, these VIC methods view the reward modeling as a regression or classification problem and train the reward model on the given action-free demos and instructions. Pioneer approaches [, , , , ] studied using pre-trained Vision Language Models (VLMs) or Video Language Models, like CLIP [], to generate rewards by assessing the similarity between visual observations and language goals. Ma et al. [] explore the possibility of using human video and language data to pre-train a reward model, later fine-tuned with in-domain data. However, existing VIC studies are all limited to short-horizon tasks, such as “picking up a block" or “opening a drawer".

Figure: Figure 1: Problems in existing VIC methods and VICtoR’s solution. Training long-horizon task with existing VIC methods commonly suffer from the listed problems. To address these problems, we propose VICtoR, a hierarchical reward model that can decompose long-horizon tasks and assign rewards by identifying the stage, motion, and progress of the agent from visual observations.
Refer to caption: x1.png

Figure 1 illustrates three challenges we observe when applying existing VIC methods to long-horizon manipulation tasks: (1) No awareness of task decomposition: Failing to divide complex tasks into manageable parts limits adaptability. (2) Confusion from variance in task difficulties: Training a reward model on long-horizon tasks impairs the learning of reward signals and fails to generate suitable progressive rewards. (3) Ambiguity from lacking explicit object state estimates: Relying on whole-scale image observations can overlook critical environmental changes. For instance, when training for the task “move the block into the closed drawer", previous VIC models would assign high rewards for moving the block even if the drawer is closed, misleading the learning process.

To this end, we propose VICtoR, a hierarchical VIC reward model for long-horizon manipulation. Figures 1 and 2 depict its concept and architecture, respectively. VICtoR learns effective rewards for long-horizon tasks by hierarchically assessing overall progress. Specifically, it decomposes long-horizon tasks (high-level) into stages (mid-level) and motions and progress (low-level, robot primitives), determining the reward by considering task progress at different granularities.
By treating each long-horizon task as an arbitrary permutation of available motions, VICtoR, trained only on action-free motion videos, can be easily adapted to unseen long-horizon tasks.

To achieve this, VICtoR first queries GPT-4 to break down a long-horizon task into stages to acquire essential task knowledge, including the expected object statuses and required motions for completion at each stage. Then, a stage detector (a local-scale VLM) retrieves the current object statuses from the current observation and verifies if they match the condition of a specific stage in the task knowledge (Challenge $\#1$). Next, from a list of required motions to complete the stage, a motion progress evaluator (a whole-scale VLM) determines the current motion and assesses the in-motion progress (Challenges $\#2$ and $\#3$) by evaluating the correlations between visual observations and each motion instruction. As a result, VICtoR can generate informative rewards to complete long-horizon tasks.

We conducted extensive experiments in both simulated and real-world settings. In the simulation, we designed ten tasks of varying horizons, including those where each step’s success depends on the success of the previous stage. For real-world experiments, we used the dataset collected from XSkill[]. Compared to the best prior VIC reward model, VICtoR achieved a 25% improvement in the average success rate across all tasks and a 43% improvement in the more challenging tasks (top half of tasks considering motion’s number). Additionally, our detailed ablation studies emphasized the importance of our hierarchical architecture and training objectives. Visualizations of the learned embedding space and rewards demonstrated that VICtoR effectively assesses task progress.

We summarize our contributions as follows: (1) We are the first to explore the potential of VIC rewards for long-horizon manipulation tasks. (2) We introduce VICtoR, a novel hierarchical VIC reward model that assesses task progress by decomposing it into various levels. (3) We present extensive results from simulated and real-world experiments, ablations, and visualizations to demonstrate VICtoR’s superiority, outperforming the best prior method by 43% on more challenging tasks.

## 2 Related Work

#### Large Pre-trained Models for Robotics.

With great ability of reasoning, Large Language Models (LLMs) have been utilized for robotics in navigation [], task planning [, ], code-completion for policies [, ], and manipulation [, , , , ]. These works leverage LLMs’ ability to translate high-level instructions into actionable sequences and break them down into sub-orders for precise agent control. Additionally, recent advancements have leveraged Vision-Language Models (VLMs) to enhance environmental understanding for agents. With a profound understanding of the physical world, VLMs facilitate the works in navigation, robotic control, and monitoring [, , , , ]. Unlike previous methods, for reward modeling, VICtoR uses LLMs to break down tasks and VLMs to assess visual observation with motion descriptions, allowing our method to better assess the task’s progress and distinguish it with different granularity.

#### Addressing Long-Horizon Tasks.

Long-horizon manipulation is a long-standing problem. Traditional Hierarchical Reinforcement Learning (HRL) methods [, , , , ] address this challenge by decomposing complex tasks into hierarchical levels or sequential orders, where higher-level policies make decisions that guide the selection of parameters for lower-level policies or pre-trained actions. Task and Motion Planning (TAMP) methods [, , ] attempt to assemble and schedule well-trained motions to accomplish long-horizon tasks. Additionally, Long-horizon Imitation Learning [] has been proposed to learn long-horizon tasks from expert demonstrations. Unlike these approaches, VICtoR aims to learn the reward model for long-horizon tasks from action-free demonstrations and task instructions. Our reward model is designed to be independent of policy design and can be applied to RL algorithms to learn long-horizon manipulation tasks.

## 3 Preliminaries

#### Problem Statements.

For a reinforcement learning task $\mu$, we assume the environment follows a partially observable Markov Decision Process (POMDP), which can be described with the tuple $\mathcal{M}_{\mu}\coloneqq(\mathcal{S},\mathcal{A},\mathcal{O},\Omega,\mathcal
{P},\mathcal{R},\gamma)$, in which the observation $o\in\Omega$ is derived from the observation function $\mathcal{O}(o\mid s)$ conditioned on the current state $s\in\mathcal{S}$. $\mathcal{A}$ is the space of actions; $\mathcal{P}(s^{\prime}\mid s,a)$ denotes the dynamics of the environment; $\mathcal{R}$ and $\gamma$ are the environment’s reward function and discount factor, respectively. The objective aims to seek a policy $\pi(a\mid o)$ that assigns a probability to action $a$, given the observation $o$, which maximizes the accumulated discounted rewards in a rollout.

#### VIC Reward Model.

This work aims to develop a Vision-Instruction Correlation (VIC) reward model to generate accurate rewards from visual observations and language instructions. We assume an instruction $L^{\mu}$ for the task $\mu$ is accessible, as the instruction is a low-cost resource to collect. Specifically, we seek a reward model $\mathcal{R}(o_{t},o_{t-1},L^{\mu})$ computes the reward by evaluating the difference in correlation between the image observation $o$ and the instruction $L^{\mu}$. Intuitively, $\mathcal{R}(o_{t},o_{t-1},L^{\mu})$ should generate a higher reward if the $o_{t}$ is moving closer to the task’s goal compared to $o_{t-1}$.

#### Stages and Motions.

In the context of long-horizon manipulation, we define motion as as the primitive movement that a robot can make, such as reaching or grasping an object. In addition, stage refers to a high-level interaction with an object in the environment, requiring the consideration of complex objects’ status, and can encompass multiple motions. For instance, the stage "open the drawer" involves the motions "reach the drawer handle" and "pull out the drawer." A comprehensive list of motions and stages is detailed in Appendix B.1.

## 4 Method

#### Overview.

VICtoR is a reward model for long-horizon manipulation, relying on visual observations and language instructions. As depicted in Figure 2, VICtoR employs a hierarchical approach to assess task progress at various levels, including stage, motion, and motion progress. It consists of three main components: (1) a Task Knowledge Generator that decomposes the task into stages and identifies the necessary object states and motions for each stage (Section 4.1); (2) a Stage Detector that detects object states to determine the current stage based on the generated knowledge (Section 4.2); (3) a Motion Progress Evaluator that assesses motion completion within stages (Section 4.3). With this information, VICtoR then transforms it into rewards (Section 4.4). Both the Stage Detector and Motion Progress Evaluator are trained on motion-level videos labeled with object states, which are autonomously annotated during video collection. This setup enables VICtoR to deliver precise reward signals for complex, unseen long-horizon tasks composed of these motions in any sequence.

Figure: Figure 2: Training and inference pipeline of VICtoR. VICtoR is trained using motion-level videos with language annotations and object state labels. It first decomposes the task into task knowledge for decomposed stages, conditional object states, and motions. Then, it uses Stage Detector to identify the stage, and a Motion Progress Evaluator (VLM) to detect the motion and in-motion progress
Refer to caption: x2.png

### 4.1 Task Knowledge Generation

To make long-horizon tasks more achievable and manageable, we leverage the reasoning ability in Large Language Models (LLMs) to decompose and analyze the tasks. Previous research [, ] introduced LLMs into policy learning by planning sub-goals or actions for explicit guidance to enhance long-horizon capabilities. Building upon this foundation, we explore more nuanced usage. Our goal is to decompose a given training task into distinct sub-stages, define the specific conditions or states of each object at these stages, and identify the motions required by the agent to advance to the next stage. To achieve this, we employ GPT-4 to analyze and construct task knowledge. Figure 2 illustrates an example of task knowledge generation for the task “move the block into the closed drawer.” Details of our prompt design and implementation are provided in Appendix F, and additional experiments for robustness are described in Appendix A.2.

### 4.2 Stage Detection

Traditional VIC models, which encode entire visual observations to assess task progress, often overlook subtle yet critical changes in object states. To address this, we differentiate between detecting local object state changes and global agent movements (discussed in the next section). In the previous section, Task Knowledge Generator decomposed long-horizon tasks into sub-stages, each conditioned on different object states. To detect these stages, we designed Stage Detector, which detects stage by identifying each object’s state in the environment and comparing these with the conditioned states of each stage as defined in the task knowledge.

#### Object State Detection.

To detect object states, our Stage Detector first leverages a language-conditioned object detector $D$, such as MDETR [], to identify objects in the image observation $o$. Essentially, we crop out the bounding box $D(o,c)$ generated from the detector $D$ based on the current observation $o$ and the queried object $c$. Subsequently, these cropped images are fed into our object state classifier $P$. The classifier predicts the likelihood of categorical object state for each cropped image as $P(L^{c}|D(o,c))$, where $L^{c}$ represents the language description of possible object states for object $c$. We summarize the training details in Appendix B.2.

#### Stage Detection.

After extracting all the object states in the environment, our Stage Detector can then determine the current stage by comparing these states with the task knowledge. A stage is identified only if, and only if, all object states align with the task knowledge for a specific stage.
For example, as illustrated in Figure 2, if the drawer is currently closed, we compare this condition with the states specified in each stage of the task knowledge. Upon finding that the drawer is expected to be closed only at the first stage, we conclude that the agent is at the first stage. This comparison should be applied to every conditioned object for the tasks. If the extracted set of object states does not match any stage, the agent will be considered to be in the initial stage.

### 4.3 Motion Progress Evaluator

Since a stage may require multiple motions to complete, relying merely on the stage reward is insufficient for learning complex tasks. To this end, we introduce the Motion Progress Evaluator (MPE) to provide the reward signals within the stage. As discussed in Section 4.1, each detected stage comes with a list of motion descriptions in the task knowledge. Building on this, the MPE assesses the agent’s current motion and progress within the motion, taking into account these descriptions. This MPE model yields a more detailed reward model that accurately reflects progress within stages.

#### Architecture.

Our MPE model employs the vision encoder $EV_{\theta}$ and language encoder $EL_{\theta}$ from CLIP[], with two additional heads (fully-connected layers) attached to $EV_{\theta}$. Initially, the vision encoder $EV_{\theta}$ generates the base embedding $f$, which is then transformed into motion-specific embedding $m$ and progress-specific embedding $p$ through separate heads. As for the language encoder $EL_{\theta}$, we employ it to generate the language embedding $l^{mo}$ of motion $mo$ without updating its weights during training. Our goal is to determine the motion class and motion progress by evaluating the similarity between the embeddings $m$ and $p$ and the language description $l^{mo}$, respectively.

#### Objectives.

To effectively evaluate progress within a stage, we expect three key capabilities in the MPE model: (1) the ability to discern the temporal difference of frames extracted from the same video, (2) the capacity to identify the motion in which the agent is engaged, and (3) the capability to assess the progress of motion completion. To this end, we introduce objectives based on InfoNCE [] to guide our MPE model, focusing on time contrast, motion contrast, and language-frame contrast.

For implementation, we utilize the negative L2 distance as the similarity function $S$. During each training iteration, we sample $B$ videos focusing on specific motions and $N$ videos featuring arbitrary agent movements, from which we randomly select a batch of frame sequences $[F_{i},F_{j>i},F_{k>j}]$.

#### Time Contrastive Loss.

To capture features relevant to physical interaction and sequential decision-making, we apply the time contrastive loss $\mathcal{L}_{tcn}$ to the base embeddings, as inspired by [, ]. Essentially, images that are temporally closer should have more similar representations (embeddings) than those that are temporally distant or from different videos. The loss $\mathcal{L}_{tcn}$ can be formulated by

$$ $\mathcal{L}_{tcn}=-\sum_{b\in B}\sum_{\begin{subarray}{c}(x,y)\in\\ \{(i,j),(j,k)\}\end{subarray}}\log\frac{e^{S(f^{b}_{x},f^{b}_{y})}}{e^{S(f^{b} _{x},f^{b}_{y})}+e^{S(f^{b}_{i},f^{b}_{k})}}\,,$ (1) $$

where $f^{b}_{x}$ is the base embedding of the $x$’th frame.

#### Motion Contrastive Loss.

To differentiate between various motions, we introduce the motion contrastive loss $\mathcal{L}_{mcn}$, which aligns each motion’s embedding with its relevant language embedding and separates it from unrelated language embeddings. Additionally, to reduce reward hacking (^1^11During RL training, agents may explore areas not covered in demonstrations and exploit reward model vulnerabilities, leading to high rewards for incorrect actions.) and enhance the robustness of our MPE model, we task it with distinguishing frames from motion videos or meaningless videos that contain arbitrary movements. The loss $\mathcal{L}_{mcn}$ is formulated by

$$ $\mathcal{L}_{mcn}=-\sum_{b\in\{B,N\}}\sum_{\begin{subarray}{c}x\in\\ \{i,j,k\}\end{subarray}}\log\frac{e^{S(m^{b}_{x},l^{b})}}{e^{S(m^{b}_{x},l^{b} )}+e^{S(m^{b}_{x},l^{\neq b})}}\,,$ (2) $$

where $m^{b}_{x}$ is the motion embedding of $x^{th}$ frame, $l^{b}$ is the language annotation of motion in the video $b$, and $l^{\neq b}$ is the instruction for other motion.

#### Language-Frame Contrastive Loss.

To assess progress within the motion, we employ the language-frame contrastive loss $\mathcal{L}_{lfcn}$. The loss $\mathcal{L}_{lfcn}$ aims to bring the progress embeddings of nearly completed steps closer to the instruction embedding of the motion while distancing the progress embeddings of frames from earlier steps, which can be stated by

$$ $\mathcal{L}_{lfcn}=-\sum_{b\in B}\log\frac{e^{S(p^{b}_{k},l^{b})}}{e^{S(p^{b}_ {i},l^{b})}+e^{S(p^{b}_{j},l^{b})}+e^{S(p^{b}_{k},l^{b})}}\,,$ (3) $$

where $p^{b}_{x}$ is the progress embedding of $x^{th}$ frame, and $l^{b}$ is the motion instruction for the video $b$.

#### Total Loss.

Finally, we employ the Adam optimizer and the total loss $L_{total}$ to guide our MPE model, where $L_{total}$ is a weighted combination of the three objectives mentioned above:

$$ $\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{tcn}+\lambda_{2}\mathcal{L}_{mcn}+ \lambda_{3}\mathcal{L}_{lfcn}.$ (4) $$

#### Inference.

During inference, along with the stage detected by the Stage Detector, the MPE model evaluates the current motion from the motion candidates $L^{M}$ within the stage, provided by the task knowledge from Section 4.1. Specifically, it calculates the similarity scores between the motion embedding $m$ of current observation and language embeddings $l^{mo}$ of each motion candidate $mo\in L^{M}$. It then selects the motion with the highest score, as formulated by

$$ $\mathrm{motion*}=\underset{mo\in L^{M}}{\mathrm{argmax}}\,S(m,l^{mo}).$ (5) $$

To determine the confidence level of the MPE model, we further calculate the confidence score by comparing the similarity difference between the motion embedding $m$ and the language embedding $l^{\mathrm{motion*}}$ of the selected motion, and the language embedding $l^{n}$ of arbitrary agent movements. We decide whether to choose the selected motion $\mathrm{motion}*$ or the first motion by

$$ $\displaystyle\mathrm{motion}=\begin{cases}\mathrm{motion*},&\mathrm{if}\> \mathrm{confidence}>\lambda_{c}\\ 0,&\mathrm{otherwise}\end{cases}\mathrm{,~{}where}$ (6) $$

$$ $\mathrm{confidence}=\frac{S(m,l^{\mathrm{motion*}})}{S(m,l^{\mathrm{motion*}}) +S(m,l^{n})}.$ (7) $$

Finally, given the determined $\mathrm{motion}$, we assess the progress within the motion by calculating the similarity score between the progress embedding $p$ and the motion’s language embedding $l^{\mathrm{motion}}$:

$$ $\mathrm{progress}=\begin{cases}S(p,l^{\mathrm{motion}}),&\mathrm{if}\>\mathrm{ confidence}>\lambda_{c}\\ 0,&\mathrm{otherwise}\end{cases},$ (8) $$

### 4.4 Reward Formulation & Policy Learning

Considering all the information acquired from previous sections, we now have knowledge regarding the current stage of the agent, its ongoing motion within the stage, and the progress within that motion. We aggregate this information and the task knowledge into a measure of the task’s potential, representing the overall progress within the task:

$$ $\phi(o_{t})=\underbrace{\lambda_{m}\left(\sum_{i=0}^{\mathrm{stage}}\#\mathrm{ motions}_{i}+\mathrm{motion}\right)}_{\text{Total number of preceding motions} }+\underbrace{\mathrm{progress}}_{\text{in-motion progress}}$ (9) $$

where $\phi(o_{t})$ denotes a potential function conditioned on the image observation at timestep $t$, $\mathrm{stage}$ is the number of the detected stage in the task. $\#\mathrm{motions}_{i}$ is an integer indicating the total number of different motions within stage $i$, $\mathrm{motion}$ is the integer representing the number of the detected motion in the stage, and $\lambda_{m}$ is a constant near the maximum of $\mathrm{progress}$, ensuring $\phi(o_{t})$ increases across motions and stages. Additionally, $\mathrm{progress}$ is a float determined by the MPE model. Note that the number of $\mathrm{stage}$, $\#\mathrm{motions}_{i}$, $\mathrm{motion}$ correspond to the sequence provided in the task knowledge. Using the potential, we implemented potential-based shaping rewards $R(o_{t},o_{t-1})=\phi(o_{t})-\phi(o_{t-1})$ from [] to guide the agent’s behavior based on potential changes.

## 5 Experiments

Our experiments aim to answer the following questions: (1) Does VICtoR provide effective rewards for long-horizon tasks? (2) Is VICtoR able to generate dense rewards from real-world videos? (3) Are all reward signals for stage, motion, and progress indispensable for learning effective rewards in long-horizon manipulation tasks, and do they capture accurate information?

### 5.1 Experiment Settings

Figure: Figure 3: Environment information. Tasks are generated from permutations of actions on interactable objects shown in the figure.
Refer to caption: x3.png

#### Evaluation Settings.

Most existing manipulation benchmarks [, , ] do not support long-horizon tasks where the success of each step depends on the outcomes of previous steps. While Calvin [] addresses the aforementioned issue, the tasks and action spaces (e.g., rotate) are too challenging for VIC reward models trained on action-free videos. Consequently, we developed a new environment to test different VIC reward models for long-horizon manipulation. For policy training in this environment, we use a ground-truth object detector to speed up the process, as we need to run multiple tasks with multiple seeds. For real-world experiments, we train and test reward models using the XSkill [] dataset. Details on the environment, tasks, and data are in Appendix B.1 and Figure 3.

#### Baselines.

We compare VICtoR with the following baselines: (1) Sparse Reward: A binary reward function assigns a reward only when the task succeeds. (2) Stage Reward: A reward function that assigns a reward equal to the stage number when the agent reaches a new stage. (3) LOReL []: A language-conditioned reward model that learns a classifier $f_{\theta}(o_{0},o_{t},l)$ to evaluate whether the progression between the $o_{0}$ and $o_{t}$ aligns with the task instruction $l$. Note that, we replaced the backbone of LOReL to CLIP for a fair comparison with VICtoR and LIV. (4) LIV []: A vision-language representation for robotics that can be utilized as a reward model by finetuning on target-domain data. We apply the same reward shaping method for LIV and LOReL as discussed in Section 4.4. (5) VICtoR (task): A baseline for VICtoR trains on task-level data, using the same MPE objective but with demonstrations of tasks not divided into stages or motions. Note that the training demos for VICtoR are action-free videos with text instructions; therefore, we do not compare our method with other works requiring action data, such as language-conditioned imitation learning.

### 5.2 Policy Learning with VIC Rewards

In this section, we train the Proximal Policy Optimization algorithm [] using various reward functions. We utilize a sparse reward as a signal for task success and shape the reward function using different reward models. Our experiments are divided into two sets: initially, we train on 3 single-stage tasks with varying numbers of motions, and subsequently, on 7 tasks with varying numbers of stages. As shown in Tables 1 and 2, VICtoR consistently outperforms current baselines, achieving a 25% average performance gain. This advantage increases to 43% in harder tasks with more than three motions, highlighting VICtoR’s effectiveness in training for long-horizon tasks. The significant benefits of the motion design are also evident when compared to the task-level baseline, especially in more complex tasks. Furthermore, as Table 1 demonstrates, even at the task level, VICtoR outperforms other baselines, showcasing the effectiveness of our training objective detailed in Section 4.3. Training details and curves are provided in Appendix B.5, and the qualitative analyses for challenges outlined in Figure 1 are also provided in Appendix A.1.

**Table 1: Experiment of tasks with one stage but different numbers of motions. The table presents the success rate [$\uparrow$] of the learned policy using different reward functions across three tasks that have one stage but vary in the number of motions. It shows that the differences between VICtoR and other baselines become larger as the number of motions in the tasks increases. This indicates that when a stage requires more detailed actions to complete, it becomes necessary to decompose the task to assess progress. Results are averaged over three different seeds.**
| #motions | 1 | 2 | 4 |
| --- | --- | --- | --- |
| Task | reach block | pick block | move block |
| Sparse | 99.8% | 0.0% | 0.0% |
| LIV [] | 99.9% | 98.6% | 0.0% |
| LOReL [] | 100.0% | 88.8% | 0.0% |
| VICtoR(task) | 99.9% | 94.2% | 36.8% |
| VICtoR | 99.9% | 98.8% | 58.9% |

**Table 2: Experiment of tasks with different numbers of stages and motions. Extending from 1, this table includes seven additional tasks with varying numbers of stages and motions. It shows that VICtoR outperforms prior methods, and the differences become larger as the complexity of the tasks increases, demonstrating the effectiveness of our design of stage and motion rewards. Results are averaged over three different seeds.**
| #stages | 1 | 1 | 2 | 2 | 2 | 3 | 3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| #motions | 2 | 2 | 3 | 4 | 6 | 4 | 5 |
| Task | open box | open drawer | open drawer +<br>open light | open box +<br>pick block | open drawer +<br>move block | open light +<br>open drawer +<br>reach block | open box +<br>open light +<br>pick block |
| Sparse | 100.0% | 79.5% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Stage | 100.0% | 79.5% | 26.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| LIV [] | 99.9% | 66.7% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| LOReL [] | 100.0% | 100.0% | 33.1% | 35.1% | 0.0% | 35.0% | 28.9% |
| VICtoR | 99.9% | 100.0% | 66.1% | 96.4% | 30.6% | 70.9% | 57.7% |

### 5.3 Ablation Studies

To better understand how each reward signal in VICtoR contributes to policy learning, we conducted an ablation study on the “open box then pick blue block" task to evaluate the effectiveness of VICtoR’s stage determination, motion determination, and progress assessment as introduced in Section 4. Based on the findings presented in Table 3, we observe that injecting reward signals at each level proves beneficial for policy learning. Furthermore, our analysis indicates that reward signals at the “progress" level notably enhance performance. This underscores the importance of nuanced reward signals, particularly in tasks with longer horizons, thereby supporting our claims. Ultimately, the combination of these three levels of reward signals enables the policy to achieve peak performance.

**Table 3: Ablation study. This table illustrates the effectiveness of each level of reward signals in VICtoR for policy learning. Results are averaged over three different seeds.**
| Ablation Settings | Success Rate |  |  |
| --- | --- | --- | --- |
| Stage | Motion | Progress |  |
| - | - | - | 0.0% |
| ✓ | - | - | 22.8% |
| ✓ | ✓ | - | 29.7% |
| ✓ | ✓ | ✓ | 96.4% |

### 5.4 Real World Experiments

To demonstrate VICtoR’s practical efficacy, we visualized the potential curves $\phi(o_{t})$ for two scenarios: one with videos that match the corresponding instructions, and another with incorrect videos using the same instructions. As illustrated in Figure 4, for the correct actions, the potential curves show that VICtoR effectively identifies task progress in real-world scenarios by increasing the potential as the agent completes tasks, a capability not matched by previous VIC reward models. For the incorrect actions, as the agent moves from the right to the left side to close the drawer, VICtoR‘s visualization shows an initial increase in potential. This increase is logical as these movements approach the light, aligning with the first stage of the instruction, “open the light.” However, as the agent continues toward the drawer, VICtoR recognizes the incorrect task and begins to decrease the reward, effectively deterring the agent’s movement. This highlights VICtoR’s ability to analyze agent movement and task progress accurately. Additional visualizations for various tasks are provided in Appendix E.

Figure: Figure 4: VICtoR on real world data. This figure displays LIV [] and VICtoR’s potential visualizations for long-horizon tasks from XSkill with correct and incorrect test videos.
Refer to caption: x4.png

### 5.5 Visualizations

#### Embedding Visualization.

To verify the effectiveness of the contrastive objectives we designed to boost understanding of task progress, we present a t-SNE analysis on motion embeddings $m$ across every motion in our simulated environment. From Figure 5, we observe that the learned embeddings of identical motion are clustered, indicating high discriminative capability. This attribute is crucial for enhancing the accuracy and efficiency of our reward model, particularly in long-horizon tasks.

#### Reward Visualization.

To assess VICtoR’s ability to differentiate motions and assess progress within a motion, we visualize motion determination per timestep and measure the embedding $z$’s distance between motion descriptions and frame embeddings. As shown in Figure 5, VICtoR accurately switches motions at the appropriate timestep and decreases the embedding distance of the determined motion as the agent approaches each motion’s goal.

Figure: Figure 5: Visualization. (a) t-SNE analysis for different motion frames, and (b) analysis of negative embedding distances and determined motions in a demonstration video.
Refer to caption: x5.png

## 6 Conclusion

We have presented VICtoR, a Hierarchical Vision-Instruction Correlation Reward Model for Long-horizon Robotic Manipulation. Using only language instructions, VICtoR is able to provide effective rewards to the reinforcement learning agent. Having been trained solely on modular short-horizon demonstration videos, VICtoR outperforms state-of-the-art approaches trained on targeted long-horizon tasks and successfully operates in both simulated and real-world environments.

#### Future works and limitations

VICtoR can provide rewards for unseen long-horizon tasks composed of seen motions. However, similar to previous works, the limitation of VICtoR is that it cannot be applied to tasks involving unseen motions. Therefore, a future direction for this project will be training with a diverse range of motion data to explore its zero-shot capabilities on unseen motions.

## Appendix

## Appendix A Additional Experiments

### A.1 Qualitative Analysis

To further analyze different reward models, we also examine the potential curves for various reward models to substantiate our statement shown in Figure 1. In the qualitative analysis, we aim to demonstrate our statement regarding (1) confusion arising from varying task difficulties, and (2) the lack of explicit object state estimation.

For statement (1), we plot out the potential curves from 3 different reward models on 4 tasks within variant task complexities. The result in Figure 6 reveals that VICtoR is capable of generating effective reward curves on composed and complex tasks as other baseline models fail to provide progressive signals for agents when the task horizon grows.

To demonstrate statement (2), we visualize the reward by giving text goals for long-horizon tasks (open light then open drawer then reach blue block) on misordered action (reach blue block) videos to see the difference in reward models. The result in Figure 7 indicates that VICtoR is capable of determining the current state of environmental objects to decide the level of potential while LIV simply generates progressive signals on each task.

Figure: Figure 6: Potential comparison across different tasks: We compare the potential generated by different reward models. In these comparisons, we can see that VICtoR provides the most progressive and near-strictly increasing potential function, especially as the horizon increases. This demonstrates its ability to provide fine-grained rewards for long-horizon tasks.
Refer to caption: x6.png

Figure: Figure 7: Potential Visualization on Misordered Action: We visualize the rewards given for long-horizon tasks (open light, then open drawer, then reach blue block) in videos with misordered actions (reach blue block) to compare different reward models. The figure shows that VICtoR can determine the current state of environmental objects to decide the level of potential, while LIV simply generates progressive signals for each task without penalizing incorrect actions, which may lead to poor learning outcomes.
Refer to caption: x9.png

### A.2 Accuracy of Task Knowledge Generation

To validate the robustness of LLMs for task knowledge generation, we conducted an additional experiment testing the accuracy of LLM outputs. We tested our model by randomly selecting tasks, which could involve 1 to 4 stages. We then used GPT-4 to translate these tasks into human-like queries. With these queries, we tested the task knowledge generation model by comparing its output against the ground truth. After randomly sampling 50 different tasks with varying free-form instructions, we observed that GPT-4 achieved an accuracy of 96% in decomposing tasks into stages, motions, and conditional object states, confirming its robustness for this application.

## Appendix B Detailed Settings for the Experiment

### B.1 Details about our simulated environment and tasks

#### Details about the environment

Our environment was developed using Coppeliasim [], with Pyrep [] serving as the coding interface, as shown in Figure 3. In this environment, there are four interactable objects: a light, a drawer, a blue block, and a box, each associated with different stages as detailed in B.1. The tested tasks can be any permutation of these stages.

#### List of Stages and Motions.

We treated the motions as the action primitives for agents to implement. Definitions of the motions in each stage in our experiments are detailed as shown in Table 4. The definition of motions of stages could be initiated by either humans or GPT.

**Table 4: Description of Separated Motions for Each Stage**
| Stage | Motion 1 | Motion 2 | Motion 3 | Motion 4 |
| --- | --- | --- | --- | --- |
| reach blue block | reach the blue block | - | - | - |
| pick blue block | reach the blue block | lift the grasped blue block | - | - |
| move blue block to drawer | reach the blue block | lift the grasped blue block | move the blue block to the top of the drawer | place the blue block down to the drawer |
| open drawer | reach the closed drawer handle top | pull the drawer out | - | - |
| close drawer | reach the opened drawer handle top | push the drawer forward | - | - |
| open box | reach the box holder back | slide the box holder forward | - | - |
| close box | reach the box holder front | slide the box holder backward | - | - |
| open light | reach and push down the button | - | - | - |
| close light | reach and push down the button | - | - | - |

#### Training Tasks Details.

The detailed success conditions, training timesteps, and the running machines for each task are shown in Table 5.

**Table 5: Training Details for each task**
|  | Reach blue block |
| --- | --- |
| Task Goal | Reach the blue block |
| Success Condition | The distance between the end-effector and the blue block |
| Max Training Step | 30000 |
| GPU | NVIDIA RTX 4090 |
|  | Pick blue block |
| Task Goal | Grasp the blue block and lift it up |
| Success Condition | The blue block is lifted higher than 0.62 |
| Max Training Step | 100000 |
| GPU | NVIDIA RTX 4090 |
|  | Move blue block to drawer |
| Task Goal | Pick up the blue block and place it into the drawer |
| Success Condition | The position of the blue block (if it is in drawer or not) |
| Max Training Step | 300000 |
| GPU | NVIDIA RTX 4090 |
|  | Open drawer |
| Task Goal | Pull out the drawer |
| Success Condition | The position of the drawer |
| Max Training Step | 50000 |
| GPU | NVIDIA RTX 4090 |
|  | Open box |
| Task Goal | Push the lid of the box to open the box |
| Success Condition | The position of the box lid |
| Max Training Step | 50000 |
| GPU | NVIDIA RTX 4090 |
|  | Open drawer then open light |
| Task Goal | Pull out the drawer and then press the button to open the light |
| Success Condition | The position of the drawer and the color of the light |
| Max Training Step | 200000 |
| GPU | NVIDIA RTX 3090 |
|  | Open box then pick blue block |
| Task Goal | Push the lid of the box to open it and pick up the blue block |
| Success Condition | The position of the box lid and the height of the blue block |
| Max Training Step | 200000 |
| GPU | NVIDIA RTX 3090 |
|  | Open drawer then move blue block to drawer |
| Task Goal | Pull out the drawer and then pick up the blue block and place it into the drawer |
| Success Condition | The position of the blue block (if it is in drawer or not) |
| Max Training Step | 400000 |
| GPU | NVIDIA RTX 3090 |
|  | Open light then open drawer then reach blue block |
| Task Goal | Press the button to open the light, then pull out the drawer, and reach the blue block |
| Success Condition | The color of the light, the position of the drawer, and the distance |
|  | between the end-effector and the blue block |
| Max Training Step | 400000 |
| GPU | NVIDIA RTX 3090 |
|  | Open box then open light then pick the blue block |
| Task Goal | Push the lid of the box to open it and press the button |
|  | to turn on the light then pick up the blue block |
| Success Condition | The position of the box lid, the color of the light, and the height of the blue block |
| Max Training Step | 400000 |
| GPU | NVIDIA RTX 3090 |
|  | Open box then open light then open drawer then then reach the blue block |
| Task Goal | Push the lid of the box to open it and press the button |
|  | to turn on the light and open the drawer then reach the blue block |
| Success Condition | The position of the box lid, the color of the light, the position of the drawer, |
|  | and the distance between the end-effector and the blue block |
| Max Training Step | 500000 |
| GPU | NVIDIA RTX 3090 |

#### Training Data Details.

To train the reward models for VICtoR and other baselines, we collected a dataset in our environment. Note that VICtoR requires only motion-level data for training, while other baselines need full-length videos. Therefore, the data used to train VICtoR and other baselines differ in type but are comparable in total amount per motion or stage. For VICtoR, we provide only the motion-level demos, with each motion accompanied by 100 demos and corresponding motion descriptions. For training other baseline models, such as LIV [] and LOReL [], we provide the full-length demo videos, each paired with task descriptions and consisting of 100 demos per task.

### B.2 Details of Stage Detector and Training Details of Motion Progress Evaluator

We provide details of the Stage Detector and Training Details of the Motion Progress Evaluator in Table 6 and Table 7. For the hyperparameter selection of $\lambda_{1,2,3}$, we tried various combinations but found that the model is not sensitive to these hyperparameters. Therefore, we have chosen to set them all to the same values.

**Table 6: MDETR configuration in VICtoR**
| Object Detector - MDETR [] |  |
| --- | --- |
| Backbone | Resnet101 |

**Table 7: Details of VICtoR Motion Reward Model and Stage Determinator Training**
| Motion Progress Evaluator |  |
| --- | --- |
| Language Encoder | CLIPTextModel |
| Image Encoder | CLIPVisionModel |
| CLIP Model | openai/clip-vit-base-patch32 |
| Learning Rate | 0.000001 |
| Batch Size | 32 |
| Training Steps | 20000 |
| Optimizer | Adam [] |
| Loss Function | Equation 4 |
| Parameters in Equation 4 | $\lambda_{1}=1$, $\lambda_{2}=1$, $\lambda_{3}=1$ |
| Stage Determinator |  |
| Language Encoder | CLIPTextModel |
| Image Encoder | CLIPVisionModel |
| CLIP Model | openai/clip-vit-base-patch32 |
| Learning Rate | 0.00001 |
| Batch Size | 32 |
| Training Steps | 40000 |
| Oprimizer | Adam |
| Loss Function | Binary Cross Entropy Loss |

### B.3 Details of MPE’s Inference and Reward Formulation

We organize our reward function parameters of Section 4.3 in Table 8, For the selection of $\lambda_{m}$, we tried to find the biggest range of the progress (embedding distance from language to motion), in order to make the proceeding motion’s potential higher than the previous one. For the selection of $\lambda_{c}$, we have tested it with different numbers and found out that when the $\lambda_{c}$ is in the range of 0.1 to 0.4, the model has similar results, and if it is less than 0.1, it cannot avoid reward hacking. Once it is larger than 0.4, the model cannot provide enough signal for the agent to learn.

**Table 8: Reward formulation and MPE model parameters of VICtoR**
| $\lambda_{c}$ | $\lambda_{m}$ |
| --- | --- |
| 0.2 | 36 |

### B.4 Training Details of Baseline Models

We provide details of other baselines in Table 9.

**Table 9: Details of Baseline Reward Models Training**
| LIV |  |
| --- | --- |
| CLIP Backbone | Resnet50 |
| Learning Config | pretrain |
| Batch Size | 32 |
| Train Steps | 70000 |
| Learning Rate | 0.00001 |
| Weight Decay | 0.001 |
| LOReL |  |
| Language Encoder | CLIPTextModel |
| Image Encoder | CLIPVisionModel |
| CLIP Model | openai/clip-vit-base-patch32 |
| Batch Size | 32 |
| Train Steps | 20000 |
| Learning Rate | 0.00001 |
| Flipped Negative | True |
| alpha | 0.25 |

### B.5 Training Details of Policy Training

#### Training details.

We trained the policy under the SubprocVecEnv from stable-baseline3 [] with reward normalization by VecNormalize. For online interactable environment, we built a simulation environment with Coppeliasim [] and Pyrep [] and wrapped the environment for policy learning under the framework of OpenAI Gym []. Training time is about 3 hours for 10000 steps. Training Details of training of each task are listed in Table 5.

**Table 10: Details of policy training**
| Policy | PPO [] |
| --- | --- |
| Learning Rate | 0.003 |
| Policy Model | MlpPolicy |
| Discount factor ($\gamma$) | 0.99 |
| Reward Normalization | True |

## Appendix C Learning Curves of Policy Learning

We evaluate the policies trained from different reward functions with the success rate in Figure 8, the shadows show the standard deviation.

Figure: Figure 8: Training Curve Visualization: We visualize the success rate across various training episodes and display the success rate curves for different methods in different tasks. The figure demonstrates that VICtoR effectively guides the RL agent to learn quickly and achieve the highest success rate in the fewest episodes, highlighting the effectiveness of the rewards generated by VICtoR.
Refer to caption: x11.png

## Appendix D Additional Comparisons between VICtoR and Other Solutions to Robotic Policy Learning

#### Imitation Learning.

Prior works have attempted to solve the robot manipulation tasks via imitation learning from either robot demonstrations or human videos. However, in our setting, we do not use action sequences from demonstrations to train VICtoR. Thus, the line of imitation learning works is not considered as our baseline. Our objective is to learn a reward model solely from vision and language inputs, without relying on action or observation vectors. This distinction sets our approach apart from imitation learning and other similar methods, which typically require action information to clone a behavior. Therefore, our method addresses a unique problem space that is not directly comparable to imitation learning.

#### Task and Motion Planning (TAMP).

Prior works investigated through accomplishing long-horizon robot manipulation tasks via arranging trained skills. While TAMP methods require a predefined and well-trained skill set and put more emphasis on planning for the execution sequence of these trained skills, our model aims to provide reward signals directly toward policy learning to guide the agent to learn a long-horizon policy from scratch with reinforcement algorithms. Therefore, we do not consider TAMP methods as baselines for VICtoR.

## Appendix E More Potential Visualization for Composed Tasks in XSkill

The visualization results are shown in Figure 9 and Figure 10.

Figure: Figure 9: Potential Visualization on XSkill: Evaluation of the potential curve across multiple test videos on different tasks for XSkill.
Refer to caption: x15.png

Figure: Figure 10: Potential Visualization on XSkill: Evaluation of the potential curve across multiple test videos on different tasks for XSkill.
Refer to caption: x19.png

## Appendix F Prompts for Task Knowledge Generation

Figure: Listing 1: Prompt for Task Knowledge Generation

## Broader Impact

Our research is focused on developing a reward model for manipulation tasks, aiming to accelerate the adoption of robotic applications in complex settings. While we haven’t identified any immediate negative impacts or ethical concerns, it’s crucial that we remain vigilant and continuously assess any potential societal implications as our work evolves.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: + I didn't find ablation studies of the three contrastive losses used in training the motion progress evaluator. It would be better to know their contributions to the overall quality of the generated reward.
+ The proposed approach utilizes motion decomposition of subtasks, a concept similar to the ones used in [2]. It would be helpful to add a discussion about it. In particular, I wonder will the proposed approach also works well at a potentially more granular motion level (e.g., a subskill in the Move Chair task in [2]).

[2] Chain-of-Thought Predictive Control
soundness: 2
strengths: + The approach explores how to leverage LLMs for providing task decomposition for long-horizon manipulation tasks by defining stages and motions as the task hierarchy. And it seems that LLMs such as GPT-4 work well for this setup.
+ The design of the modules sounds reasonable, i.e., leverage MDETR for stage and object status detection and leverage CLIP for motion progress evaluator. The training losses are well-motivated. The results are good.
summary: The authors propose a hierarchical reward model that provides reward signals for long-horizon manipulation tasks. First, It proposes using a Task Knowledge Generator to decompose a task into stages and identify the necessary object states and motions for each stage. It then adopts a Stage Detector to detect the stage and object states given the visual observations. Finally, with a Motion Progress Evaluator that assesses motion completion within stages, the framework gives the overall reward for the long-horizon task. While the task knowledge generator leverages LLMs, the other parts are trained solely on primitive motion demonstrations. The paper shows that their proposed framework produces better rewards for long-horizon manipulation tasks, thus facilitating policy learning with methods such as PPO.
weaknesses: - The long-horizon tasks used to evaluate the proposed framework are a bit limited, e.g., even compared to a popular benchmark [1] proposed in 2020. A more diverse set of tasks would help to fully verify the usefulness and adaptability of this approach. 

[1] ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Line 160, how are the objects states autonomously annotated during real-world experiments?
1. Line 205-208, What if the object detector fails? Will using cropped images for object states classification have certain problems? For example, some object states are positional relationships that need a larger visual field to accurately determine them. What is the model structure for the classifier P?
1. Line 471, why the curves are called ``potential curves''? Are the rewards calculated with collected videos? Do you use the rewards for action rollouts?
1. How do you deal with the long inference time of LLM, when you are training the policy with the proposed design?
soundness: 3
strengths: 1. The method is intuitive. Decomposing a long-horizon task with LLM and utilizing VLM for the short-term progress evaluator are both reasonable and have been validated in previous works.
1. The visualizations are intuitive. I can easily get the idea of most figures and they help to understand the paper. The reward curves and t-SNE graph prove the effectiveness of the proposed reward design and contrastive loss designs.
1. The results shown in the paper are good, compared to other baseline methods.
summary: This paper proposes a method called VICtoR that utilizes LLM (GPT-4) and VLM (CLIP) to calculate the task rewards for long-horizon robotic manipulation. Specifically, it decomposes the language instructions to different stages with the help of GPT-4, and uses MDETR to detect objects and classify their states. Comparing the states with the desired states from GPT-4, the algorithm can determine if the stage has been completed. For the motions inside each stage, VICtoR trains the CLIP encoders with multiple contrastive losses to learn the task progress corresponding to the instruction. Combined with the above reward, a model is trained with self-constructed benchmarks and shows better performance compared to multiple baselines.
weaknesses: 1. The technical novelty is relatively limited. The proposed method mainly has two parts, LLM for decomposing the long-horizon task and CLIP for the short-term progress evaluator. These two parts have both been extensively explored. There have been multiple related works mentioned in the paper. Even in Line 186, the authors acknowledged that the differences are relatively nuanced.
1. The experiments are conducted on a self-collected dataset, not the widely adopted CALVIN benchmark. As mentioned in Line 343-346, the proposed method has critical limitations when dealing with some meticulous actions like rotations.
1. Some technical details are confusing and need further clarification. Please see the questions part below.
1. Minors. Typo in Line 143 (as as)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness.

My questions are:
1. Could this method generalize to novel tasks?
2. Could other methods achieve comparable results when fine-tuned on the target task?
soundness: 3
strengths: 1. This paper proposes a hierarchical reward estimation method to provide vision-based reward for long-horizon manipulation tasks.
2. This paper describes this method clearly.
3. The simulation and real-world experiments prove the effectiveness of this method.
summary: This paper introduces a hierarchical vision-instruction correlation method to estimate rewards for long-horizon robotic manipulation tasks. Specifically, the approach initially identifies task-aware objects and employs GPT-4V to associate the object status with the corresponding motion stage. Additionally, this paper proposes a contrastive learning-based method that evaluates the reward of visual observations by calculating the similarity between visual observations and language instructions. Experimental results validate the efficacy of this method.
weaknesses: 1. The method requires lots of relevant data for estimating visual object information, which makes it struggle to generalize to new task scenarios due to reliance on the object status classifier. Could this method generalize to any novel task? 

2. The comparison results are unfair. This method utilizes the robotic manipulation videos of the same tasks to train the contrastive learning method, while the compared methods such as LIV are pretrained encoders.  Would these vision-language correlation methods achieve comparable results when the encoder is fine-tuned on the target tasks?

3. The conservative learning method for vision-language reward estimation is common. The stage and motion detection method seems to simply achieve state-based rewards through visual models by overfitting on specific manipulation tasks.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Refer to the Weaknesses section.
soundness: 3
strengths: - The paper effectively advances research by integrating LLMs for high-level hierarchical planning, which allows the model to assess the robot’s progress at each step.
- It incorporates an innovative approach to extract relevant information from short video clips, enabling the system to understand key aspects of the task without viewing the entire sequence.
- The methodology section is well-structured, clearly outlining the three core components of the proposed approach, aiding reader comprehension.
- Results in the experiments section demonstrate substantial improvements over previous methods in complex task settings, showcasing the effectiveness of the hierarchical framework.
- Reward visualizations effectively illustrate the approach’s ability to accurately assess performance.
summary: This paper proposes VICtoR, a hierarchical reward model for long-horizon manipulation tasks that learns from action-free videos and language instructions. The proposed approach addresses challenges in task complexity, sub-stage awareness, and object state estimation through a stage detector and motion progress evaluator. VICtoR outperforms existing models by using a novel stage detector and motion progress evaluator, achieving a 43% improvement in success rates for complex tasks.
weaknesses: - A more thorough evaluation of each individual component would clarify its contribution to the overall system’s performance.
- Additional evaluations on zero-shot capabilities, specifically with unseen trajectories, would be valuable in understanding the approach’s generalization.
- The paper lacks an assessment of the computational cost and resource requirements of the proposed method.
- The reliance on GPT-4 could pose potential challenges, which are not thoroughly addressed; analyzing GPT-4 outputs in greater detail could reveal possible limitations.
- A comparative discussion of other methods for long-horizon manipulation, such as hierarchical reinforcement learning (HRL) and task and motion planning (TAMP), is missing and would have enriched the analysis.
- While the authors mention using data from XSkill3 for real-world experiments, they do not provide sufficient details on the dataset’s size, diversity, or the types of tasks involved.

Meta-review: The paper is rated positively by all reviewers (6,6,6,6). The reviewers initially raised several concerns such as limited set of tasks compared to other benchmarks (Reviewer D5gq), no results on widely adopted benchmarks (Reviewer sb4X), unfair comparisons (Reviewer R2Xa), and lack of thorough evaluation of individual components (Reviewer Hvoi). The rebuttal addressed the concerns (details below). The AC follows the recommendation of the reviewers and recommends acceptance. However, the setup is simplistic in general. Generalization to new scenes, objects, and more complex tasks would make the paper better.

Decision: Accept (Poster)