VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
Authors: Guangyan Chen
Meiling Wang
Te Cui
Yao Mu
Haoyang Lu
Tianxing Zhou
Zicai Peng
Mengxiao Hu
Haizhou Li
Li Yuan
Yi Yang
Yufeng Yue
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal language models
Vision language models
Robotic manipulation
Code generation
Visual imitation learning
Primary area: robotics
TL;DR: VLMimic is a novel visual imitation learning paradigm that leverages VLMs to directly learn skills with fine-grained action levels, from a limited number of human videos, outperforming baselines in both simulated and real-world experiments.

Abstract: Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable performance in vision and language reasoning capabilities for VIL tasks. Despite the progress, current VIL methods naively employ VLMs to learn high-level plans from human videos, relying on pre-defined motion primitives for executing physical interactions, which remains a major bottleneck. In this work, we present VLMimic, a novel paradigm that harnesses VLMs to directly learn even fine-grained action levels, only given a limited number of human videos. Specifically, VLMimic first grounds object-centric movements from human videos, and learns skills using hierarchical constraint representations, facilitating the derivation of skills with fine-grained action levels from limited human videos. These skills are refined and updated through an iterative comparison strategy, enabling efficient adaptation to unseen environments. Our extensive experiments exhibit that our VLMimic, using only 5 human videos, yields significant improvements of over 27% and 21% in RLBench and real-world manipulation tasks, and surpasses baselines by more than 37% in long-horizon tasks. Code and videos are available on our anonymous homepage.

VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
Guangyan Chen'
Meiling Wang'
Te Cui
Yao Mu?
Haoyang Lu'
Tianxing Zhou
Zicai Peng
Mengxiao Hu
Haizhou Li
Li Yuan
Yi Yang!
Yufeng Yue
1 Beijing Institute of Technology
The University of Hong Kong
Peking University
Abstract
Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable performance in vision and language reasoning capabilities for VIL tasks. Despite the progress, current VIL methods naively employ VLMs to learn high-level plans from human videos, relying pre-defined motion primitives for executing physical interactions, which remains major bottleneck. In this work, present VLMimic, novel paradigm that harnesses VLMs to directly learn even fine- ime-grained action levels, only given lim- ited number of human videos. Specifically, VLMimic first grounds object-centric movements from human videos, and learns skills using hierarchical constraint representations, facilitating the derivation of skills with fine-grained action levels from limited human videos. These skills are refined and updated through an iter- ative comparison strategy. enabling efficient adaptation Our extensive experiments exhibit that our VLMimic, using only 5 human videos, yields sigmificant improvements of over 27% and 21 % in RLBench and real-world manipulation tasks, and surpasses baselines by over 37% in long-horizon tasks. Code and videos are available aturur home page
unseen environments.
Introduction
Visual Imitation Learning (VIL.) has demonstrated remarkable efficacy in addressing various visual 7
I0] Diverging from conventional approaches reliant precise robot action labels, which often necessitates substantial human effort for data collection. Researchers increasingly turn to learning from human-object interaction videos that are easily accessible to reduce high data requirements.
control tasks within intricate environments [ 4 5
6
8
Existing methods for skill acquisition leveraging video data can be broadly categorized into two classes. One typical approach learns efficient visual representations for robotic manipulation through
self-supervised learning from large volumes of videos [1] [2] 13: 14 15, 16.
18
19
20 ].
Another approach focuses on learning task-relevant priors to guide robot behaviors or derive heuristic reward function for reinforcement learning
25
26
27
28
"Yufeng Yue and Yang are co-correpoodding authors. work was supported by National Natural Science Foundation of China under Grant No. NSFC 62233002, 92370203, (email: yueyufeng edt..edu.cn)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Generalize to unseen scenes
Realwartd mosipulation
KIM-IP
Eodo
Cado
Bresh
Policies
High level plans
Skill Adapter
Ab
Step one Step two
Human-object
Generate low-
Unseen
Interaction Grounding
level actions
RL.Becch
I Leng-borizon tanks
10
challleng
challenge
Generalize
Low-level
Fine-grained motions
Gako
Eado
Ngical
04
Skill Learner
(a) Typical VIL (6)LM as Planner
(e VLM Visual Imiitation Leaner for Fine-grained Actions
(d) Success Rates
Figure I: Illustration of our VLMimic. (a) Typical VIL methods struggle to generalize to unseen environments, and (b) current methods naively utilize VLMs planners, encounter difficulties in generating low-level actions. (c) VLMimic grounds human videos to obtain action movements, and learns skills with fine- mee-graieed actions, while the skill adapter updates skills for generalization. (d) Our method achieves superior performance given a limited collection of human videos.
However, these approaches often encounter challenges when generalizing to unseen environments. Therefore, efficiently acquiring generalizable skills from limited videos remains highly challenging.
An appealing prospect for handling this challenge is employ large pretrained models by encap- sulating extensive prior knowledge from broad data. Recent advances in vision-language models (VLMs) provide particularly promising tools this regard, with their emergent and Fast-growing conceptual understanding, commonsense knowledge, and reasoning abilities. However, current VIL 40 naively employ VLMs to learn high-level plans, and typically rely repertoire of pre-defined motion primitives. This reliance individual skill acquisition is often considered a major bottleneck of the system due to the lack of large-scale robotic data. The question then arises: how can we leverage VLMs to learn even fine-grained action levels directly from human videos, eliminating the reliance on predefined primitives?
methods 30 32 33 35
36
37
38
39
However, adapting VLMs to achieve visual imitation learning for fine- aineigrinee actions is non-trivial due the following critical reasons: (I) Lack of fine-grained action recognition ability. Despite existing advancements in VLMs, they still struggle to recognize low-level actions in videos. To overcome this obstacle, a human-object interaction grounding module proposed, which parses videos into multiple segments, and estimates object-centric actions for subsequent analysis. Such that the intricate low-level action recognition task converted into the pattern reasoning task, which is more tractable for existing VLMs. (II) Difficulty for VLMs in understanding motion signals. Motion signals are characterized by inherent redundancy, hindering models from extracting valuable information. To overcome this challenge, we propose hicrarchical constraint representations for VLM reasoning, which exhibit semantic constraints through visualized actions and illustrate geometric constraints using keypoint values. This representation effectively reduces redundancy and facilitates a comprehensive understanding. enabling our method to learn skills from a limited set of human videos. (III) Disparities in demonstration and target scenes. Demonstration and execution scenes may involve different objects and tasks, impeding direct skill transfer. To this end, we propose skill adapter with iterative comparison strategy, which updates skills by iteratively contrasting with the demonstrated knowledge, facilitating the adaptation of learned skills to unseen scenes.
Based the above analysis, we present VLMimic, approach that employs VLMs directly learn even fine- mnne-gained action levels from a limited number of human videos, and generalize to novel scenes. As shown in Fig. [ our method parses videos into multiple segments and captures object-centric movements using the human-object interaction grounding module. Then, a skill learner employing hierarchical constraint representations extracts knowledge from estimated motions, deriving skills with -re-grained actions. In unseen environments, a skill adapter with an iterative comparison strategy revises and updates the learned skills based observations and task instructions. Extensive experiments demonstrate that VLMimic achieves strong performance across various scenes, utilizing only 5 human videos without requiring additional training.
Our main contributions can be summarized as follows: (I) We propose VIMimic, a novel visual imitation learning framework empowered by VLMs, to learn generalizable robotic skills from

---

human demonstration videos. VLMimic features a skill learner for knowledge extraction and skill adapter for iterative skill refinement, enabling efficient skill acquisition and adaptation. (II) We build an effective human-object interaction grounding algorithm to enhance fine-grained action recognition capabilities, and propose hierarchical constraint representations for VL.M reasoning to reduce information redundancy and facilitate comprehensive action comprehension. (III) Our method outperforms other methods by over 27% on the RLBench. In real-world manipulation tasks, VLMimic achieves an improvement exceeding 21% in seen environments and 34% in unseen environments. Moreover, VLMimic exhibits an improvement of over 37% in long-horizon tasks.
2 Related Work
2.1 Learning from Human videos
Conventional learning approaches necessitate access to expert demonstrations, which include ob- servations and precise actions for each timestep. Drawing on human capabilities, learning from observation offers efficient and intuitive methods for robots to develop new skills. A plethora of recent researches explore leveraging large-scale human video data to improve robot policy learning 14 15 16
18
19 20
41]. Representative methods, R3M 13] and MVP [12), which
employ the internet-scale Ego4D dataset [1] pretrain visual representations for subsequent imita- tion learning tasks. Another thread of work (21 22 23
28 29 focuses on learning task-relevant priors from videos to guide robot behaviors or derive a heuristic reward function for reinforcement learning. Learning by watching (27) learns human-to-roiie
24
25
26
27
translation, the resulting representations are used to guide robots to learn robotic skills. WHIRL [21] infers trajectories and interaction details to establish prior, but learns policy through real-world exploration and requires a large number of rollouts to converge. GraphIRL (24) performs graph abstraction on the videos followed by temporal matching to measure the task progress, and dense reward function is employed to train reinforcement learning algorithms. Despite these advancements, acquiring generalizable skills efficiently from limited demonstration videos remains highly challenging.
2.2 Visual Imitation Learning with VLMs
Motivated by the notable success VLMs across various domains, recent research 32 33
35
36 (3]] investigate their potential in VIL. GPT-4V for Robotics [33] analyzes videos of humans performing tasks and outputs robot programs that incorporate insights into affordances. Digknow [2] distills generalizable knowledge with hierarchical structure, enabling the effective generalization to novel scenes. Demo2code (37 generates robot task code from demonstrations via an extended chain- of-thought and defines common latent specification to connect the two. VLaMP 34 predicts visual planning from videos through video action segmentation and forecasting, handling long video history and complex action dependencies. However, these approaches often rely predefined movement primitives or pre-trained skills to execute lower-level actions, thereby only partially solving the control stack. In contrast, our investigation aims to push these boundaries and learn all lower-level actions for the robot, eliminating the reliance on predefined primitives.
3 VLMimic
Considering video demonstrations V of human performing manipulation tasks, recorded using an RGB-D camera. The overall pipeline of VLMimic is illustrated Fig. 2 Our method first grounds human videos, segmenting them into subtask intervals (T,11= and capturing object-centric interactions I. A skill learner with hierarchical representations then extracts knowledge from the obtained interactions, deriving skills with fine-grained actions. In unseen environments, skill adapter employs an iterative comparison strategy to revise and update the learned skills based on observations and task instructions.
3.1 Human-object Interaction Grounding
Despite VL.Ms demonstrating proficiency in various vision tasks, they still struggle with fine- ine-grained action recognition within videos. To mitigate this limitation, a four-stage process, illustrated in Fig. 3
3

---

(a) Human-object Interaction Grounding
(b) Skill L Learner with Hierarchical Representations
Human videos
Video parsing
Interaction extraction Grasp the brush
Hierarchical representations
Knowledge Bank
Graep it vertiolly the middln, pinch the and. G_ A. B, B;G.
Semantic:
liri Bayy Ma
Keggoirr Obpect peoperties P.
Geometric:
Grasping phase Spread the sauce
R. - Bb" _B Bbox
Task recognition
Semantic:
hptrak...
Make a pie
Spini raclases from the p cunter. It envelops and with the pie.
Pir: plate
Plate:
Brush:: (in] bool
Keygoirr Obpect peoperties P.
Geometric:
Bowl: (on) lubtle
Manipulation pluse
Radius - mintX
New scenes
Indel Bank
Iterative comparison
Greeging place
i,
af
R
Please
M
B2
D1
pan with brush.
Grosp it vertically the middle
[C3004] (62.00]
1,
I
Manipolation phose
Spini envelops the
Reduoe cndres
Brink piraa.1
(c) Skill A dapter with Iterative Comparison
Figure 2: Illustration of our VLMimic. (a) The human-object interaction grounding module parses videos into multiple segments and captures object-centric movements. Then, (b) a skill learner extracts knowledge from action motions and derives skills. novel scenes, (c) skill adapter updates the learned skills to facilitate adaptation.
utilized to extract object-centric interactions for skill learning, transforming this intricate problem into pattern reasoning problems, typically more tractable for existing VLMs.
Task recognition. Keyframes IC are intermittently extracted from videos V. vision foundation models VFM 42
44 are utilized to detect objects within these frames. Utilizing keyframes IC and textual detection results Ta. VLMs are instructed to transcribe videos into task instructions Tú, and compile the task-related objects T, into textual information. The object information is predicated on the initial frame of the video data, comprising a list of object names and their spatial relationships. The task recognition procedure is formulated as follows:
Ta VFM((K)
Tr, T. VLM(
(1)
Video parsing, Videos are parsed into segments (T.)L= using interaction markers that identify interaction periods. SAM-Track (45] 46
47 48 predicts hand and task-related object masks for each frame, and corresponding point clouds P are generated through back back-projection. Markers are then identified by determining the interaction start time and end time te. partitioning videos V into multiple segments. Segments with hand motion trajectory lengths below than are filtered out, yielding final set of segments (T; Y Coneretely, the interaction markers are obtained as follows:
d = dist(P), t, [ -odd-l>e e d' < c}, te [t|d'_1 <e d' > E),
(2)
where function dist calculates the distance between any two point clouds.
Subtask recognition. Each segment т; is analyzed by VLMS, which generate a subtask textual description Ty,, and categorize the segment into grasping or manipulation phases based on the interacting entities and T.,. VLMs also identify master objects Om and slave objects O.. In the grasping phase, the agent performs a wrach-and-grase action targeting Om designating the hand as O.. In the manipulation phase, the agent employs O, interact with O,..
Object-centric interaction extraction. FrankMocap [50] and the Iterative Closest Point (ICP) algorithm [5]] 522 are employed to derive precise hand pose trajectories. which are subsequently
converted into robot gripper pose trajectories. Furthermore, BundleSDF (53] is employed for object reconstruction, and FoundationPoss reconstructed objects O. In grasping phases, interactions are represented as grasp poses hand- object contacts. For manipulation phases, I are defined as trajectories of slave objects O, relative to
54 is leveraged for object pose estimation based on

---

Grounding resolts
Task recognition
Video parsing
(c) Interaction
N Fiddes
Key point values: 4: Object properties, Abox (3304,2)
Keefframes
a Object Information
Trle
Pir (in) plate
A red brush
Detection
plate
Brook: (in) bowl
(conzains)
Graup the brush
Graying
Hond
Put brush pie Manipufation
Brush
Sprend the sauce
Manipulation
9 Brush
8
Key point valtues: " =1jo,0),1- 1.11 Object properties: (528,2533
Bowk (on) table
# Brusb
Pie
Pic
Figure 3: Illustration of Human-object interaction grounding module. (a) It recognizes tasks and related objects from human videos, (b) parses videos into multiple segments based on this information, and subsequently (c) identifies object-centric interactions within each segment.
master objects Om This object-centric paradigm facilitates efficient skill acquisition and enables VLMimic accommodate demonstrations across diverse viewpoints.
3.2 Skill Learner with Hierarchical Representations
A straightforward approach for learning skills involves directly discerning the numerical trajectory patterns (5). However, VLMs face challenges in reasoning about inherently redundant motion signals, limiting their ability to extract valuable information. To reduce redundancy and foster comprehensive comprehension, hierarchical constraint representations are proposed for skill learning. as illustrated in Fig. These representations exhibit semantic constraints via visualized interaction I, and further detail the fine- one-grained geometric constraints by integrating keypoint values Vy.
Learning with hierarchical constraint representations. Rendering interaction I, and textual nota- tions T, objects O derive visualized interaction I.,. VLMimic facilitates reasoning capabilities to analyze semantic constraints 4. by encouraging VLMs to attend to objects and their related actions, and integrating keypoint values V& and object properties P. (e.g.. 3-D bounding boxes) to derive geometric constraints & Formally, constraints are learned as follows:
I, Render( [r, T,I, O), & (1,) 4 G V P.),
where Si, and G are functions to learn semantic, and geometric constraints, respectively.
(I) Grasping constraints. Inspired by task space regions (TSRs) [7], the grasping constraint P, can be approximated as series of bounded regions (R Interactive grasp poses I are exhibited on objects, each associated with an index notation T., These visualized interactions I, are presented to VLMs, leveraging their inherent knowledge and visual understanding ability to summarize semantic constraints & and group these poses. Geometric constraints Lg represented as bounded regions, derived by calculating ranges of grasp pose values V. within the same group, and associating them with object properties P.- This approach simplifies the complex task of constraint region generation into series of visual understanding based multiple-choice question answering, Moreover, representing constraints through object properties enhances generalization across objects.
(II) Manipulation constraints. Interaction trajectory I is delineated on the master object O... in- corporating keypoints V& in the textual prompt. Semantic constraints 4 are identified by VLMs based the visualized interaction T, and subtask description Tr-- Geometric constraints 4 are then formulated based on semantic constraints &.. keypoint values Vs, and object properties P.. expressing ④, via the trajectory code. The code comprises two components: parameter estimation functions fp, which derives trajectory parameters from object properties, and trajectory generation functions f., employing estimated parameters to generate a sequence of slave object poses relative to the master object, promoting effective generalization across various objects and spatial configurations.
During execution, grasp candidates are uniformly sampled within the learned grasping constraints, and object-centric trajectories predicted from manipulation constraints are converted to end-effector trajectories in the world frame using each grasp candidate of the slave object and object poses of master and slave objects. The resulting end-effector trajectory candidates are evaluated using motion planner, such OMPL (58), the trajectory with the highest fraction is selected.
5

---

Knowledge bank construction. A knowledge bank B is established to archive both high-level planning and low-level skill insights, storing knowledge with key-value pairs (k; U.). High-level planning knowledge is indexed using task description T, as keys, paired with the consequent action sequence T- as values. For low-level skill knowledge, keys are constituted by the object images and subtask description Tr. and values comprise reconstructed objects, as well as semantic constraints 2, and geometric constraints o representing learned skills.
3.3
Skill Adapter with Iterative Comparison
Even though the skill learner exhibits efficient skill acquisition, the demonstration and execution scenes may differ in objects and tasks, impeding direct skill transfer to unseen environments. To mitigate these challenges, VLMs are instructed to adapt skills via an iterative comparison strategy, as depicted in Fig. 2 This approach updates Iearned skills by iteratively contrasting with the demonstrated knowledge, thereby enabling effective adaptation of retrieved skills to novel scenes.
High level planning. High-level planning knowledge T, is retrieved from knowledge bank B based on the task instruction, which acts as the in-context example for VLMs, along with the scene observation. VLMs serve as a physically-groundee actionable steps and descriptions of task-related objects To
task planner (59] 60), generating a sequence of
Iterative comparison, In each iteration, VLMs perform comparative analysis between the adapted interaction and retrieved interaction f, subsequently updating the skill constraints &, and oy. This iterative process persists until either convergence is achieved or the maximum number of iterations N₂ reached. This approach facilitates reasoning VLMS by directing their attention to discrepancies, and enables VLMs to pinpoint the best available solution through an iterative process. The adapting procedure at the i-th iteration can be formally represented as:
I = RRnder ( O),
T T = S. I क., T.),, T g
G. c
+
Vr, (4)
q²
where & and & denote referential constraints, extracted from the knowledge base. The functions S, and G2 adapt semantic and geometric constraints, respectively.
(I) Grasping constraint adaptation. As the grasping orientation is typically derivable from position constraints using grasping models [6]] 62; 633, our work focuses on transferring position constraints. The visualized grasping position space is discretized into X n grid (m, n E Z) and annotated with textual notations TR, obtaining ro VLMs instructed to update semantic constraints T by contrasting with the referential interaction f, and semantic constraints & 2. and adapt geometric constraints o by sampling K outputs of grasping region selection. The updated L are then visualized for the next iteration. The 3-D positional region is represented using two perspectives, and the consistency of the selected regions for the overlapping area validates the VLM outputs. The obtained constraints are expressed via object properties enhance generalization.
(II) Manipulation constraint adaptation. VLMs are instructed to iteratively self-summarise and update manipulation constraints based on the task instruction and seene differences. VLMimic generates trajectories adhering to geometric constraints 4,, which are exhibited on master objects. VLMs are instructed to analyze the deviation of the adapted interaction I, from the referential interaction f, to revise semantic constraints 4., and geometric constraints & undergo refinement predicated on the updated 4 along with trajectory keypoint values VR and object properties P..
Failure reasoning. Despite the ability of VLMs to generate effective constraints, environmental noise, such as trajectory estimation errors, impedes successful task execution. Thus, we leverage VLMs detect and address failures during execution by providing them with perceptual results, such as object pose and robot end-effector trajectories, enabling autonomous failure identification and reasoning for rectification.
4 Experiments
Baselines. VLMimic compared with five representative methods: (1) R3M-DP that utilizes the pre-trained R3M visual representation [3] with the state-of-the-art (SOTA) diffusion policy [7]; (2) Diffusion Policy (DP) 77. a SOTA end-to-end policy method; (3) GraphIRI. 224, a method that
employs graph abstraction and learns reward functions for reinforcement learning (RL); (4) Code

---

Table Success rates on RLbench. "Obs-act", "Template", and "Video" indicate paired observation- action sequences, code templates, and videos performing subtasks.
Methods
R3M-DP
DP
GraphIRL
CaP
Demo2Code
Ours
Overall
0.13(+0.12)
0.15(+0.13)
0.12(#0.12)
0.44(+0.33)
0.49(+0.32)
0.76(#0.17)
Type of
Num of demos
Reach target
Take lid off saucepan
Pick
Toilet seat up
Open box
Open
Methods
demos
up cup
door
R3M-DP
Obs-act
100
0.02
DP
Obs-act
100
0.43
0.25
0.24
0.05
0.04
0.22
GraphIRI.
Video
100
0.39
0.14
0.23
0.03
0.03
0.21
CaP
Template
0.95
0.90
0.58
0.05
0.12
0.65
Demo2Code
Video
5
0.94
0.86
0.65
0.06
0.19
0.83
Ours
Video
5
0.97
0.94
0.80
0.76
0.75
0.90
Type of demos
Num
Meat off grill
Open drawer
Open grill
Open microwave
Open oven
Knife on
Methods
demos
board
R3M-DP
Obs-act
100
0.15
0.25
0.07
0.03
0.00
0.00
DP
Obs-act
100
0.17
0.28
0.09
0.07
0.00
0.00
GraphIRL
Vidco
100
0.16
0.18
0.04
0.04
0.02
0.00
Template
5
0.17
0.46
0.12
0.16
0.78
Demo2Code
Video
5
0.57
0.22
0.40
0.14
0.21
0.79
Ours
Video
5
0.79
0.75
0.81
0.45
0.43
0.76
as Policy (CaP) (64), an LLM--riven method that re-composes API calls to generate new policy code; and (5) Demo2code [37], an LLM-driven planner method that translates demonstrations into task code. We modify it to integrate the analysis results from GPT-4V for Robotics (33), enabling to transcribe videos into code. R3M-DP and DP are trained using the robot demonstrations with paired observation and action sequences. GraphIRL is trained in simulators with paired robot videos, Demo2code and our method learns skills with human vidcos in real-world experiments and robot videos in simulation experiments.
4.1 Simulation Manipulation Tasks
Experimental setup. To assess approach challenging robotic manipulation tasks, the RLBench [65] benchmark utilized for simulation tasks. Due to the unavailability human videos in simula- tions, demo2code and our method utilize robot videos captured from single-camera perspective during demonstrations, incorporating robot gripper trajectories.
Results. We investigate the capacity of VI.imic to acquire skills from limited collection of video demonstrations, without requiring additional training. Our evaluation encompasses 12 manipulation tasks, as detailed in Table demonstrating that our method surpasses all other methods in 11 out of these tasks. Our method, learned with only human videos, obviously outperforms R3M-DP and DP by over 61% in overall performance, despite both being trained on 100 robot demonstrations. Compared to CaP and demo2code, our method demonstrates an improvement exceeding 27%, highlighting the significant performance enhancements facilitated by the VLMimic framework.
4.2 Real-world Manipulation Tasks
Experimental setup. The real-world testing environment (E) is divided into "seen" (SE) and "unseen" (UE) categories. The "seen" category allows for testing in the environment where demonstrations were collected, whereas the "unseen" category involves testing in distinct environment characterized by different objects and layouts. Success criteria are human-eval evalluate and the success rate calculated from 10 randomized object positions and orientations.
Results: To validate the effectiveness VLMimic real-world settings, we conduct experiments involving 14 challenging real-world manipulation tasks selected from recent robotics research [66]
67 4 68 Quantitative results, presented in Table demonstrate that VI.imicc clearly outperforms
other methods across all tasks, particularly in the "unscen" environment (UE). VLMimic achieves an

---

Table Success rates on woaldw manipulation experiments. "Obs-act", "Template", and "Video" indicate paired observation-actio sequences, code templates, and videos performing subtasks. "SE" and "UE" are seen and unseen environments.
Methods
R3M-DP
DP
GraphIRL
CaP
Demo2Code
Ours
Overall (SE) Overall (UE)
(..(2+0220)
0.55 0.2)1
0.39(+:.22) 0.25(=0021) 0.07(+0.08)
0.76(+0.111 0.43(+00211 (-..())
0.09(++010)
0.10( 00.10)
0.37(+0.24)
0.71
Open drawer
Stack block
Open
Put fruit
Press button
Open microwave
Put tray
Methods
Type of demos
Num of demos
oven
on plate
oven
SE
UE
SE
UB
SE
UE
SE
UB
UE
R3M-DP
Obs-nct
100
0.2
0.1
0.6
0.2
0.3
0.0
0.8
0.3
0.7
0.2
0.2
0.0
0.4
0.0
DP
Obs-act
0.3
0.1
0.4
0.7
0.1
GraphIRI.
Video
0.2
0.0
04
0.1
0.0
0.7
0.2
0.4
0.2
0.0
CaP
Template
0.3
0.3
0.5
0,5
0.3
0.2
0.8
0.8
0.7
0.7
0.1
0.1
0.2
0,1
Demo2Code
Video
0.3
0.3
0.5
0.4
0.3
0.1
0.8
0.9
0.8
0.8
0.2
0.1
0.2
Ours
0.8
Tum
Sweep tuble
Insert
Brush pan
Sauce spreud
Put
Pour from
Methods
Type of
Num demos
oven
box
to drawer
cup to cup
demos
SE
UE
SE
UE
SE
UB
SE
UE
SE
UE
SE
UB
SE
UE
R3M-DP
Ohs-act
100
0.2
0.0
0.7
0.2
0.4
0.0
0.6
0.1
0.6
0.1
0.6
0.1
0.5
0.0
DP
Obs-act
100
0.3
0.0
0.8
0.1
0.3
0.1
0.7
0.1
0.7
0.0
0.7
0.1
0.6
0.1
GraphIRL.
Video
0.1
0,2
0.0
CaP
Template
0.3
0.3
0.6
0.5
0.3
0.4
0.3
0.3
0.6
0.2
Demo2Code
Video
0.2
0.1
0.6
0.6
0.3
0.2
0.4
0.3
0.3
0.4
0.7
0.6
0.3
0,2
Ours
Video
0.8
0.7
0.9
0.9
0.6
0.4
0.8
0.7
0.8
0.7
0.8
0.5
Table 3: Success rates on long-horizon tasks. "Obs-act", "Template", and "Video" indicate observation- action sequences, code templates, and videos performing tasks.
Type of demos
Num of demos
Make coffee
Clean table
Make
Wash pan
Make slices
Chem. exp.
Methods
Overall
a pie
R3M-DP
Obs-act
100
0.10
0.30
0.20
0.10
0.00
0.10
0.13(=0.09)
DP
Obs-act
0.00
0.20
0.10
0.10
0.00
0.07(1:0.07)
GraphIRL
Video
100
0.00
0.10
0.00
0.00
0.00
0.00
(+0.0)
CaP
Template
0.00
0.10
0.00
0.00
0.00
0.00
0.02(++004)
Demo2Code
Video
0.00
0.10
0,00
0.00
0.00
0.00
0.02(110.4)
Ours
Video
0.40
0.70
0.70
0.40
0.50
0.30
0.50
improvement exceeding 21% in SE and more than 34% in UE. Results reveal the outstanding ability of VLMimic to acquire skills from human videos and adapt them unseen environments.
4.3 Real-world Long-Horizon Tasks
Experimental setup. Since bascline methods struggle to complete long- long-horizon tasks in the UE setting, experiments are conducted in the SE setting. All other experimental settings consistent with those in the real-world manipulation task.
The performance of VLMimic on long-horizon tasks is quantitatively evaluated by its successful completion of six distinct tasks, each comprising at least five subtasks. Experimental results, as depicted in Table 3 obviously exhibit substantial enhancement achieved by our method over baseline methods. These outcomes suggest that the proposed method is capable of developing robust skills, thereby achieving promising performance in even long-horizon tasks.
Results.
4.4 Robustness against viewpoint variance
The keypoint-centric representation approach enables our method to tolerate different observational perspectives. To demonstrate the robustness of our method to varying viewpoints. Experiments are conducted in real-world unseen environments, utilizing distinet viewpoints, as shown in Figure where the first angle serves the default perspective used in our experiments. Experimental results shown in Table prove that our method exhibits only a 7% fluctuation in performance under varying viewpoints, demonstrating the resilience of VLMimic viewpoint changes.

---

Table Robustness against viewpoint variance.
Methods
Viewpoint I
Viewpoint 2
Viewpoint 3 0.70(10.15)
Viewpoint
Ours
0.71(10.15)
0.67(#0.16)
0.64(
View 1
View
View 3
View
Figure 4: Configuration of various viewpoints.
Open microwave
Oen
Chemistry experiment
IK Error
IK Error
Collision
Figure 5: Examples of failure cases.
4.5 Real-world failure cases
Figure ellucidates scenarios that present significant challenges for resolution through VLM reasoning. These scenarios encompass: (I) The task exccution may exceed the hardware limitations of the physical robot, inducing inverse kinematics (IK) errors. (II) Incomplete environmental perception increases the risk of obstacle collisions, leading to task failure. Since the training datasets for VLMs exhibit significant lack of data related to robot dynamics, these models lack associated knowledge, exhibiting limited capacity for error analysis and struggling to infer correction strategies when confronted with these failures.
4.6 Ablation Studies
Comprehensive ablation studies are conducted to investigate the fundamental designs of our VLMimic approach. The effects of these design decisions assessed by measuring the success rate real- world manipulation tasks, which computed across 10 randomized object positions and orientations.
Hierarchical constraint representations. Table 5 (a) analyzes three distinct constraint representa- tions. Variants that exclusively reason semantic constraints or directly obtain geometric constraints without semantic analysis, lead to diminished performance. The results exhibit that hierarchical constraint representations enhance skill acquisition capabilities, demonstrating the pivotal role in facilliatiig the understanding and reasoning capabilities of VLMs.
Grasping learning. Table (b) presents variants and their respective performance. The first variant utilizes VLMs for direct prediction of constraint region values, resulting in a significant performance decline. The second variant employs the DBScan clustering algorithm to group grasp poses and derive constraints as bounded regions. However, this method only considers numerical distributions without incorporating grasping common sense, leading to performance degradation.
Number of human videos. T(cle(5(c) presents an analysis of the impact of human video quantity on performance. Results indicate that our method attains high success rates on complex tasks with single human video demonstration, and increasing the number of videos yields performance gains. The results show that our approach can efficiently learn generalizable skills from a limited number of human videos. We choose to use 5 demonstration videos to balance data availability and performance.
Comparison strategy. Table 5 (d) analyzes the impact of the comparison strategy in skill adapters. Variants compare constraints exclusively utilizing either visualized interactions or keypoints exhibit decreased success rates. Visual comparison facilitates semantic contrast in VLM, while keypoint values provide fine- fine-graine greengraied geometric information. Experimental results illustrate that our strategy facilitates reasoning for both semantic and geometric constraint adaptation.
Number of iterations. We conduct an analysis on the impact of iteration count in skill adapter and search for the optimal choice, shown in Table Reducing the number of iterations to results in

---

Table 5: Ablation experiments with VLMimic on real- world manipulation experiments. "SE" and "UE" are seen and unseen environments. Default settings are marked in gray
(a) Hicrarchical representations.
(b) Grasping learning.
(c) Number of videos.
Variants
SE
Variants
SE
Number
SE
Number
SE
Geometric constraints
0.61
Value prediction
0.52
I 0.68
7
0.67
Semantic constraints
0.68
Grouping (DBSCAN)
0.59
3
0.72
9
0.78
Hierarchical constraints
0.76
Grouping with VL.Ms
0.76
5
0.76
11
0.78
(d) Comparison strategy.
(e) Number of iterations.
(1) Failure reasoning.
Variants
UE
Number
UE
Number
UE
Number
UE
Number
UE
Visual comparison
0.61
0
0.58
3
0.68
0.65
0.72
Keypoint comparison
0.60
1
0.61
0.71
0.68
0.70
Visual with keypoints 0.71
2 0.66
5
0.71
0.71
0.71
a noticeable decrease in performance. Strong results are observed in the initial iteration, with modest improvements in subsequent iterations. The findings indicate that this iterative approach enhances the effectiveness skill adaptation by enabling VLMs to identify the best available solution. For enhanced performance, 4 iterations selected.
Failure reasoning. The impact of failure reasoning is investigated in Table 5 f). The success rate exhibits upward trend with increasing iterations, reaching an elbow point at 2 iterations, providing an optimal trade-off between real-time performance and success rate. Failure reasoning proves crucial for tasks demanding sege-precision manipulation, which are susceptible to environmental noise. It enhances both the success rate and the robot's ability to operate in intricate environments.
Conclusion
In this paper, we present VLMimic, novel approach that harnesses VLMMs to learn skills with even fine-grained action levels from a limited number of human videos, and effectively generalize them to unseen environments. VLMimic first extracts object-centric interactions from human videos, and learns skills based on these interactions, using hicrarchical constraint representations. In unseen environments, these skills updated through iterative comparison strategy. Extensive experiments conducted on various manipulation and challenging long-houd horizon tasks demonstrate the superior performance achieved by our VLMimic, utilizing only 5 human videos without requiring additional training. exhibiting strong skill acquisition and adaptation capabilities.
Limitations. Despite the promising performance exhibited by VI.Mimic, current VI.Ms are still limited by inference latency and computational resource requirements. We believe that the progression of lightweight VLMs will mitigate these limitations.
References
[1] Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Silvio Savarese, and Li Fei-Fei. Learning to generalize across long- horizon tasks from human demonstrations. arXiv preprint urXiv:2003.06085 2020.
[2] Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martin, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Learning multi-arm manipulation through collaborative teleoperation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9212-9219. IEEE, 2021.
[3] Andy Zeng, Pete Florence, Jonathan Tompson. Stefan Welker. Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747, PMLR, 2021.
[4] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint ar 22Xv:22222.068 2022.
[5] Anthony Brohan, Noah Brown. Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding. Danny Driess. Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision caroon--nnnugg--atte action models transfer web knowledge robotic control. arXiv preprint arXiv:2307.158811 158188 2023.
10

---

[6] Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan. Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310008880 2023.
[7] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0413 2023.
[8] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302001877 2023.
[9] Mingxiao Huo, Mingyu Ding. Chenfeng Xu, Thomas Tian, Xinghao Zhu, Yao Mu, Lingfeng Sun. Masayoshi Tomizuka, and Wei Zhan. Human-orienter representation learning for robotic manipula- tion. arXiv preprint 10.00023, 2023.
[10] Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16467-16476, 2024.
[11] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar. Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world 3,000 hours of egocentric vidco. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 18995-19012, 2022.
[12] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint wrXiv:2203.06/73. 2022.
[13] Suraj Nair. Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 2022.
[14] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, pages 654-665. PMLR, 2023.
[15] Karl Schmeckpeper, Oleh Rybkin, Kostas Danillidis, Sergey Levine, and Chelsea Finn. Reinforcement learning with videos: Combining offline observations with interaction. arXiv preprint (tti::2011.005500 2020.
[16] Ashley D Edwards and Charles I. Isbell. Perceptual arXiv: 09050076), 2019.
values from observation. arXiv preprint
[17] Karl Schmeckpeper. Annie Xie, Oleh Rybkin, Stephen Tian, Kostas Danilidiss, Sergey Levine, and Chelsea Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pages 708 725. Springer, 2020.
[18] Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103. 16817, 2021.
[19] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concep12robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14): 1419:)444491444,, 2021.
[20] Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg. and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement lcarning. In Conference Robot Learning, pages 537-546. PMLR, 2022.
[21] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-rohot imitation in the wild. arXiv preprint arXiv:2207.09450 2022.
[22] Maximilian Sieb, Zhou Xian, Audrey Huang, Oliver Kroemer, and Katerina Fragkiadaki. Graph-structured visual imitation. In Conference on Robot Learning, pages 979-989. PMLR, 2020.
[23] Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
[24] Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. In Conference on Robot Learning, pages 55-66. PMLR, 2023.
[25] Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi- tasks via pixel-level translation of human videos. arXiv preprint 1912.v1112..4444 2019.
11

---

[26] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE international conference on roboticx and automation (ICRA), pages 1118-1125. JEEE, 2018.
[27] Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj. Samarth Sinha, and Animesh Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 202/ IEFF/RS] International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEFE, 2021.
[28] Oier Mees, Markus Merklinger, Gabriel Kalweit, and Wolfram Burgard. Adversarial skill networks: Unsupervised robot skill learning from video. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4188 4194, IEEE, 2020.
[29] Eric Jang, Alex Irpan. Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[30] Yao Mu, Qinglong Zhang. Mengkang Hu. Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang. Jifeng Dai. Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advancex in Neural Information Processing Systems, 36, 2024.
[31] Yao Mu, Junting Chen, Qinglong Zhang. Shoufa Chen, Qiaojun Yu, Chongjian Ge, Runjian Chen, Zhixuan Liang, Mengkang Hu, Chaofan Tao, ct al. Robocodex: Multimodal code generation for rohotic behavior synthesis. arXiv preprint a402v:2402. /6/ 2024.
[32] Guangyan Chen, Te Cui, Tianxing Zhou, Zicai Peng, Mengxiao Hu, Meiling Wang, Yi Yang, and Yufeng Yue. Human demonstrations are generalizable knowledge for robots. arXiv preprint arXiv:231..02419, 2023.
[33] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration. arXiv preprint arX1v:2311.12015 2023.
[34] Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai. Pretrained language models as visual planners for human assistance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15302-15314, 2023.
[35] Yuetian Weng. Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvim: Efficient long video understanding via large language models. arXiv preprint arXiv:2404. 2404.33444 2024.
[36] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint (tl:2335500355. 2023.
[37] Yuki Wang, Gonzalo summarizing demonstrations to synthesizing code via extended chain-of-thoughe Advances in Neural Information Processing Systems, 36, 2024.
Yash Sharma, and Sanjiban Choudhury. Demo2code: From
[38] Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempe: Large language models as decision makers for autonomous driving. arXiv preprint 2rXiv:2310.03022 2023.
[39] Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wengi Shao, Qiguang Chen, Bin Wang. Yu Qiao, and Ping Luo. Tree-planner: Efficient close-loop task planning with large language models. arXiv preprint ar XXX::231000856
2023.
[40] Zeyu Gao, Yao Mu, Jinye Qu, Mengkang Hu, Lingyue Guo, Ping Luo, and Yanfeng Lu. Dag-plan: Generating directed acyelic dependency graphs for dual-arm cooperative planning. arXiv preprint 24ki::2406.099533 2024.
[41] Neha Das, Sarah Bechtle, Todor Davchev. Dinesh Jayaraman, Akshara Rai, and Franziska Meier. Model- based inverse reinforeement learning from visual demonstrations, In Conference on Robot Learning, pages 1930-1942, PMLR, 2021.
[42] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306.000 03514. 2023.
[43] Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Minggi Gao, Shanshan Zhao, Ying Shan, et al. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint 2305.0267. 2023.
12

---

[44] Ting Pan. Lulu Tang. Xinlong Wang, and Shiguang Shan. Tokenize anything via prompting. arXiv preprint urXiv:2312.09128. 2023.
[45] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre- are-training for open-set object detection, arXiv preprint arXiv:2303.05499 2023.
[46] Zongxin Yang. Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[47] Zongxin Yang and Yi Yang. Decoupling features in hierarchical propagation for video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[48] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0260 02643,, 2023.
[49] Yangming Cheng, Liulei Li. Yuanyou Xu, Xiaodi Li, Zongxin Yang. Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06558, 2023.
[50] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008. 08324, 2020.
[51] Paul Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[52] Szymon Rusinkiewicz and Mare Levoy. Efficient variants of the iep algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pages 145 155-152. IEEE, 2001.
[53] Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 606-617,
2023.
[54] Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. arXiv preprint arXiv:2312.08344 2023.
[55] Yen-Jen Wang, Bike Zhang, Jianyu Chen, and Koushil Sreenath. Prompt a robot to walk with large language models. arXiv preprint (3XX::3309009900 2023.
[56] Suvir Mirchandani, Fei Xia, Pete Florence, Brian lchter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.0472.1 2023.
[57] Dmitry Berenson, Siddhartha Srinivasa, and James Kuffner. Task space regions: A framework for posc- constrained manipulation planning. The International Journal of Robotics Research, 30(12):1435-1445 2011.
[58] Ioan A Sucan. Mark Moll, and Lydia E Kavraki. The open motion planning library. IEEE Robotics Automation Magazine, 19(4):2--82 82, 2012.
[59] Marta Skreta, Zihan Zhou, Jia Lin Yuan, Kourosh Darvish, Alán Aspuru Aspuru-Guzik, and Animesh Garg. Replan: Robotic replanning with perception and language models. arXiv preprint arXiv: 04117, 2024.
[60] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look hefore you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.1784 2023.
[61] Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet- Graspnet-Ibillion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11 11444 11453, 2020.
[62] Hao-Shu Fang, Chenxi Wang, Hongjic Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yiehen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception spatial and temporal domains. JEEE Transactions on Robotics, 2023.
[63] Yuanchen Ju. Kaizhe Hu, Guowei Zhang, Gu Zhang. Mingrun Jiang, and Huazhe Xu. Robo-abe: Affordance generalization beyond categories via semantic correspondence for robot manipulation. arXiv preprint arXiv:2401.07487. 2024.
13

---

[64] Jacky Liang. Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[65] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5(2):3019-302 2020.
[66] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Keerthana Gopalakrishnan. Karol Hausman, et as can, not say: Grounding language in robotic affordances. arXiv preprint orXiv:2204.0169/ 2022.
[67] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Authony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:2211.11777 2022.
[68] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang. Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian lchter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint ar3iv: /150, 2023.
[69] Peize Sun, Shoufa Chen, Chenchen Zhu, Fanyi Xiao, Ping Luo, Saining Xie, and Zhicheng Yan. Going denser with open- vocabulary part segmentation. arXiv preprint 1Xxv:2305..1173 2023.
[70] Peize Sun, Shoula Chen, and Ping Luo. Grounded segment anything: From objects to parts. https
Btt.a//emmmm/eeeeeeeeemmiiii //github. com/ Cheemsh Semi hww.gggooondddddeeeeee.....oeeee htt.a/awww.......o..ooooooooaona any-parts 2023.
[71] Sami Haddadin, Sven Parusel, Lars Johannsmeier, Saskia Golz, Simon Gabl, Florian Walch, Mohamadreza Sabaghian, Christoph Jähne, Lukas Hausperger. and Simon Haddadin. The franka emika robot: A reference platform for robotics research and education. IEEE Robotics Automation Magazine, 29(2):46-6 64, 2022.
14

---

A Implementation details
In sluman-objedt interaction grounding module, the Tokenize Anything 44 model is employed during task recognition to improve tine-grained scene understanding ability. The textual detection results are integrated using VLMs to generate concise task descriptions and detailed object information. The videos are segmented using threshold e of 2cm. Segments with hand motion trajectory lengths below y 10cm are discarded. During the grasping constraint learning phase, the number of regions Ne is automatically determined by the VLMs. In manipulation constraint learning, keypoints are obtained by uniformly sampling 10 points. For the skill adapter, the maximum number of iterations is set to N2 = 4. During grasping constraint adaptation, visualized grasping position space is discretized into a 5 x 5 grid, with K = 4 outputs sampled per iteration.
During skill execution, the pretrained Grounded-sege seeedddt--meetteee may-parts model [69] 70 is used to generate segmentation maps of queried objects or parts. These segmentation maps are then utilized to predict object-centric pose sequences using codes generated by VLMs. FoundationPose [54) employed to track object poses, transforming the object-centric poses into the world frame. The robotic arm's motion planning is facilitated by the integration of the Movelt module, renowned for its comprehensive motion planning capabilities, and the OMPL [58) (Open Motion Planning Library), which offers a suite of advanced algorithms for efficient path planning and obstacle avoidance. Upon action completion, the real-time object positions are used to assess task success until manual confirmation or preset time reached. In case of failure detection, object and gripper poses are employed for fallure reasoning, where the gripper poses are estimated using the attatched QR scan.
B Experimental Setup
B.1 Baseline setup
and DP 7 employ CNN-basee network architecture for its robustness across diverse tasks. These methods are trained on robot demonstrations using default demonstrations consist of paired observation and action sequences.
R3M-DP 13
robot
To ensure that GraphIRL. 24 trained and tested under the same scenario the SE setting. GraphIRI. is trained in the simulator with corresponding paired robot videos, and the SE results are obtained from the same simulated environments, while UE results are acquired from real-world scenarios under the UE setting. Since the original GraphIRL method struggle to learn the gripper switch information, we additionally provide this information to GraphIRL.
Following Cap [64]. the primitives for Cap [64] and demo2code [37 include: move to pos, rotate by quat, set vel, open gripper, close gripper. pick obj, and place at pos. Cap employs natural language instruction directly for reasoning, Demo2code generates code from textual video analysis results provided by GPT-4V for Roboties [33], video analysis approach for robotics, enabling demo2code to learn from human videos. Specifically, the detailed task analysis results and affordance analysis outcomes from GPT-4V for Robotics are incorporated as contextual information within the textual prompt for demo2code.
B.2 Real-world experimental setup
Experiments are conducted on Franka Emika 771, employing three RGB-D cameras (ORBBEC Femto Bolt) for environment observation: one the top right of the table, one the top left, and one mounted on the robot's wrist. All cameras start recording and return real-time RGB-D observations frequency of 30 Hz. All experiments evaluated Intel 17-10700 CPU with RTX 3090 graphics card.
C Details of the long-horizon task designs
The definition of our long-horizon tasks is listed below. For each task, the initial state and subgoals are pre-defined. The whole task is completed if and only if all subgoals are completed in the correct order.
15

---

C.1 Kitchen
Make pie
Initial state: On the table, there is bowl filled with sauce, pan containing pie, brush
placed on the shelf, and a microwave.
Criteria: Brush the sauce on the pie and put the pie in the microwave to heat up.
Subgoals: (1) Grasp the bowl; (2) Pour sauce from the bowl to the pie; (3) Grasp the brush; (4) Spread sauce; (5) Open the microwave; (6) Place the pan the microwave; (7) Close
the microwave; (8) Turn on the microwave.
(2) Pour sauce from the bowl
Initial state
(1) Grasp bowl
to the ple
(3) Grasp the brush
(4) Spread sauce
(5) Open the microwave
Place the pan In the microwave
(7) Cose the microwave
(8) Turn on the microwave
Figure 6: Visualization of the make-a-pie task.
Wash pan
Initial state: The pan that needs to be washed is located on the left side of the table, the pan rack and brush are the right side of the table, and the sink and faucet in the middle of the table.
Criteria: Rinse the pan with water, scrub it with a brush, and place the on the rack.
Subgoals: (1) Place the pan in the sink; Al)gn the faucet with the pan; (3) Turn the faucet; (4) Turn off the faucet; (5) Place the pan on the table; (6) Wipe the pan with a brush; (7) Place the pan the pan rack.
16

---

Align the faucet with the
(0) state
(1) Place pan sink
Turn the faucet
(4) Turn off the faucet.
(5) Place pan the table
(6) Wipe the pan with brush
(7) Place the pan on
pan rack
Figure 7: Visualization of the wash-pan task.
Make cucumber slices (Make slices)
Initial state: The refrigerator is to the left of the table, the cutting board is on the shelf to the right of the table, next which is knife inserted into the knife rack.
Criteria: Take the cucumber out of the refrigerator and cut it with knife.
Subbools: (1) Place the cutting board the table; (2) Open the refrigerator; (3) Place the cucumber on the cutting board; (4) Close refrigerator; (5) Remove the knife from the knife rack; (6) Cut the cucumber; (7) Place the knife back on the knife rack.
Initial state
Turn the faucet Turn the faucet
(3) Place the cutting board on table
(4) Place the cucumber on the cutting board
(5) Remove the knife from the knife rack
(7) Place the knife back the knife rack
(6) Cut the cucumber
Figure 8: Visualization of the make-cucumberrstices task.
17

---

C.2
Table
Make coffee
Initial state: The coffee machine and capsules are placed on the tabletop, with the capsule
chamber placed on the cup.
Criteria: place the coffee capsule into the capsule chamber, insert it into the coffee machine,
place cup under the coffee machine's dispenser, and finally turn the coffee machine.
Subgoals: (1) Grasp the coffee capsule; (2) Place the coffee capsule in the capsule chamber; (3) Grasp the capsule chamber; (4) Insert the capsule chamber into the coffee machine; (5) Place the cup under the coffee machine's 's water outlet; (6) Grasp the lever; (7) Turn on the coffee machine.
(2) Place the coffee capsule in the capsule chamber
(0) Initial
(1) Grab coffee capsule
(3) Grasp the capsule chamber
(4) Insert the capsule chamber the coffee machine
(5) Place the cup under the coffee machine's water outlet
(7) Turn the coffee machine
(6) Grasp the lever
Figure 9: Visualization of the make-coffee task.
Clean table
Initial state: Bananas, mangoes, cups, and paint brushes are scattered on the table. In addition, there is drawer, a plate, and dust brush on the table.
Criteria: Put the fruits (banana and mango) back into the plate, and put the tools (cup and paint brush) back into the drawer, and sweep the tabletop with the dust brush.
Subgooas:: (1) Place the banana on the plate; (2) Place the mango on the plate; (3) Open the drawer: (4) Place the brush in the drawer; (5) Place the cup the drawer; (6) Close the
-
drawer; (7) Grasp the brush; (8) Sweep the tabletop.
18

---

(1) Place the banana on the plate
(2) Place the mango on
(0) Initial state
the plate
(4) Place the brush the drawer
(5) Place the cup drawer
(3) Open the drawer
(6) Close the drawer
(7) Grasp the brush
(8) Sweep the tabletop
Figure 10: Visualization of the clean-table task.
C.3
Chemistry Lab
Chemistry experiments (Chem. exp.)
Initial state: On the desktop, there are two beakers, two conical flasks, a test tube rack equipped with test tube, along with retort stand fitted with funnel.
Subgoals: (1) Place conical flask A under the funnel. (2) Pour the contents of the test tube into beaker A. (3) Pour the contents of beaker A into conical flask B. (4) Pour the contents of beaker B into conical flask (5) Shake the mixed solution in conical flask (6) Pour the contents of conical flask B into the funnel.
(1) Place conical flask under the funnel
Pour contents test tube beaker
(0) Initial state
(3) Pour the contents of beaker into conical flask B
Pour the contents beaker B into conical flask
(5) Shake the mixed solution in conical flask B
(6) Pour the contents of conical flask into the funnel
Figure 11: Visualization of the Chemistry Lab task.
19

---

D Visualization of experimental results
Figure 12: Manipulated objects in SE setting.
Figure 13: Manipulated objects in US setting.
20

---

Open drawer
Stack block
Open oven
Put fruit on plate
Press button
Open microwave
Put tray in oven
Turn oven
Sweep table
Insert box
Brush pan
Sauce spread
Put toy to drawer
Pour from cup to cup
Figure 14: Visualization of manipulation task results in seen environments.
21

---

Oen drawer
Stack block
Open oven
Put fruit on plate
Press button
Open microwave
Put tray in oven
Turn oven
Sweep table
Insert box
Brush pan
Sauce spread
Put toy to drawer
Pour from cup to cup
Figure 15: Visualization of manipulation task results in unseen environments.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We harnesses VLMs to directly learn even fine- fine-grained action levels, only given limited number of human videos.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: please refer Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not involve theoretical derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental setup please refer to Sec. and Appendix and implementation details please refer to Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[Yes]
Justification:
Our code and data will be made pubilcly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public htllc//guiesseeee CodeeSeoe ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. w/npps.cpppbblli wttpccccpppppppcteeeceeccccccccp cesii///uies///eeeeeeeoeooooeeee ies CodeeSumm ss cioooooliio ooooicyyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental setup please refer to Sec. and Appendix and implementation details please refer to Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: please refer to Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research presented in this paper adheres the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not foresee obvious undesirable ethical or social impacts at this moment.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve the release of data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators and original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, baaasettsppperrsseeee paperswws thoode. com datasett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The experiments and research in this paper do not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The experiments and research in this paper do not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "ntkFxaPO9B", "reviewer": "Reviewer_Y6C5", "summary": "This paper proposes a method to learn a policy from human videos utilizing advances in vision-language models. The authors first parse human videos into several keyframes, then detect and track hand-object interactions. From the segmented interactions, they learn low-level actions by following the 3D hand-object trajectories in each segment. Additionally, the authors propose an iterative comparison stage to adapt to new scenes or objects. The method achieves better performance compared to other visual imitation learning algorithms in both simulation and real-world evaluations.", "questions": "The proposed system is rather complicated. While I appreciate the authors' efforts in assembling such a complex system, it is honestly a bit difficult to understand. I have detailed a few technical questions below.\nIn the grounding module:\n* How are the keyframes obtained?\n* How does the language model identify the important object to detect (e.g., pie, plate, brush, and bowl)?\n* How are candidate action primitives (such as grasping and manipulation) determined?\nFor skill learning:\n* Can you provide some examples of the Multiple-choice QA?\n* How are the keypoints obtained?\n\nI appreciate the evaluation on unseen scenes and objects. However, how similar do the camera viewing angles need to be? I assume there must be some degree of robustness because the perception operates in 3D, but is there any evaluation of this aspect?\n\nAs mentioned in the previous section, the results for R3M-DP and DP are surprisingly low. Have you tested the implementation on other tasks (such as those reported in the original paper) to verify the correctness?", "limitations": "The authors report limitations in the supplementary material. The limitations section is comprehensive, though I would suggest moving it to the main text.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a novel usage of VLM for imitation learning. The perception module is well-designed, and the ability to learn skills from perceived trajectories is particularly interesting.\n\nThe paper presents comprehensive evaluations in both simulation and the real world. It also includes detailed ablation experiments on different design choices of the proposed system.\n\nIt demonstrates superior performance over previous methods, especially on unseen objects and tasks.", "weaknesses": "While this paper is well-written and achieves good performance, I have a few concerns regarding some claims and evaluations of the proposed method.\n\nFirst, the paper focuses on fine-grained action imitations. However, I’m not convinced that the skills demonstrated in the paper are fine-grained or distinct from previous work that uses VLM for planning. Specifically, the skills in this paper primarily involve grasping and moving along certain trajectories, without complex interactions (actions requiring precise manipulation such as insertion or grasping that utilizes the environment).\n\nOn the other hand, similar skills can also be learned using the 'VLM as Planner' framework. For example, the 'Code as Policy' approach can perform tasks such as writing with pens. This skill is quite similar to the 'Spread sauce' motion described. Therefore, the claim that 'Fine-grained motions' cannot be learned in the 'VLM as Planner' framework needs further explanation.\n\nThe proposed method seems to categorize motion skills as 'grasping' and 'manipulation'. If the point is that 'VLM as Planner' requires each skill to be defined manually while the proposed method can automate this process, it would be beneficial to clarify that the proposed method still defines motion skills, but in a broader or more generalizable manner.\n\nI think the performance of the baseline imitation learning algorithms is surprisingly low. Given 100 demonstrations, R3M-DP and DP should be quite successful, at least in simpler tasks such as Reach Target. This assessment is based on the original paper where they achieve a high success rate in more difficult tasks, as well as my empirical experience. Could you clarify whether the evaluation protocol was changed to make the task much harder?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The tasks lack complex interactions.**\n\nThanks for your valuable feedback! We extend our evaluation to include complex tasks, and **demonstrate strong performance on FMB [1], which demands precise manipulation**. FMB evaluates complex robotic manipulation tasks encompassing the grasping and reorienting that utilizes the fixture, and inserting objects. Our method outperforms comparison methods, requiring only five human videos. An example is illustrated in Figure 1 of the attached PDF.\n\n|Methods|Type of demos|Num of demos|Overall|Round|Rectangle|Oval|Hexagon|Arch|Star|Double Square|Square Circle|Triple Prong|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|DP|Obs-act|100|$0.47(\\pm0.16)$|0.7|0.6|0.4|0.4|0.3|0.7|0.4|0.3|0.4|\n|CaP|Template|5|$0.11(\\pm0.19)$|0.6|0.2|0.1|0.0|0.0|0.1|0.0|0.0|0.0|\n|Ours|Video|5|$0.49(\\pm0.13)$|0.8|0.5|0.4|0.5|0.4|0.5|0.5|0.4|0.4|\n\n[1] Luo J, et al. FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning[C]//CoRL 2023 Workshop on LEAP.\n\n**Q2. Similar skills can also be learned using the 'VLM as Planner' framework.**\n\nSorry for the confusion. We would like to clarify that **while methods utilizing VLMs as planners can perform similar tasks, fine-grained motion skills are not learned by themselves**. As shown in Figure 2 of the attached PDF,  (a) the motion skill in the drawing experiments encompasses fine-grained control to ensure continuous contact and trajectory adherence of the pen on the whiteboard surface, this skill is implemented via the **human-written function** `draw(pts_2d)`. In contrast, (b) these fine-grained motion skills are autonomously acquired by our VLMimic.\n\n**Q3. The proposed method still defines motion skills.**\n\nThanks for your valuable feedback! We will clarify that VLMimic defines skills in a broader manner. Furthermore, it is crucial to emphasize that **the motion primitives for 'VLM as planner' methods encompass specific skill implementations**, necessitating manual function coding, or specific model training. In contrast, **our approach merely categorizes skills into grasping and manipulation, without additional human efforts**, therefore significantly augmenting the task learning capabilities for robotic systems.\n\n**Q4. Performance of Diffusion policy.**\n\n**In RLBench**, we follow the commonly utilized mutli-task training paradigm and implement the task-conditioned DP approach, however, DP fails to manage different skills and generalize to test examples. Following your suggestions, we conduct single-task experiments, wherein each DP model is constrained to execute a singular task. Despite these constraints, VLMimic still exhibits superior performance, requiring only 5 human videos.\n\n|Methods|DP|Ours|\n|-|-|-|\n|Overall|$0.66(\\pm0.28)$|${0.80(\\pm0.16)}$|\n\n|Methods|Type of demos|Num of demos|Reach target|Take lid off saucepan|Pick up cup|Toilet seat up|Open box|Open door|\n|-|-|-|-|-|-|-|-|-|\n|DP|Obs-act|100|1.00|1.00|0.85|0.77|0.57|0.67|\n|Ours|Video|5|1.00|0.98|0.85|0.79|0.76|0.92|\n\n|Methods|Type of demos|Num of demos|Meat off grill|Open drawer|Open grill|Open microwave|Open oven|Knife on board|\n|-|-|-|-|-|-|-|-|-|\n|DP|Obs-act|100|0.83|0.79|0.73|0.21|0.10|0.41|\n|Ours|Video|5|0.87|0.77|0.85|0.47|0.50|0.79|\n\n**In the real-world experiments**, we follow the evaluation protocol in the original paper of DP and discover two primary failure factors: (1) DP exceeds robot hardware limits, inducing inverse kinematics errors. (2) DP exhibits safety-violating behaviors, such as hitting the ground.\n\n**Q5. How are the keyframes obtained?**\n\nKeyframes are extracted by sampling video frames every three seconds. In instances where the extracted keyframes are numbered fewer than five, we uniformly sample five keyframes.\n\n**Q6. Important object identification.**\n\nAs shown in Figure 3 of the attached PDF, keyframes are grouped in fives and concatenated into consolidated images. These images prompt VLMs to generate task descriptions. VLMs then analyze individual keyframes based on task descriptions, selecting important objects from detection results.\n\n**Q7. Candidate action primitives determination.** \n\nKeyframes are extracted from subtask video segments. These keyframes, along with interacting objects within the subtask, are input to VLMs to distinguish between the grasping and manipulation phases. The interacting objects are the two objects satisfy ${d}^{t-1}>\\epsilon\\wedge{d}^{t}<\\epsilon$, where $d$ denotes the distance between two objects. Examples are provided in Figure 9 of the attached PDF.\n\n**Q8. Examples of the Multiple-choice QA.**\n\nAs illustrated in Figure 4 of the attached PDF, VLMimic demonstrates proficiency in inferring grasping constraints.\n\n**Q9. How are the keypoints obtained?**\n\nIn the grasping phase, the keypoints are represented by the grasp poses. \nIn the manipulation phase, keypoints are derived through a uniform sampling of 10 points along the estimated trajectory.\n\n**Q10. Viewpoint variance.**\n\nThanks for your constructive suggestion! Experiments are conducted in real-world unseen environments, utilizing distinct viewpoints, as shown in Figure 5 of the attached PDF. Results demonstrate the resilience of VLMimic to viewpoint changes.\n\n|Methods|Viewpoint 1|Viewpoint 2|Viewpoint 3|Viewpoint 4|\n|-|-|-|-|-|\n|Ours|$0.71(\\pm0.15)$|$0.67 (\\pm 0.16)$|$0.70 (\\pm 0.15)$|$0.64 (\\pm 0.17)$|\n\n**Q11. Diffusion policy on similar tasks.**\n\nDP performs best in 'Lift' and 'Can' tasks. Tests on similar RLBench tasks show success rates in single-task training align with the original study's findings.\n\n|Methods|Type of demos|Num of demos|Overall|Reach target|Take lid off saucepan|press button|Meat off grill|\n|-|-|-|-|-|-|-|-|\n|DP|Obs-act|100|$0.93(\\pm0.08)$|1.00|1.00|0.91|0.83|\n\nReal-world experiments reveal decreased success rates, primarily due to inverse kinematics errors and safety violations, as mentioned in Q4.\n\n|Methods|Type of demos|Num of demos|Overall|Reach target|Lift|Can|\n|-|-|-|-|-|-|-|\n|DP|Obs-act|100|$0.81(\\pm0.08)$|0.90|0.80|0.75|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcUEAeHdQ8", "reviewer": "Reviewer_xpmL", "summary": "This paper introduces a novel paradigm named VLMimic, which employs Vision-Language Models (VLMs) and multiple vision tools to ground object-centric information from demonstrations into fine-grained actionable skill sets. VLMimic exhibits a remarkable success rate in manipulation tasks and demonstrates significant generalization capabilities, learning effectively from a minimal number of human and robot videos.", "questions": "1. In the iterative comparison process, how does VLMimic determine which object, from the extensive knowledge bank of diverse objects, is utilized as the reference for guiding the adaptation of grasping and manipulation constraints?\n\n2. Has the potential for cumulative error in the proposed modular design been taken into account? Specifically, is there a risk that an error at any stage of the process could ultimately lead to the failure of the manipulation task?", "limitations": "Possible limitations are discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method performs well even with very few demonstrations (less than 5 videos).\n2. The evaluations are substantial, encompassing both simulation and real-world experiments, with diverse tasks conducted and generalization of multiple aspects (position, intervention, cross-embodiment and cross-task) verified.\n3. The writing in general is well-organized and easy to follow.", "weaknesses": "The proposed object-centric manipulation pipeline encompasses multiple foundation models / vision tools to generate various intermediate representations (object detection results, object 6D poses, hand mask and trajectories, and task description from VLMs). This integration relies heavily on human prior knowledge to string all these models, rendering the process lengthy and complex.  Moreover, this approach may only be adaptable to a limited set of interaction objects (for example, whose 'affordances',  or 'object properties' as proposed in the paper, can be easily defined and recognized).\n\n\n*[Minor]* According to line167-168, the semantic constraints are identified based on both visual interaction $I_{v}$ and sub-task description $T_{\\tau}$. But only  $I_{v}$ is incorporated in Equation 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The proposed pipeline encompasses multiple foundation models/vision tools.**\n\nThanks for your feedback! We hope to address your concerns through the following two points.\n\n(1) **Our VLMimic demonstrates strong robustness against the cumulative error.** Motivated by your insightful feedback, we conduct an evaluation of the resilience of VLMimic to cumulative errors through perturbation introduction at various stages. The empirical results substantiate the robust performance of our framework. Details are presented in section Q5.\n\n|VLMimic|Interaction grounding errors|Knowledge retrieval errors|Execution errors|\n|-|-|-|-|\n|$0.76 (\\pm 0.17)$|$0.68 (\\pm 0.21)$|$0.64 (\\pm 0.21)$|$0.73 (\\pm 0.17)$|\n\n(2) **Foundation models/vision tools are primarily utilized in the data generation phase, which can facilitate efficient skill learning without compromising robustness.** The implemented foundation models are primarily distributed in the human-object interaction grounding module, which functions as a data generator. This automated data generation approach substantially mitigates the need for human data annotation. Furthermore, selective manual annotation of a minimal subset of erroneous data can further enhance data quality. Therefore, the integration of foundation models enables accelerated skill acquisition paradigms while preserving the robustness of the learning process.\n\n**Q2. Learned interactions are confined to the easily defined affordances or object properties.**\n\nOur extensive experiments demonstrate the method's broad versatility across diverse objects and interactions:\n\n**Object diversity**: The introduced object properties primarily comprise bounding boxes, universally applicable to all objects. The experiments encompass **42 distinct object classes**, ranging from household objects to laboratory equipment, illustrating adaptability to varied objects.\n\n**Affordance complexity**: Our method demonstrates proficiency in **62 low-level skills**, including complex interactions like capsule insertion and liquid transfer, showcasing its ability to learn diverse affordances. Moreover, we extend our evaluation to include more challenging tasks, **demonstrating strong performance on FMB [1], which demands precise manipulation**. FMB evaluates complex robotic manipulation tasks encompassing the grasping and reorienting that utilizes the fixture, and inserting objects. Our method achieves superior performance to DP, requiring only five human videos. An example is illustrated in Figure 1 of the attached PDF.\n\n|Methods|Type of demos|Num of demos|Overall|Round|Rectangle|Oval|Hexagon|Arch|Star|Double Square|Square Circle|Triple Prong|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|DP|Obs-act|100|$0.47 (\\pm 0.16)$|0.70|0.60|0.40|0.40|0.30|0.70|0.40|0.30|0.40|\n|Ours|Video|5|$0.49 (\\pm 0.13)$|0.80|0.50|0.40|0.50|0.40|0.50|0.50|0.40|0.40|\n\n[1] Luo J, et al. FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning[C]//CoRL 2023 Workshop on LEAP.\n\n**Q3. Equation 3.**\n\nThanks for your feedback! We will modify the equation according to your feedback and review all parts of the article to address any similar issues.\n\n**Q4. How does VLMimic determine which object is utilized as the reference?**\n\nThe retrieval example is presented in the lower part of Figure 8 within the attached PDF: Based on the queried subtask description, we leverage the text encoder [2] to select $N_{t}=2$ demonstrations with the highest similarity between their associated subtask descriptions and the query. Subsequently, from the selected demonstrations, those exhibiting the highest degree of object similarity are retrieved. The image similarities based on the CLIP model [3] are calculated for both master and slave objects between the current scenarios and demonstrations. The average of these similarities is utilized as the similarity metric.\n\n[2] Liu Y, Ott M, Goyal N, et al. Roberta: A robustly optimized Bert pretraining approach[J]. arXiv preprint arXiv:1907.11692, 2019.\n\n[3] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//ICML. PMLR, 2021: 8748-8763.\n\n**Q5. Cumulative error.**\n\nThanks for your constructive suggestion! **Our VLMimic demonstrates strong robustness against the cumulative error.** Validation is conducted on the simulation manipulation tasks to assess our system's resilience to cumulative errors. We inject errors at the input of skill learning, adapting, and execution phases, corresponding to interaction grounding errors, knowledge retrieval errors, and execution errors, respectively:\n\n- **Interaction grounding errors**. Gaussian noise ($\\sigma = 5cm$ for the position, $\\sigma = 5^{\\circ}$ for rotation) is applied to pose estimation results from human videos, referring to the Megapose evaluation metrics [4]. These metrics quantify prediction accuracy based on the percentage of estimates within $5^{\\circ}$ rotational and $5cm$ translational thresholds from the ground truth.\n- **Knowledge retrieval errors**. A knowledge base is constructed using RLBench experimental learning data. During testing, relevant knowledge is retrieved from this knowledge base. To introduce errors, the most similar matches among all 12 matches are deliberately omitted. \n- **Execution errors**.  Gaussian noise ($\\sigma = 5cm$ for position, $\\sigma = 5^{\\circ}$ for rotation) is applied to object pose estimation results. \n\n|VLMimic|Interaction grounding errors|Knowledge retrieval errors|Execution errors|\n|-|-|-|-|\n|$0.76 (\\pm 0.17)$|$0.68 (\\pm 0.21)$|$0.64 (\\pm 0.21)$|$0.73 (\\pm 0.17)$|\n\nFurthermore, several illustrative examples of robustness against perturbation and failure reasoning are provided in Appendix E and K of the paper.\n\n[4] Labbé Y, Manuelli L, Mousavian A, et al. MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare[C]//CoRL 2022-Conference on Robot Learning. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Adayo7q8FB", "reviewer": "Reviewer_aTuD", "summary": "The paper proposes a system that can perform imitation learning based on human demonstration videos. The authors design a system using many pre-trained components like VLMs and hand and object tracker and 3D reconstruction and pose estimation tools. They show that by combining these existing tools they can achieve compelling success rates on multiple robots.", "questions": "1. Which models are actually used for the system? The authors mention multiple models many times in the paper: VFM [34; 35; 36] , SAM-Track [37; 38; 39; 40; 41]. Which of these are actually used?\n2. What is the knowledge bank used in the paper? Examples of how it is constructed and accessed?\n3. How many times is each method run during evaluation? How much scene randomization is done between these episodes and how different are they from the training demonstrations?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Impressive performance across real robots and multiple tasks.\n2. Ablation studies showing importance of different components. \n3. Detailed analysis of runtime of each component.", "weaknesses": "1. The paper and the abstract give a lot of credit to \"VLMs\". However there are a lot of other learnt modules like SAM-Track, BundleSDF and FoundationPose in the system. A lot of hand-designed priors like 3D projection, grasp prediction etc are also baked into the system. These should also be given due credit along with VLMs.\n\n2. The paper seems to be more of a robotics systems paper where many modules are connected to perform a task than a machine learning paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Give a lot of credit to VLMs**\n\nThanks for your valuable suggestions! We give credit to VLMs since the majority of **other foundation models serve as data generators** within the human-object interaction grounding module, providing motion data for VLMs to analyze, while **the acquisition and generalization of skills are mainly achieved through VLMs**, enabling the acquisition of robust and generalizable skills from limited demonstrations. \n\n**We acknowledge the important role of these foundation models in our framework**. Specifically, we will revise our manuscript to emphasize:\n- The utilization of foundation models for direct learning of fine-grained actions.\n- The critical function of foundation models in estimating object-centric actions for VLM analysis.\n\n**Q2. The paper seems to be more of a robotics systems paper.**\n\nThanks for your feedback! We have indeed submitted our paper on the topic of robotics. Our research primarily focuses on **leveraging machine learning approaches to enhance embodied intelligence in robotic systems**.\n\nOur paper presents a novel approach that leverages the capabilities of VLMs, aiming to **address a crucial challenge in machine learning: the acquisition of robust, generalizable skills from limited demonstrations**. Furthermore, our study represents **a pioneering achievement of VLMs in fine-grained motion skill learning**, demonstrating its ability to comprehend real-world interactions. To achieve this, we propose a novel framework incorporating hierarchical constraint representations and an iterative comparison methodology to enhance the reasoning capabilities of VLMs. Our approach leverages the advanced capabilities of foundation models, aligning with the prevailing paradigm in machine learning research that seeks to harness these advanced models for the augmentation of diverse task performance.\n\nFinally, we would like to clarify that the implemented models, primarily within the human-object interaction grounding module, function as a data generator. **This multi-module approach for robust, automated data generation is consistent with practices in many machine learning papers.**\n\n**Q3. Which models are used for the system?**\n\nThanks for your valuable feedback! The cited VFM models are capable of executing our tasks, with the employed model detailed in the Appendix's Implementation Details section. To enhance clarity, we will append a statement to the mentioned paragraph specifying that Tokenize Anything [5] is utilized in our experiments.\n\nConcerning SAM-Track [6], we adhere to the authors' guidelines on GitHub and cite these works. To reduce ambiguity, we will exclusively cite the SAM-Track paper.\n\n[5] Pan T, Tang L, Wang X, et al. Tokenize anything via prompting[J]. arXiv preprint arXiv:2312.09128, 2023.\n\n[6] Cheng Y, Li L, Xu Y, et al. Segment and track anything[J]. arXiv preprint arXiv:2305.06558, 2023.\n\n**Q4. Knowledge bank and its examples.**\n\nThe knowledge bank functions as a repository for archiving both high-level planning and low-level skill insights. Upon encountering novel environments, relevant knowledge is retrieved from this repository. The intuitive example is presented in Figure 8 of the attached PDF.\n\n**Construction.** A knowledge bank is established to archive both high-level planning and low-level skill insights, storing knowledge with key-value pairs. High-level planning knowledge is indexed using task description as keys, paired with the consequent action sequence as values. For low-level skill knowledge, keys are constituted by the object images and subtask description, and values comprise reconstructed objects, as well as semantic constraints and geometric constraints representing learned skills. \n\n**Access.** (1) High-level knowledge retrieval: The text encoder [7] is leveraged to obtain the similarity between the queried task description and the task descriptions associated with stored demonstrations. Then, the demonstration with the highest similarity is retrieved. (2) Low-level skill knowledge retrieval: Based on the queried subtask description, we leverage the text encoder [7] to select $N_{t}=2$ demonstrations with the highest similarity between their associated subtasks and the query. Subsequently, from the selected demonstrations, those exhibiting the highest degree of object similarity are retrieved. The image similarities based on the CLIP model [8] are calculated for both master and slave objects between the current scenarios and demonstrations. The average of these similarities is utilized as the similarity metric.\n\n[7] Liu Y, et al. Roberta: A robustly optimized Bert pretraining approach[J]. arXiv preprint arXiv:1907.11692, 2019.\n\n[8] Radford A, et al. Learning transferable visual models from natural language supervision[C]//ICML. PMLR, 2021: 8748-8763.\n\n**Q5. Number of evaluations and scene randomization.**\n\n**Number of evaluations**. The success rate in the RLBench simulation environment is calculated as the mean of 100 independent trials. For real-world experiments, following Voxposer [9], the success rate is determined by computing the average outcome of 10 discrete tests.\n\n**Scene randomization**. The position and orientation of objects are randomly determined in both the RLBench simulation and real-world experimental setups. Specifically, the spatial coordinates of objects are randomly sampled within the operational range of the robotic manipulator on the task workspace. Concurrently, the rotational orientation of objects is randomly selected from a predetermined set of feasible angular configurations (such as $[-\\pi/2,\\pi/2]$ for the $Z$-axis of the microwave). Additionally, within the RLBench simulation framework, the color attributes of the objects are subjected to probabilistic sampling. \n\n[9] Huang W, Wang C, Zhang R, et al. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models[C]//NeurIPS 2023 Foundation Models for Decision Making Workshop.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bOKe909gH7", "reviewer": "Reviewer_ovGy", "summary": "This work introduces VLMimic, a framework to acquire robotic skills by imitating from videos.\nThe system uses VL foundation models to ground human-object interaction videos.\nHierarchical representations are used to learn robotic skills and recorded in knowledge bank.\nIn unseen environments, the skill adapter iteratively compares to transfer the skill.\nIn extensive experiments, VLMimic outperforms baselines on the RLBench, as well as comprehensive results in real-world manipulation tasks.", "questions": "I am interested in real-world failure cases of your system, in the cases that can't be corrected by failure reasoning. Which is the main challenge, high-level or low-level?", "limitations": "Discussed in Appendix B.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "It's a strong work.\n1. VLMimic not only learns high-level action planning but also learns fine-grained low-level actions, a feature not possessed by past works.\n2. The proposed approach looks novel and intuitive.\n3. The experiments are impressive. VLMimic outperforms on RLBench simulation. Real-world experiments are conducted to solve long-horizon tasks. Qualitative videos are provided.", "weaknesses": "major:\n1. Can you predict the system scalability? What's the memory cost of the knowledge bank and the speed cost of iterative comparisons when scaling up? Is it possible to process scaled datasets like Ego-4D?\n\nminor:\n1. Some definitions in Line 151 are vague to me. What's the keypoint value? What's the visualized interaction? How do you do the rendering? I suggest some revisions in this part.\n2. Inconsistent descriptions: 'high level' and 'high-level'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. System scalability.**\n\nWe greatly appreciate your constructive suggestions! The computational time and memory requirements are estimated as follows:\n\n**Computational time of iterative comparison**. The majority of computational time in iterative comparison is allocated to awaiting responses from VLMs, accessed via online APIs. In the context of increasing video content, computational efficiency can be maintained by enhancing the rate of parallel API requests, thereby inherently exhibiting favorable scaling properties. \n\nWe posit that your primary concern may lie in the time required for skill acquisition, rather than only iterative comparison. Therefore, we estimate the computational time for the entire skill acquisition process.\n\n**Computational time of skill acquisition**. Our approach achieves an average learning time of $443s$ on real-world manipulation tasks, utilizing video sequences of $9s$ on average, captured at 30 Hz. To enhance scalability, we implement the following simple but efficient strategies without compromising accuracy:\n\n- The VLMs are accessed via online APIs, obviating the need for local computational resources. Therefore, we utilize multi-threading to process subsequent videos concurrently with VLM operations, reducing the learning time from $443s$ to $75s$.\n- We decrease the video frame rate from 30 fps to 5 fps, reducing learning time from $75s$ to $12s$.\n- We employ distributed processing by allocating distinct video sequences to separate GPUs, achieving a 9-fold acceleration with 10 GPUs, reducing averaged learning time to $1.4s$.\n\nConsequently, the computational time for learning skills from the 120-hour subset of Ego4D is estimated to be $120 \\times 1.4 / 9 = 18.6h$. Furthermore, acceleration can be further achieved by eliminating invalid video clips.\n\n**Memory cost**. The average memory allocation per skill is approximately 5MB. Given that the average temporal duration of skills is $9s$, the estimated number of skills in Ego-4D is $120 \\times 60 \\times 60 / 9 = 48000$, and the memory cost is $48000 \\times 5/1024 = 234.3GB$.\n\nOverall, our method exhibits promising scaling properties, with an estimated computational time of less than $19h$ and a required storage capacity below $240 GB$.\n\n**Q2. Some definitions in Line 151.**\n\nThanks for your valuable feedback! We will respond to your questions sequentially.\n\n**Keypoint value**. keypoint values denote critical interaction information. In the grasping phase, the keypoints are represented by the grasp poses. In the manipulation phase, keypoints represent a compressed object-centric trajectory, derived through a uniform sampling of 10 points along the estimated trajectory.\n\n**Visualized interactions**. Visualized interactions comprise a series of images that illustrate object grasping poses and manipulation trajectories. For the grasping phase, the visualized interactions comprise the object and multiple grippers. These grippers are displayed according to the object-centric grasp poses. For the manipulation phase, the visualized interactions consist of the master object and the estimated master object-centric motion trajectories of slave objects. We recall that the agent employs a slave object to interact with a master object in the manipulation phase.\n\n**Rendering pipeline**. In the grasping phase, we position the object at the origin of the 3D coordinate system, then simplified grippers are placed by the estimated object-centric grasping poses. Following this, this 3D scene undergoes projection onto 2D images along the X, Y, and Z directions, obtaining visualized interactions for the grasping phase. For the manipulation phase, the master object is positioned at the origin of the 3D coordinate system. Subsequently, the 3D trajectory is generated in concordance with the estimated master object-centric slave pose trajectory. The visualized interactions for the manipulation phase are obtained by projecting this scene into 2D images along the X, Y, and Z directions.\n\n**Q3. Inconsistent descriptions.**\n\nThanks for your valuable feedback! We will make the revisions to ensure consistency and conduct a comprehensive review of the entire article to identify and rectify any potential inconsistencies.\n\n\n**Q4. Real-world failure cases.**\n\nInsightful question! Figure 6 is presented in the attached PDF to illustrate situations challenging to resolve through failure reasoning, including:\n- The task execution may exceed the hardware limitations of the physical robot, inducing **inverse kinematics (IK) errors**. \n- Incomplete environmental perception increases the risk of **obstacle collisions**, leading to task failure. \n\nThe training datasets for VLMs exhibit a significant lack of data related to robot dynamics. Consequently, these models exhibit a limited capacity for error analysis and struggle to infer correction strategies when confronted with these failures.\n\n\n**Q5. Which is the main challenge?**\n\nExcellent question! We believe that **low-level challenges are more significant**, due to several factors:\n- **Low-level planning is more complex compared to high-level planning.** (1) Low-level planning requires **detailed operational specifications**, like grasping poses and motion trajectories, while high-level planning only focuses on overall task objectives. (2) Low-level planning must account for **diverse physical constraints**, such as hinge mechanisms of microwave doors. Conversely, high-level planning abstracts these physical interactions. (3) Low-level planning needs to **handle uncertainties** such as sensor noise, while high-level planning assumes an idealized environment.\n- **LLMs/VLMs exhibit superior proficiency in high-level planning**. High-level planning focuses on abstract strategic formulation, which aligns with the training corpus of VLMs. Conversely, low-level planning necessitates reasoning about fine-grained interactions, which is scarce in VLMs' training corpora, resulting in a capability deficit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We are glad to find that reviewers generally recognized our contributions:
-  **Method Development**. Presenting an innovative and intuitive approach, introducing a novel application of VLMs [ovGy, Y6C5]. The perception module is well-designed, with the capacity to acquire skills from observed trajectories being noteworthy [Y6C5].
-  **Comprehensive experiments**. Demonstrating impressive performance across substantial real robots and simulation tasks [Y6C5, xpmL, aTuD, ovGy]. Comprehensive ablation studies are conducted to evaluate various design choices [Y6C5, aTuD].
-  **Writing**. Well-organized and easy to follow [xpmL, Y6C5].

We also thank all reviewers for their insightful and constructive suggestions, which help a lot in further improving our paper. In addition to the pointwise responses below, we summarize supporting experiments and illustrative figures added in the rebuttal according to reviewers’ suggestions.

**New experiments**.

- **Precise manipulation tasks** [Y6C5, xpmL]. Our method demonstrates strong performance on the Functional Manipulation Benchmark (FMB) [1], which encompasses the grasping and reorienting that utilizes the fixture, and high-precision object insertion. Experimental results indicate that our method significantly outperforms CaP. Moreover, our method achieves superior performance to DP, requiring only five human videos compared to DP's 100 robot demonstrations.
- **Robustness against viewpoint variance** [Y6C5]. Experiments are conducted in real-world unseen environments, utilizing four distinct viewpoint configurations. Results demonstrate the significant resilience of VLMimic to viewpoint changes.

**New illustrative figures** in the attached PDF.
- Examples of **VLMimic in complex FMB**, which demands precise manipulation.
- Comparative illustration of **fine-grained motion skill acquisition** between VLMimic and VLM-based planning approaches.
- Examples of **important object reasoning**.
- **Multi-choice QA examples** for grasping constraint reasoning.
- **Failure cases** of VLMimic.
- Illustration of **data generation role** of human-object interaction grounding.
- Visualization of **knowledge bank construction and retrieval**.
- Illustration of **grasping and manipulation phase identification**.


[1] Luo J, Xu C, Liu F, et al. FMB: A functional manipulation benchmark for generalizable robotic learning[J]. arXiv preprint arXiv:2401.08553, 2024.

Meta-review: The paper introduces VLMimic, an approach for robotic imitation learning using vision-language models. The method demonstrates strong performance and generalization from few demonstrations, earning recommendation for acceptance from all reviewers.
Reviewers consistently praised the strong performance across diverse tasks, the ability to learn from few demonstrations, comprehensive experiments, and the innovative use of VLMs for fine-grained skill learning. These strengths were evident in both simulation and real-world settings, with VLMimic significantly outperforming baselines.
Initial concerns focused on the complexity of the pipeline, scalability questions, and lack of clarity on technical details. The authors provided thorough responses, clarifying the roles of different components, demonstrating promising scalability, and offering additional implementation details. These responses satisfactorily addressed most reviewer concerns, with two reviewers explicitly improving their scores post-rebuttal.
While some reservations remain about the ad-hoc nature of the design, the overall consensus is that VLMimic represents a valuable contribution to the field. The paper's comprehensive evaluations and strong results outweigh its limitations.
For the final version, I encourage the authors to incorporate their rebuttal explanations, particularly regarding component roles, scalability analysis, and real-world failure cases. Additionally, improving consistency in terminology and addressing the potential for a more generalizable framework would further strengthen the paper.
Decision: Accept (poster)