VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions

Authors: guangyan  chen
meiling  wang
te  cui
yao  mu
haoyang  lu
tianxing  zhou
zicai  peng
mengxiao  hu
haizhou  li
li  yuan
yi  yang
yufeng  yue
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 05:44:10

Abstract: Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable performance in vision and language reasoning capabilities for VIL tasks. Despite the progress, current VIL methods naively employ VLMs to learn high-level plans from human videos, relying on pre-defined motion primitives for executing physical interactions, which remains a major bottleneck. In this work, we present VLMimic, a novel paradigm that harnesses VLMs to directly learn even fine-grained action levels, only given a limited number of human videos. Specifically, VLMimic first grounds object-centric movements from human videos, and learns skills using hierarchical constraint representations, facilitating the derivation of skills with fine-grained action levels from limited human videos. These skills are refined and updated through an iterative comparison strategy, enabling efficient adaptation to unseen environments. Our extensive experiments exhibit that our VLMimic, using only 5 human videos, yields significant improvements of over 27% and 21% in RLBench and real-world manipulation tasks, and surpasses baselines by more than 37% in long-horizon tasks. Code and videos are available on our anonymous homepage.

Main paper: 
 introduction: 
 Visual Imitation Learning (VIL) has demonstrated remarkable efficacy in addressing various visual control tasks within intricate environments [1; 2; 3; 4; 5; 6; 7; 8; 9; 10]. Diverging from conventional approaches reliant on precise robot action labels, which often necessitates substantial human effort for data collection. Researchers increasingly turn to learning from human-object interaction videos that are easily accessible to reduce high data requirements.Existing methods for skill acquisition leveraging video data can be broadly categorized into two classes. One typical approach learns efficient visual representations for robotic manipulation through self-supervised learning from large volumes of videos [11; 12; 13; 14; 15; 16; 17; 18; 19; 20? ]. Another approach focuses on learning task-relevant priors to guide robot behaviors or derive a heuristic reward function for reinforcement learning [21; 14; 21; 22; 23; 24; 25; 26; 27; 28; 29] . Based on the above analysis, we present VLMimic, an approach that employs VLMs to directly learn even fine-grained action levels from a limited number of human videos, and generalize to novel scenes. As shown in Fig. 1 , our method parses videos into multiple segments and captures object-centric movements using the human-object interaction grounding module. Then, a skill learner employing hierarchical constraint representations extracts knowledge from estimated motions, deriving skills with fine-grained actions. In unseen environments, a skill adapter with an iterative comparison strategy revises and updates the learned skills based on observations and task instructions. Extensive experiments demonstrate that VLMimic achieves strong performance across various scenes, utilizing only 5 human videos without requiring additional training.Our main contributions can be summarized as follows: (I) We propose VLMimic, a novel visual imitation learning framework empowered by VLMs, to learn generalizable robotic skills from human demonstration videos. VLMimic features a skill learner for knowledge extraction and a skill adapter for iterative skill refinement, enabling efficient skill acquisition and adaptation. (II) We build an effective human-object interaction grounding algorithm to enhance fine-grained action recognition capabilities, and propose hierarchical constraint representations for VLM reasoning to reduce information redundancy and facilitate comprehensive action comprehension. (III) Our method outperforms other methods by over 27% on the RLBench. In real-world manipulation tasks, VLMimic achieves an improvement exceeding 21% in seen environments and 34% in unseen environments. Moreover, VLMimic exhibits an improvement of over 37% in long-horizon tasks.2 Related Work 
learning from human videos: 
 Conventional learning approaches necessitate access to expert demonstrations, which include observations and precise actions for each timestep. Drawing on human capabilities, learning from observation offers efficient and intuitive methods for robots to develop new skills. A plethora of recent researches explore leveraging large-scale human video data to improve robot policy learning [11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 41] . Representative methods, R3M [13] and MVP [12] , which employ the internet-scale Ego4D dataset [11] to pretrain visual representations for subsequent imitation learning tasks. Another thread of work [21; 22; 23; 24; 25; 26; 27; 28; 29] focuses on learning task-relevant priors from videos to guide robot behaviors or derive a heuristic reward function for reinforcement learning. Learning by watching [27] learns human-to-robot translation, the resulting representations are used to guide robots to learn robotic skills. WHIRL [21] infers trajectories and interaction details to establish a prior, but it learns policy through real-world exploration and requires a large number of rollouts to converge. GraphIRL [24] performs graph abstraction on the videos followed by temporal matching to measure the task progress, and a dense reward function is employed to train reinforcement learning algorithms. Despite these advancements, acquiring generalizable skills efficiently from limited demonstration videos remains highly challenging.Motivated by the notable success of VLMs across various domains, recent research [32; 33; 34; 35; 36; 37] investigate their potential in VIL. GPT-4V for Robotics [33] analyzes videos of humans performing tasks and outputs robot programs that incorporate insights into affordances. Digknow [32] distills generalizable knowledge with a hierarchical structure, enabling the effective generalization to novel scenes. Demo2code [37] generates robot task code from demonstrations via an extended chainof-thought and defines a common latent specification to connect the two. VLaMP [34] predicts visual planning from videos through video action segmentation and forecasting, handling long video history and complex action dependencies. However, these approaches often rely on predefined movement primitives or pre-trained skills to execute lower-level actions, thereby only partially solving the control stack. In contrast, our investigation aims to push these boundaries and learn all lower-level actions for the robot, eliminating the reliance on predefined primitives. 
vlmimic: 
 Considering video demonstrations V of a human performing manipulation tasks, recorded using an RGB-D camera. The overall pipeline of VLMimic is illustrated in Fig. 2 . Our method first grounds human videos, segmenting them into subtask intervals {τ i } V i=1 and capturing object-centric interactions I. A skill learner with hierarchical representations then extracts knowledge from the obtained interactions, deriving skills with fine-grained actions. In unseen environments, a skill adapter employs an iterative comparison strategy to revise and update the learned skills based on observations and task instructions.Despite VLMs demonstrating proficiency in various vision tasks, they still struggle with fine-grained action recognition within videos. To mitigate this limitation, a four-stage process, illustrated in Fig. 3 , is utilized to extract object-centric interactions for skill learning, transforming this intricate problem into pattern reasoning problems, typically more tractable for existing VLMs.Task recognition. Keyframes K are intermittently extracted from videos V, vision foundation models VFM [42; 43; 44] are utilized to detect objects within these frames. Utilizing keyframes K and textual detection results T d , VLMs are instructed to transcribe videos into task instructions T t , and compile the task-related objects T o into textual information. The object information is predicated on the initial frame of the video data, comprising a list of object names and their spatial relationships. The task recognition procedure is formulated as follows:EQUATIONVideo parsing. Videos are parsed into segments {τ i } V i=1 , using interaction markers that identify interaction periods. SAM-Track [45; 46; 47; 48; 49] predicts hand and task-related object masks for each frame, and corresponding point clouds P are generated through back-projection. Markers are then identified by determining the interaction start time t i and end time t e , partitioning videos V into multiple segments. Segments with hand motion trajectory lengths below than γ are filtered out, yielding final set of segments {τ i } V i=1 . Concretely, the interaction markers are obtained as follows:EQUATIONwhere function dist calculates the distance between any two point clouds.Subtask recognition. Each segment τ i is analyzed by VLMs, which generate a subtask textual description T τi , and categorize the segment into grasping or manipulation phases based on the interacting entities and T τi . VLMs also identify master objects O m and slave objects O s . In the grasping phase, the agent performs a reach-and-grasp action targeting O m , designating the hand as O s . In the manipulation phase, the agent employs O s to interact with O m .Object-centric interaction extraction. FrankMocap [50] and the Iterative Closest Point (ICP) algorithm [51; 52] are employed to derive precise hand pose trajectories, which are subsequently converted into robot gripper pose trajectories. Furthermore, BundleSDF [53] is employed for object reconstruction, and FoundationPose [54] is leveraged for object pose estimation based on reconstructed objects O. In grasping phases, interactions I are represented as grasp poses at handobject contacts. For manipulation phases, I are defined as trajectories of slave objects O s relative to A straightforward approach for learning skills involves directly discerning the numerical trajectory patterns [55; 56] . However, VLMs face challenges in reasoning about inherently redundant motion signals, limiting their ability to extract valuable information. To reduce redundancy and foster comprehensive comprehension, hierarchical constraint representations are proposed for skill learning, as illustrated in Fig. 2 . These representations exhibit semantic constraints via visualized interaction I v and further detail the fine-grained geometric constraints by integrating keypoint values V k .Learning with hierarchical constraint representations. Rendering interaction I, and textual notations T n on objects O to derive visualized interaction I v , VLMimic facilitates reasoning capabilities to analyze semantic constraints Φ s by encouraging VLMs to attend to objects and their related actions, and integrating keypoint values V k and object properties P o (e.g., 3-D bounding boxes) to derive geometric constraints Φ g . Formally, constraints are learned as follows:EQUATIONwhere S l , and G l are functions to learn semantic, and geometric constraints, respectively.(I) Grasping constraints. Inspired by task space regions (TSRs) [57] , the grasping constraint Φ g can be approximated as a series of bounded regions {R i } N C i=1 . Interactive grasp poses I are exhibited on objects, each associated with an index notation T n . These visualized interactions I v are presented to VLMs, leveraging their inherent knowledge and visual understanding ability to summarize semantic constraints Φ s and group these poses. Geometric constraints Φ g , represented as bounded regions, are derived by calculating ranges of grasp pose values V k within the same group, and associating them with object properties P o . This approach simplifies the complex task of constraint region generation into a series of visual understanding based multiple-choice question answering. Moreover, representing constraints through object properties enhances generalization across objects.(II) Manipulation constraints. Interaction trajectory I is delineated on the master object O m , incorporating keypoints V k in the textual prompt. Semantic constraints Φ s are identified by VLMs based on the visualized interaction I v and subtask description T τi . Geometric constraints Φ g are then formulated based on semantic constraints Φ s , keypoint values V k , and object properties P o , expressing Φ g via the trajectory code. The code comprises two components: parameter estimation functions f p , which derives trajectory parameters from object properties, and trajectory generation functions f s , employing estimated parameters to generate a sequence of slave object poses relative to the master object, promoting effective generalization across various objects and spatial configurations.During execution, grasp candidates are uniformly sampled within the learned grasping constraints, and object-centric trajectories predicted from manipulation constraints are converted to end-effector trajectories in the world frame using each grasp candidate of the slave object and object poses of master and slave objects. The resulting end-effector trajectory candidates are evaluated using motion planner, such as OMPL [58] , the trajectory with the highest fraction is selected.A knowledge bank B is established to archive both high-level planning and low-level skill insights, storing knowledge with key-value pairs (k i , v i ). High-level planning knowledge is indexed using task description T t as keys, paired with the consequent action sequence T τ as values. For low-level skill knowledge, keys are constituted by the object images and subtask description T τi , and values comprise reconstructed objects, as well as semantic constraints Φ s and geometric constraints Φ g representing learned skills.Even though the skill learner exhibits efficient skill acquisition, the demonstration and execution scenes may differ in objects and tasks, impeding direct skill transfer to unseen environments. To mitigate these challenges, VLMs are instructed to adapt skills via an iterative comparison strategy, as depicted in Fig. 2 . This approach updates learned skills by iteratively contrasting with the demonstrated knowledge, thereby enabling effective adaptation of retrieved skills to novel scenes.High level planning. High-level planning knowledge T τ is retrieved from knowledge bank B based on the task instruction, which acts as the in-context example for VLMs, along with the scene observation. VLMs serve as a physically-grounded task planner [59; 60] , generating a sequence of actionable steps and descriptions of task-related objects T o .Iterative comparison. In each iteration, VLMs perform a comparative analysis between the adapted interaction I and retrieved interaction Î, subsequently updating the skill constraints Φ s and Φ g . This iterative process persists until either convergence is achieved or the maximum number of iterations N I is reached. This approach facilitates reasoning in VLMs by directing their attention to discrepancies, and enables VLMs to pinpoint the best available solution through an iterative process. The adapting procedure at the i-th iteration can be formally represented as:EQUATIONwhere Φg and Φs denote referential constraints, extracted from the knowledge base. The functions S a and G a adapt semantic and geometric constraints, respectively.(I) Grasping constraint adaptation. As the grasping orientation is typically derivable from position constraints using grasping models [61; 62; 63] , our work focuses on transferring position constraints. The visualized grasping position space is discretized into an m × n grid (m, n ∈ Z) and annotated with textual notations T n , obtaining I 0 v . VLMs are instructed to update semantic constraints Φ s , by contrasting with the referential interaction Îv and semantic constraints Φs , and to adapt geometric constraints Φ g by sampling K outputs of grasping region selection. The updated Φ g are then visualized for the next iteration. The 3-D positional region is represented using two perspectives, and the consistency of the selected regions for the overlapping area validates the VLM outputs. The obtained constraints are expressed via object properties to enhance generalization.(II) Manipulation constraint adaptation. VLMs are instructed to iteratively self-summarise and update manipulation constraints based on the task instruction and scene differences. VLMimic generates trajectories adhering to geometric constraints Φ g , which are exhibited on master objects. VLMs are instructed to analyze the deviation of the adapted interaction I v from the referential interaction Îv to revise semantic constraints Φ s , and geometric constraints Φ g undergo refinement predicated on the updated Φ s , along with trajectory keypoint values V k and object properties P o .Failure reasoning. Despite the ability of VLMs to generate effective constraints, environmental noise, such as trajectory estimation errors, impedes successful task execution. Thus, we leverage VLMs to detect and address failures during execution by providing them with perceptual results, such as object pose and robot end-effector trajectories, enabling autonomous failure identification and reasoning for rectification.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: Our code and data will be made pubilcly accessible.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 Baselines. VLMimic is compared with five representative methods: (1) R3M-DP that utilizes the pre-trained R3M visual representation [13] with the state-of-the-art (SOTA) diffusion policy [7] ; (2) Diffusion Policy (DP) [7] , a SOTA end-to-end policy method; (3) GraphIRL [24] , a method that employs graph abstraction and learns reward functions for reinforcement learning (RL); (4) Code as Policy (CaP) [64] , an LLM-driven method that re-composes API calls to generate new policy code; and (5) Demo2code [37] , an LLM-driven planner method that translates demonstrations into task code. We modify it to integrate the analysis results from GPT-4V for Robotics [33] , enabling it to transcribe videos into code. R3M-DP and DP are trained using the robot demonstrations with paired observation and action sequences. GraphIRL is trained in simulators with paired robot videos, Demo2code and our method learns skills with human videos in real-world experiments and robot videos in simulation experiments.Experimental setup. To assess our approach on challenging robotic manipulation tasks, the RLBench [65] benchmark is utilized for simulation tasks. Due to the unavailability of human videos in simulations, demo2code and our method utilize robot videos captured from a single-camera perspective during demonstrations, incorporating robot gripper trajectories.Results. We investigate the capacity of VLMimic to acquire skills from a limited collection of video demonstrations, without requiring additional training. Our evaluation encompasses 12 manipulation tasks, as detailed in Table 1 , demonstrating that our method surpasses all other methods in 11 out of these tasks. Our method, learned with only 5 human videos, obviously outperforms R3M-DP and DP by over 61% in overall performance, despite both being trained on 100 robot demonstrations.Compared to CaP and demo2code, our method demonstrates an improvement exceeding 27%, highlighting the significant performance enhancements facilitated by the VLMimic framework.Experimental setup. The real-world testing environment (E) is divided into "seen" (SE) and "unseen" (UE) categories. The "seen" category allows for testing in the environment where demonstrations were collected, whereas the "unseen" category involves testing in a distinct environment characterized by different objects and layouts. Success criteria are human-evaluated and the success rate is calculated from 10 randomized object positions and orientations.Results: To validate the effectiveness of VLMimic in real-world settings, we conduct experiments involving 14 challenging real-world manipulation tasks selected from recent robotics research [66; 67; 4; 68] . Quantitative results, presented in Table 2 , demonstrate that VLMimic clearly outperforms other methods across all tasks, particularly in the "unseen" environment (UE). VLMimic achieves an improvement exceeding 21% in SE and more than 34% in UE. Results reveal the outstanding ability of VLMimic to acquire skills from human videos and adapt them to unseen environments.Experimental setup. Since baseline methods struggle to complete long-horizon tasks in the UE setting, experiments are conducted in the SE setting. All other experimental settings are consistent with those in the real-world manipulation task.Results. The performance of VLMimic on long-horizon tasks is quantitatively evaluated by its successful completion of six distinct tasks, each comprising at least five subtasks. Experimental results, as depicted in Table 3 , obviously exhibit a substantial enhancement achieved by our method over baseline methods. These outcomes suggest that the proposed method is capable of developing robust skills, thereby achieving promising performance in even long-horizon tasks.The keypoint-centric representation approach enables our method to tolerate different observational perspectives. To demonstrate the robustness of our method to varying viewpoints. Experiments are conducted in real-world unseen environments, utilizing distinct viewpoints, as shown in Figure 4 , where the first angle serves as the default perspective used in our experiments. Experimental results shown in Table 4 prove that our method exhibits only a 7% fluctuation in performance under varying viewpoints, demonstrating the resilience of VLMimic to viewpoint changes. Figure 5 elucidates scenarios that present significant challenges for resolution through VLM reasoning. These scenarios encompass: (I) The task execution may exceed the hardware limitations of the physical robot, inducing inverse kinematics (IK) errors. (II) Incomplete environmental perception increases the risk of obstacle collisions, leading to task failure. Since the training datasets for VLMs exhibit a significant lack of data related to robot dynamics, these models lack associated knowledge, exhibiting a limited capacity for error analysis and struggling to infer correction strategies when confronted with these failures.Comprehensive ablation studies are conducted to investigate the fundamental designs of our VLMimic approach. The effects of these design decisions are assessed by measuring the success rate on realworld manipulation tasks, which is computed across 10 randomized object positions and orientations.Hierarchical constraint representations. Table 5 (a) analyzes three distinct constraint representations. Variants that exclusively reason semantic constraints or directly obtain geometric constraints without semantic analysis, lead to diminished performance. The results exhibit that hierarchical constraint representations enhance skill acquisition capabilities, demonstrating the pivotal role in facilitating the understanding and reasoning capabilities of VLMs.Grasping learning. Table 5 (b) presents variants and their respective performance. The first variant utilizes VLMs for direct prediction of constraint region values, resulting in a significant performance decline. The second variant employs the DBScan clustering algorithm to group grasp poses and derive constraints as bounded regions. However, this method only considers numerical distributions without incorporating grasping common sense, leading to performance degradation.Results indicate that our method attains high success rates on complex tasks with a single human video demonstration, and increasing the number of videos yields performance gains. The results show that our approach can efficiently learn generalizable skills from a limited number of human videos. We choose to use 5 demonstration videos to balance data availability and performance.Variants compare constraints exclusively utilizing either visualized interactions or keypoints exhibit decreased success rates. Visual comparison facilitates semantic contrast in VLM, while keypoint values provide fine-grained geometric information. Experimental results illustrate that our strategy facilitates reasoning for both semantic and geometric constraint adaptation.We conduct an analysis on the impact of iteration count in skill adapter and search for the optimal choice, as shown in Table 5 (e). Reducing the number of iterations to 0 results in Failure reasoning. The impact of failure reasoning is investigated in Table 5 (f). The success rate exhibits an upward trend with increasing iterations, reaching an elbow point at 2 iterations, providing an optimal trade-off between real-time performance and success rate. Failure reasoning proves crucial for tasks demanding high-precision manipulation, which are susceptible to environmental noise. It enhances both the success rate and the robot's ability to operate in intricate environments. 
conclusion: 
 In this paper, we present VLMimic, a novel approach that harnesses VLMs to learn skills with even fine-grained action levels from a limited number of human videos, and effectively generalize them to unseen environments. VLMimic first extracts object-centric interactions from human videos, and learns skills based on these interactions, using hierarchical constraint representations. In unseen environments, these skills are updated through an iterative comparison strategy. Extensive experiments conducted on various manipulation and challenging long-horizon tasks demonstrate the superior performance achieved by our VLMimic, utilizing only 5 human videos without requiring additional training, exhibiting strong skill acquisition and adaptation capabilities.Limitations. Despite the promising performance exhibited by VLMimic, current VLMs are still limited by inference latency and computational resource requirements. We believe that the progression of lightweight VLMs will mitigate these limitations. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: please refer to Sec. 4Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research presented in this paper adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We do not foresee obvious undesirable ethical or social impacts at this moment.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not involve the release of data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators and original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In human-object interaction grounding module, the Tokenize Anything [44] model is employed during task recognition to improve fine-grained scene understanding ability. The textual detection results are integrated using VLMs to generate concise task descriptions and detailed object information. The videos are segmented using a threshold ϵ of 2cm. Segments with hand motion trajectory lengths below γ = 10cm are discarded. During the grasping constraint learning phase, the number of regions N c is automatically determined by the VLMs. In manipulation constraint learning, keypoints are obtained by uniformly sampling 10 points. For the skill adapter, the maximum number of iterations is set to N I = 4. During grasping constraint adaptation, visualized grasping position space is discretized into a 5 × 5 grid, with K = 4 outputs sampled per iteration.During skill execution, the pretrained Grounded-segment-any-parts model [69; 70] is used to generate segmentation maps of queried objects or parts. These segmentation maps are then utilized to predict object-centric pose sequences using codes generated by VLMs. FoundationPose [54] is employed to track object poses, transforming the object-centric poses into the world frame. The robotic arm's motion planning is facilitated by the integration of the MoveIt module, renowned for its comprehensive motion planning capabilities, and the OMPL [58] (Open Motion Planning Library), which offers a suite of advanced algorithms for efficient path planning and obstacle avoidance. Upon action completion, the real-time object positions are used to assess task success until manual confirmation or a preset time is reached. In case of failure detection, object and gripper poses are employed for failure reasoning, where the gripper poses are estimated using the attatched QR scan.B.1 Baseline setup R3M-DP [13] and DP [7] employ a CNN-based network architecture for its robustness across diverse tasks. These methods are trained on robot demonstrations using default hyper-parameters, robot demonstrations consist of paired observation and action sequences.To ensure that GraphIRL [24] is trained and tested under the same scenario in the SE setting. GraphIRL is trained in the simulator with corresponding paired robot videos, and the SE results are obtained from the same simulated environments, while UE results are acquired from real-world scenarios under the UE setting. Since the original GraphIRL method struggle to learn the gripper switch information, we additionally provide this information to GraphIRL.Following Cap [64] , the primitives for Cap [64] and demo2code [37] include: move to pos, rotate by quat, set vel, open gripper, close gripper, pick obj, and place at pos. Cap employs natural language instruction directly for reasoning, Demo2code generates code from textual video analysis results provided by GPT-4V for Robotics [33] , a video analysis approach for robotics, enabling demo2code to learn from human videos. Specifically, the detailed task analysis results and affordance analysis outcomes from GPT-4V for Robotics are incorporated as contextual information within the textual prompt for demo2code.Experiments are conducted on Franka Emika [71] , employing three RGB-D cameras (ORBBEC Femto Bolt) for environment observation: one at the top right of the table, one at the top left, and one mounted on the robot's wrist. All cameras start recording and return real-time RGB-D observations at a frequency of 30 Hz. All experiments are evaluated on an Intel i7-10700 CPU with an RTX 3090 graphics card.The definition of our long-horizon tasks is listed below. For each task, the initial state and subgoals are pre-defined. The whole task is completed if and only if all subgoals are completed in the correct order.• Make coffee -Initial state: The coffee machine and capsules are placed on the tabletop, with the capsule chamber placed on the cup. -Criteria: place the coffee capsule into the capsule chamber, insert it into the coffee machine, place a cup under the coffee machine's dispenser, and finally turn on the coffee machine. -Subgoals: (1) Grasp the coffee capsule; (2) Place the coffee capsule in the capsule chamber;(3) Grasp the capsule chamber; (4) Insert the capsule chamber into the coffee machine; (5) Place the cup under the coffee machine's water outlet; (6) Grasp the lever; (7) Turn on the coffee machine. -Initial state: Bananas, mangoes, cups, and paint brushes are scattered on the table. In addition, there is a drawer, a plate, and a dust brush on the table. -Criteria: Put the fruits (banana and mango) back into the plate, and put the tools (cup and paint brush) back into the drawer, and sweep the tabletop with the dust brush. Justification: please refer to Sec. 5 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The paper introduces VLMimic, an approach for robotic imitation learning using vision-language models. The method demonstrates strong performance and generalization from few demonstrations, earning recommendation for acceptance from all reviewers.
Reviewers consistently praised the strong performance across diverse tasks, the ability to learn from few demonstrations, comprehensive experiments, and the innovative use of VLMs for fine-grained skill learning. These strengths were evident in both simulation and real-world settings, with VLMimic significantly outperforming baselines.
Initial concerns focused on the complexity of the pipeline, scalability questions, and lack of clarity on technical details. The authors provided thorough responses, clarifying the roles of different components, demonstrating promising scalability, and offering additional implementation details. These responses satisfactorily addressed most reviewer concerns, with two reviewers explicitly improving their scores post-rebuttal.
While some reservations remain about the ad-hoc nature of the design, the overall consensus is that VLMimic represents a valuable contribution to the field. The paper's comprehensive evaluations and strong results outweigh its limitations.
For the final version, I encourage the authors to incorporate their rebuttal explanations, particularly regarding component roles, scalability analysis, and real-world failure cases. Additionally, improving consistency in terminology and addressing the potential for a more generalizable framework would further strengthen the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to learn a policy from human videos utilizing advances in vision-language models. The authors first parse human videos into several keyframes, then detect and track hand-object interactions. From the segmented interactions, they learn low-level actions by following the 3D hand-object trajectories in each segment. Additionally, the authors propose an iterative comparison stage to adapt to new scenes or objects. The method achieves better performance compared to other visual imitation learning algorithms in both simulation and real-world evaluations.
strength_weakness: strength_weakness: While this paper is well-written and achieves good performance, I have a few concerns regarding some claims and evaluations of the proposed method.

First, the paper focuses on fine-grained action imitations. However, I’m not convinced that the skills demonstrated in the paper are fine-grained or distinct from previous work that uses VLM for planning. Specifically, the skills in this paper primarily involve grasping and moving along certain trajectories, without complex interactions (actions requiring precise manipulation such as insertion or grasping that utilizes the environment).

On the other hand, similar skills can also be learned using the 'VLM as Planner' framework. For example, the 'Code as Policy' approach can perform tasks such as writing with pens. This skill is quite similar to the 'Spread sauce' motion described. Therefore, the claim that 'Fine-grained motions' cannot be learned in the 'VLM as Planner' framework needs further explanation.

The proposed method seems to categorize motion skills as 'grasping' and 'manipulation'. If the point is that 'VLM as Planner' requires each skill to be defined manually while the proposed method can automate this process, it would be beneficial to clarify that the proposed method still defines motion skills, but in a broader or more generalizable manner.

I think the performance of the baseline imitation learning algorithms is surprisingly low. Given 100 demonstrations, R3M-DP and DP should be quite successful, at least in simpler tasks such as Reach Target. This assessment is based on the original paper where they achieve a high success rate in more difficult tasks, as well as my empirical experience. Could you clarify whether the evaluation protocol was changed to make the task much harder?
questions: questions: The proposed system is rather complicated. While I appreciate the authors' efforts in assembling such a complex system, it is honestly a bit difficult to understand. I have detailed a few technical questions below.
In the grounding module:
* How are the keyframes obtained?
* How does the language model identify the important object to detect (e.g., pie, plate, brush, and bowl)?
* How are candidate action primitives (such as grasping and manipulation) determined?
For skill learning:
* Can you provide some examples of the Multiple-choice QA?
* How are the keypoints obtained?

I appreciate the evaluation on unseen scenes and objects. However, how similar do the camera viewing angles need to be? I assume there must be some degree of robustness because the perception operates in 3D, but is there any evaluation of this aspect?

As mentioned in the previous section, the results for R3M-DP and DP are surprisingly low. Have you tested the implementation on other tasks (such as those reported in the original paper) to verify the correctness?
limitations: limitations: The authors report limitations in the supplementary material. The limitations section is comprehensive, though I would suggest moving it to the main text.
paper_summary: paper_summary: This paper introduces a novel paradigm named VLMimic, which employs Vision-Language Models (VLMs) and multiple vision tools to ground object-centric information from demonstrations into fine-grained actionable skill sets. VLMimic exhibits a remarkable success rate in manipulation tasks and demonstrates significant generalization capabilities, learning effectively from a minimal number of human and robot videos.
strength_weakness: strength_weakness: The proposed object-centric manipulation pipeline encompasses multiple foundation models / vision tools to generate various intermediate representations (object detection results, object 6D poses, hand mask and trajectories, and task description from VLMs). This integration relies heavily on human prior knowledge to string all these models, rendering the process lengthy and complex.  Moreover, this approach may only be adaptable to a limited set of interaction objects (for example, whose 'affordances',  or 'object properties' as proposed in the paper, can be easily defined and recognized).


*[Minor]* According to line167-168, the semantic constraints are identified based on both visual interaction $I_{v}$ and sub-task description $T_{\tau}$. But only  $I_{v}$ is incorporated in Equation 3.
questions: questions: 1. In the iterative comparison process, how does VLMimic determine which object, from the extensive knowledge bank of diverse objects, is utilized as the reference for guiding the adaptation of grasping and manipulation constraints?

2. Has the potential for cumulative error in the proposed modular design been taken into account? Specifically, is there a risk that an error at any stage of the process could ultimately lead to the failure of the manipulation task?
limitations: limitations: Possible limitations are discussed in the paper.
paper_summary: paper_summary: The paper proposes a system that can perform imitation learning based on human demonstration videos. The authors design a system using many pre-trained components like VLMs and hand and object tracker and 3D reconstruction and pose estimation tools. They show that by combining these existing tools they can achieve compelling success rates on multiple robots.
strength_weakness: strength_weakness: 1. The paper and the abstract give a lot of credit to "VLMs". However there are a lot of other learnt modules like SAM-Track, BundleSDF and FoundationPose in the system. A lot of hand-designed priors like 3D projection, grasp prediction etc are also baked into the system. These should also be given due credit along with VLMs.

2. The paper seems to be more of a robotics systems paper where many modules are connected to perform a task than a machine learning paper.
questions: questions: 1. Which models are actually used for the system? The authors mention multiple models many times in the paper: VFM [34; 35; 36] , SAM-Track [37; 38; 39; 40; 41]. Which of these are actually used?
2. What is the knowledge bank used in the paper? Examples of how it is constructed and accessed?
3. How many times is each method run during evaluation? How much scene randomization is done between these episodes and how different are they from the training demonstrations?
limitations: limitations: Yes
paper_summary: paper_summary: This work introduces VLMimic, a framework to acquire robotic skills by imitating from videos.
The system uses VL foundation models to ground human-object interaction videos.
Hierarchical representations are used to learn robotic skills and recorded in knowledge bank.
In unseen environments, the skill adapter iteratively compares to transfer the skill.
In extensive experiments, VLMimic outperforms baselines on the RLBench, as well as comprehensive results in real-world manipulation tasks.
strength_weakness: strength_weakness: major:
1. Can you predict the system scalability? What's the memory cost of the knowledge bank and the speed cost of iterative comparisons when scaling up? Is it possible to process scaled datasets like Ego-4D?

minor:
1. Some definitions in Line 151 are vague to me. What's the keypoint value? What's the visualized interaction? How do you do the rendering? I suggest some revisions in this part.
2. Inconsistent descriptions: 'high level' and 'high-level'
questions: questions: I am interested in real-world failure cases of your system, in the cases that can't be corrected by failure reasoning. Which is the main challenge, high-level or low-level?
limitations: limitations: Discussed in Appendix B.

Comments: title: Thanks for your feedback!
To reviewer Y6C5: Please respond to rebuttal
Thanks for your feedback!
Thanks for your feedback!
comment: # Summary of rebuttal
We extend our profound gratitude to the Area Chair for their dedicated efforts and to the reviewers for their comprehensive discussions, which have substantially improved the quality of our work. The recommendation for acceptance from all reviewers is particularly gratifying, especially the positive assessment of our method development, comprehensive experiments, and manuscript composition.

This paper is further enriched by additional empirical investigations, encompassing high-precision manipulation tasks and the robustness assessment of our method against variations in viewpoint. Furthermore, new illustrative figures are incorporated to enhance clarity and facilitate understanding for readers.

We reiterate our sincere appreciation to the Area Chair and reviewers for their invested time, diligent effort, and invaluable feedback, which are pivotal in refining our work.
Thanks for considering our responses and recommending acceptance! We will update our revised paper according to our discussions. Thanks again for your insightful and constructive suggestions for improving paper quality. We are happy to address any further questions or concerns about the work.
Hi reviewer Y6C5,

Thank you for your initial review. Please kindly respond to the rebuttal posted by the authors.
Does the rebuttal answer your questions/concerns? If not, why?

Best,
AC
The reviewer thanks authors' responses. My concerns are addressed. I keep my original rating.
Thanks for considering our responses and recommending acceptance! We will update our revised paper according to our discussions. Thanks again for your insightful and constructive suggestions for improving paper quality. We are happy to address any further questions or concerns about the work.
Thanks for the thorough rebuttal provided by the authors. Most of my concerns were addressed, but the method's ad-hoc design may still limit its impact to broader audiences. I’m maintaining my initial score.
Thanks for considering our responses and raising the score! We will update our revised paper according to our discussions. Thanks again for your insightful and constructive suggestions for improving paper quality. We are happy to address any further questions or concerns about the work.
Thanks for the clarifications. I have updated the score to weak accept. Please include the answers in the final version of the paper.
Dear Reviewer ovGy,

Thank you again for reviewing our manuscript. We have tried our best to address your questions (see our rebuttal in the top-level comment and above). Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer aTuD,

Thank you again for reviewing our manuscript. We have tried our best to address your questions (see our rebuttal in the top-level comment and above). Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer xpmL,

Thank you again for reviewing our manuscript. We have tried our best to address your questions (see our rebuttal in the top-level comment and above). Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer Y6C5,

Thank you again for reviewing our manuscript. We have tried our best to address your questions (see our rebuttal in the top-level comment and above). Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.