Title: OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: One-Shot Visual Imitation Learning

World Model

Robot Learning
Abstract: Visual imitation learning enables robotic agents to acquire skills by observing expert demonstration videos. In the one-shot setting, the agent generates a policy after observing a single expert demonstration without additional fine-tuning. Existing approaches typically train and evaluate on the same set of tasks, varying only object configurations, and struggle to generalize to unseen tasks with different semantic or structural requirements. While some recent methods attempt to address this, they exhibit low success rates on hard test tasks that, despite being visually similar to some training tasks, differ in context and require distinct responses. Additionally, most existing methods lack an explicit model of environment dynamics, limiting their ability to reason about future states. To address these limitations, we propose a novel framework for one-shot visual imitation learning via world-model-guided trajectory generation. Given an expert demonstration video and the agent’s initial observation, our method leverages a learned world model to predict a sequence of latent states and actions. This latent trajectory is then decoded into physical waypoints that guide the agent’s execution. Our method is evaluated on two simulated benchmarks and three real-world robotic platforms, where it consistently outperforms prior approaches, with over 30% improvement in some cases.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a one-shot visual imitation learning method for unseen robotic manipulation tasks. By taking a single expert demonstration as context information, this model generates a sequence of waypoints to complete novel tasks. To enhance the framework's ability for modeling dynamic environments, this paper introduces a world-model-guided trajectory generation proxy loss. The effectiveness of this method is validated through extensive experiments in both simulation and the real world.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. This paper provides sufficient real-world experiments to prove the effectiveness.\n2. This paper is well-organized, and the proposed method is clearly explained.\n3. It is impressive to imitate human videos for a novel manipulation task.\n\n## Weaknesses\n\n1. Although the experimental results indicate that incorporating the world model loss is beneficial, it remains unclear why this leads to improvements on unseen tasks. Training a model on a small dataset is prone to overfitting, which would be expected to hinder its generalization capabilities on novel tasks. More detailed qualitative and quantitative analyses of the world model loss should be provided.\n\n2. The similarity between training scenarios and novel validation scenarios would significantly influence the performance of one-shot imitation learning systems. However, this lacks a comprehensive analysis of this relationship.\n\nQUESTIONS:\n1. Why does the world model loss bring benefits to the one-shot imitation learning on novel scenarios? For example, the Human-Franka-PP task only contains 3 training manipulation tasks; the world model would overfit to these tasks, so why would it have better generalization ability for novel tasks?\n\n2. For dynamic world modeling, various methods exist to reconstruct subsequent frames. Could you provide a comparison with other techniques, like VQ-VAEs and diffusion models, to further assess how well the representation captures dynamic scenes?\n\n3. How do different types of similarity—such as semantic similarity and visual similarity—between the training dataset and validation tasks affect learning outcomes? Furthermore, the impact of various related tasks within the training dataset on overall performance remains underexplored.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OSVI-WM, a novel one-shot visual imitation learning framework based on world-model-guided trajectory generation, designed to address the limited generalization capability of existing methods on unseen tasks. OSVI-WM integrates expert demonstration videos with the agent's initial observation to predict sequences of latent states and actions through a learned world model, which are subsequently decoded into physical waypoints for task execution. Extensive experiments conducted across both simulated and real-world robotic platforms demonstrate the effectiveness of the proposed method, with results showing that OSVI-WM significantly outperforms existing approaches, particularly on \"hard\" tasks that exhibit semantic or structural differences. Furthermore, the framework incorporates waypoint re-planning to enhance execution robustness. \nThe key contributions of this work include: (1) an end-to-end architecture requiring no large-scale pretraining; (2) a dedicated world-model-guided trajectory generation module for one-shot visual imitation; (3) improved test-time robustness through re-planning; and (4) comprehensive experimental validation across diverse environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1.Novel Integration of World Models: OSVI-WM is the first to introduce a world model into one-shot visual imitation learning, establishing an explicit environment dynamics model. By predicting latent states and generating trajectories, it significantly enhances the model’s ability to reason about future states.\n2.Superior Generalization Performance: Experimental results demonstrate that OSVI-WM outperforms existing methods on unseen tasks in both simulation and real-world robotic platforms, achieving over 30% improvement in success rates on challenging tasks with semantic or structural variations.\n3.Strong Cross-Embodiment Adaptability: The method performs robustly across different embodiments (e.g., human-to-robot imitation) without requiring additional training data or large-scale pretraining, lowering the barrier for real-world deployment.\n4.Enhanced Robustness via Replanning: The waypoint re-planning mechanism dynamically adjusts the execution strategy upon failure, further improving task completion rates.\n\n\nWeaknesses：\n1.Lack of End-Effector Orientation Control: The framework's limitation to predicting only 3D positions and gripper states restricts its applicability in tasks that require precise end-effector orientation control. Many real-world manipulation tasks, such as screwing, drilling, or side-grasping, necessitate accurate orientation control.\n2.Limited Handling of Dynamic Environments: Although the world model aims to capture environment dynamics, its ability to handle highly dynamic or uncertain environments remains unexplored. In scenarios with multiple moving objects, sudden changes in the environment, or significant variations in object properties, the model's predictions may become less reliable,.\n3.Redundancy in Latent States: Although latent states are designed to be “compact” representations, the features extracted by the Res-Net encoder may contain a large amount of redundant information unrelated to the task (such as texture or lighting). This redundant information can interfere with the model's learning and prediction of key task features, reducing the model's efficiency and accuracy. Therefore, further research is needed on how to remove this redundant information from the latent states to improve the model's performance and generalization ability.\n4.Limited Theoretical Analysis: The paper lacks a rigorous theoretical analysis of the framework's properties.\n\nQUESTIONS:\n1.Generalization to Unseen Positions and Objects: The paper states that OSVI-WM can generalize to new tasks with different object positions and across different embodiments, but it does not provide a detailed explanation of how the model achieves this capability. For example, how does the model ensure the accuracy of action prediction when encountering object placements or new object categories that were not seen during training? Additionally, compared to baseline methods, does the robustness of OSVI-WM to changes in object positions stem from the dynamic modeling of the world model? Further clarification is needed.\n2.Visualization and Analysis of Latent States and Actions: The paper demonstrates the visualization of latent states through image reconstruction (Figure 9), but lacks an in-depth analysis of the visualization of latent actions and their relationship with states. For example, do latent actions correspond to the movements of the robotic arm and changes in the interacted objects? How do changes in latent states drive action generation? It is suggested to supplement the visualization and analyze the correlation between the two to enhance interpretability.\n3.The Significance and Efficiency of Replanning: Replanning is described as “generating new waypoints when the task is not completed,” which seems equivalent to repeated execution until success. If so, this strategy may reduce efficiency in practical applications due to multiple attempts. The triggering conditions for replanning (such as timeouts or deviation thresholds) and its comparison with the success rate of single planning need to be explained. For example, Figure 8 shows that replanning improves performance, but it does not quantify its time cost. If replanning is merely a repeated call to the same module, then its significance is limited.\n4.Redundancy in Latent States: Although latent states are designed to be “compact” representations, the features extracted by the Res-Net encoder may contain a large amount of background information unrelated to the task (such as texture or lighting). How can the world model focus on task-relevant features? Moreover, if there is redundancy in the latent space, will it reduce the efficiency of trajectory generation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OSVI-WM, a novel framework for one-shot visual imitation learning (OSVI) designed to generalize to unseen tasks. The core problem it addresses is that existing OSVI methods often fail when test tasks are semantically or structurally different from training tasks. Also, the agent and the expert have different embodiments and there is no direct alignment between their states or actions. The proposed solution is a model that, given a single expert demonstration video and the robot's initial observation, uses a learned action model (to infer the intention of the task shown by the expert and output a latent action) and a learned world model to plan a sequence of future latent agent states corresponding to the task. The sequence is then decoded into a series of physical waypoints for the robot's controller to imitate and execute. The entire framework, including the world model, is trained end-to-end on in-domain data without requiring massive external video pretraining. The authors demonstrate through experiments on simulated and real-world robotic platforms that OSVI-WM significantly outperforms prior methods, particularly on challenging, unseen tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles a crucial and difficult problem in robotics: one-shot generalization to new tasks from a single video. Moving beyond simple variations in object positions to tasks that are semantically distinct is a significant step towards more capable and flexible robots.\n2. The technical quality of the paper is high. The architecture is well-designed and the experimental evaluation is exceptionally thorough, including challenging human-to-robot imitation with embodiment mismatch. The comprehensive ablation studies are also convincing.\n3. The core idea lies in learning a meta-action-model to infer the semantics of the given task and output high-level latent actions, and integrating a world model to generate future latent trajectory plans given the actions and the expert video, which is then decoded it into low-level waypoints. This is a well-motivated approach.\n\nWeaknesses:\n1. The re-planning mechanism boosts performance on hard tasks, but it need manual assessment in real-world autonomy, requiring a human in the loop. An automated failure detection mechanism is needed for re-planning in real-world application.\n2. The paper's main claim is generalization to unseen tasks, but the \"unseen\" test tasks in are semantic or combinatorial variations of tasks seen during training. This is a valid and challenging setup, but it leaves open the question of how the model would perform on tasks that are truly out-of-domain (e.g., training on pick-place tasks and testing on a wiping or drawer-opening task). The generalization claim, while supported by the experiments, may be limited to new tasks within the same family of manipulations.\n\nQUESTIONS:\n1. The action model is an inverse dynamic model (IDM) conditioned on given tasks, and the world model is a corresponding forward dynamics model (FDM), in the context of latent action learning [1-3]. Could you elaborate on the benefit of this architecture, versus a single, larger model that directly predicts the next latent states $r_{i+1}$ from the history and the expert demonstration? Other design choices also include, directly learning a seq2seq-style connector, to translate the expert latent state sequence into corresponding agent space [4, 5]. Does the factorization of action model and dynamics model encourage learning a better representation or extracting the task semantics than other approaches? \n2. The world model here is conditioned on the expert videos (tasks), so it is more like a goal-conditioned future state predictor rather than pure dynamics model. While effective, I think the framing could be more precise.\n3. The process of transforming the generated waypoints into low-level robot actions requires further clarification (Attributed Waypoint Prediction in Section 3.2). For example, how does the waypoint controller computes low-level control commands for execution using inverse kinematics of the robot?\n4. The current method assumes the availability of aligned image-state pairs in the agent sequence. Could the authors discuss how the proposed approach could be adapted to a setting where only image-action trajectories are provided, where proprioceptive states of the agent are replaced by robot actions?\n\n[1] Ye W, Zhang Y, Abbeel P, et al. Become a proficient player with limited data through watching pure videos[C]//The Eleventh International Conference on Learning Representations. 2022.\\\n[2] Schmidt D, Jiang M. Learning to act without actions[J]. arXiv preprint arXiv:2312.10812, 2023.\\\n[3] Bruce J, Dennis M D, Edwards A, et al. Genie: Generative interactive environments[C]//Forty-first International Conference on Machine Learning. 2024.\\\n[4] Mazzaglia P, Verbelen T, Dhoedt B, et al. GenRL: Multimodal-foundation world models for generalization in embodied agents[J]. Advances in neural information processing systems, 2024, 37: 27529-27555.\\\n[5] Wang Y, Yu R, Wan S, et al. FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making[C]//Forty-second International Conference on Machine Learning. 2025.\n\nLIMITATIONS:\nThe authors should be more precise about the boundaries of their method's generalization capabilities. Acknowledging that the \"unseen\" tasks are still \"in-domain\" (i.e., from the same family of manipulations as the training data) would provide a more nuanced and accurate picture of the current work's scope.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles poor generalization in one-shot visual imitation learning by proposing a framework guided by a learned world model. Instead of direct policy learning, the method plans a trajectory of states and actions in the world model's latent space, which is then decoded into physical waypoints for execution. This approach is designed to better handle unseen tasks with different underlying contexts. Evaluated across simulated and three real-world platforms, the method reportedly demonstrates consistent and significant improvements over prior art, with over a 30% performance gain in some scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Technically solid method pipeline. One-shot transfer to unseen tasks is a challenging topic.\n- Comprehensive experiments conducted on 2 sim and 3 real-world benchmarks.\n- Conducting Human-to-Robot skill transfer, where the embodiment gap is quite large.\n\n## Weaknesses\n- Lack of comparison against one-shot imitation learning methods through alignment and open-loop replay, e.g., DINOBot.\n- I am not familiar with world-model-based methods for manipulation. While Human-to-Robot skill transfer is impressive to me, the studied manipulation tasks are generally simple, i.e., top-down grasping with low precision requirements. Given that BC-based or VLA-based policies have enabled very dexterous manipulation behaviors, I believe a discussion is necessary to illustrate the high-level pros and cons of this world-model-based approach when compared with more prevalent and straightforward BC-based methods.\n- Please refer to the questions for more minor weaknesses.\n\nDINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models. ICRA'24.\n\nQUESTIONS:\n- In the simulated experiments, it seems that how to divide the train and test tasks will dramatically affect the experimental results. For instance, the Button-Press task shares very similar action behaviors with the Button-Press-Topdown task. So what are the considerations behind this split? How are the similar tasks in the trainset different from the testing tasks?\n- What are the specific settings of Human-Franka tasks? What is the camera setup? Are the objects fully randomized on the tabletop?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}