Autonomous Functional Play with Correspondence-Driven Trajectory Warping
Authors: William Liang
Sam Wang
Hung-Ju Wang
Osbert Bastani
Yecheng Jason Ma
Dinesh Jayaraman
Venue: ICLR 2026 Poster (2026)
Keywords: Robot Manipulation
Autonomous Play
Robot Data Generation
Primary area: applications to robotics, autonomy, planning
TL;DR: Autonomous real-world play with robust correspondence-based trajectory warping generates high-quality data for effective downstream policy learning.

Abstract: The ability to conduct and learn from interaction and experience is a central challenge in robotics, offering a scalable alternative to labor-intensive human demonstrations. However, realizing such "play" requires (1) a policy robust to diverse, potentially out-of-distribution environment states, and (2) a procedure that continuously produces useful robot experience. To address these challenges, we introduce Tether, a method for autonomous functional play involving structured, task-directed interactions.
First, we design a novel open-loop policy that warps actions from a small set of source demonstrations (≤10) by anchoring them to semantic keypoint correspondences in the target scene. We show that this design is extremely data-efficient and robust even under significant spatial and semantic variations. Second, we deploy this policy for autonomous functional play in the real world via a continuous cycle of task selection, execution, evaluation, and improvement, guided by the visual understanding capabilities of vision-language models. This procedure generates diverse, high-quality datasets with minimal human intervention. In a household-like multi-object setup, our method is the first to perform many hours of autonomous multi-task play in the real world starting from only a handful of demonstrations. This produces a stream of data that consistently improves the performance of closed-loop imitation policies over time, ultimately yielding over 1000 expert-level trajectories and training policies competitive with those learned from human-collected demonstrations.

Published as a conference paper at ICLR 2026
AUTONOMOUS FUNCTIONAL PLAY WITH CORRESPONDENCE-DRNVN TRAJECTORY WARPING
William Liang! .2
Sam Wang'
Hung-Ju Wang
Osbert Bastani'
Yecheng Jason Ma 1.3
Dinesh Jayaraman" it
Universiiy of Pennsylvania
UUniversity of California, Berkeley 3 *Dyna Robotics
ht
.1
ABSTRACT
The ability to conduct and learn from interaction and experience is a central challenge in robotics, offering scalable alternative to labor-intensive human demonstrations. However, realizing such "play" requires (1) a policy robust to diverse, potentially antoof-distributiinn environment states, and (2) a procedure that continuously produces useful robot experience. To address these challenges, we introduce Tether, a method for autonomous functional play involving structured, task-directed interactions. First, we design a novel open-loop policy that warps actions from a small set of source demonstrations (S 10) by anchoring them semantic keypoint correspondences in the target scene. We show that this design is extremely data-efficient and robust even under significant spatial and semantic variations. Second, we deploy this policy for autonomous functional play in the real world via a continuous cycle of task selection, execution, evaluation, and improvement, guided by the visual understanding capabilities of vision- models. This procedure generates diverse, high-quality datasets with minimal human intervention. In a household-like multi-object setup, method is the first to perform many hours of autonomous multi-task play in the real world starting from only handful of demonstrations. This produces stream of data that consistently improves the performance of closed-loop imitation policies over time, ultimately yielding over 1000 expert-level trajectories and training policies competitive with those learned from human-colllectd demonstrations.
INTRODUCTION
Recent advances in robotic manipulation have been powered by imitation learning policies (Zhao
ct al. 2023 Chi et al., al. [223 Zhao et al. 2024 Black et al. 2024 Kim et al. 2024 Brohan et al.] 2022]
2023
Collaboration al. 2024) trained real-worke teleoperated demonstrations. In all these cases,
the human effort involved in teaching skill is substantial: while there are continued efforts to simplify
teleoperation interfaces (Zhao et al. 2023 Shafiullah al., 2023 Chi al. 2024 Cheng et al. 2024
Fu et al.] (024).. such demo datasets can fundamentally only scale linearly with human time, and
these data-hungry policy architectures need large spatially and semantically diverse datasets order to generalize usefully (Lin et al., 2024). In this paper, we propose an alternative paradigm: inspired by functional play in developmental psychology (Piaget, 1962, SSmilanky 1968) characterized by structured, task-directed interactions and repetitive practice we design Tether, a method for autonomous robot play that circumvents the human effort bottleneck.
Our method involves two key components. First, autonomous functional play requires extremely robust policy that can reliably recover from mistakes and ouitt--distiibution states. Without relying on massive datasets for training large neural policy architectures, instead design a new open-loop policy class that specifically supports generalization with few demonstrations. Specificallyy our architecture exploits the remarkable leaps in semantic image keypoint correspondences: given new scene with potentially new task-relevant object instances in new spatial layout with new distractors, first computes keypoint correspondences with the demo images, selects the dastste demo,
Equal advising.

---

Published as a conference paper at ICLR 2026
computes 3D transformations associated with each correspondence, and accordingly warps the robot trajectory to fit the new scene. Validated on 12 manipulation tasks in a household-like setting, we show that our policy surpasses the performance of alternative methods, including those that rely on foundation models or pretraining with large robotics or internet datasets.
Second, we run our policy within cyclic functional play procedure that autonomously produces data for continuous downstream policy training. For a collection of tasks with few human demonstrations each, we query a vision-language model (VLM) repeatedly plan and select tasks our policy should attempt. This is real-world play without any manual resets: the procedure naturally induces resets with a growing initial state distribution as the object configurations drift away from the beginning of play. Additionally, to evaluate the suboptimal play data, we query a VLM to detect successful executions, which are then used downstream for filtered imitation learning. We show that this procedure can collect over 1000 new expert-level demonstrations across 26 hours with minimal human intervention (5 cases requiring minute total of human time, 0.26% of executions). We also validate that this stream of newly generated demonstrations progressively enhances the training of imitation learning policies, which consistently improve with more play data and reach high success rates competitive with counterparts trained on human-collected demonstrations.
In summary, our contributions are:
1. A keypoint correspondence-driveee trajectory warping policy that exhibits impressive spatial and semantic robustness for diverse manipulation tasks.
2. A multi-task VL.M-quided play procedure that generates diverse expert-level demonstrations over many hours, powering downstream neural policy training.
2 RELATED WORK
Robustness in Imitation Learning. To build robust policies that excel in diverse environments, many prior efforts turn to scaling data and policy architectures, often with foundation models or large
human-collected demo datasets (Khazatsky et al.) [024] Collaboration et al. 2024 WoiBottWorld
Contributors et al.] 2025). Prominent techniques include training representations for robotics on
non-robot data (e.g.. human videos) (Nair al. 2022 Ma et al. 2023; Shi al. 2025). ) querying
vision- language models (VLMs) or large language models (LLMs) (Nasiriany et al. 2024 Goetting
al. 2024 Fang et al. (2024), and finetuning vision- waano--nnngg---eteee
models (VLAs) (Black et al.
2024
NVIDIA et al. 2025; Intelligence et al. 2025).
Another class of approaches design strong built-in priors for models trained on much fewer demos.
with some executing actions open-loop. These works build action affordances or primitives (Kuang
al.
2024| Halddrr
Pinto 2025). retrieve from existing datasets (Du et al., [023] Memmel et al.
2024
Xie et al.] 2025). leverage pretrained representations and models (P (Pari et al. [02]] 1; Burns
cct al.
2023 Shi et al.| 2024), or exploit 3-D scene geometry (Rashid et al. 2023 Goyal et al. 2024
Ke et al. 2024 Ze et al. 2024). Closest to our work are methods that operate
visual semantic
keypoint correspondences. Like object-centric approaches (Devin et al.) 2018; Qian et al. 2024 Zhao
et al. 2025), they benefit from recent advances in scene understanding and naturally ly robust to distractors, yet provide higher spatial precision and avoid the rigid "objectness" assumptions that fail on deformable objects and granular partieles. One class of approaches tracks keypoints through frames of human or robot videos and retargets the dense trajectory to the desired setting (Wen et al., 2023 Bharadhwaj al., 2024 Ren al. 2025). Another class instead uses keypoint correspondences as compact trajectory representation. Among these, KAT (Di Palo & Johns 2024) queries an LLM with keypoints to generate open-loop actions, while P3-PO (Levy et al. 2024) and SKIL (Wang et al. 2025) input keypoints to point-conditioned policies. We too use keypoint correspondences, but demonstrate the advantages of a more direct approach: to select and warp demonstrated trajectory to fit the new scene. We compare against KAT (Di Palo & Johns 2024) in experiments.
Autonomous Data Generation. To reduce the need for large human-collected manipulation datasets, recent efforts have explored data generation for policy learning. Some works collect data in simulation by querying foundation models to propose and solve tasks (Ha et al. 2023 Wang et al. 2024) or leveraging privileged simulation state to adapt demos for new scene configurations (Mandlekar et al.) 2023) Jiang et al. 2025] Lin et al. 2025a). However, simulation based approaches struggle with sim-to-real transfer within cluttered, unstructured environments, which remains open challenge
2

---

Published as a conference paper at ICLR 2026
especially for tasks involving complex contacts and vision-based policies trained synthetic renders (Blanco-Mulen) Mulero et al.) 2024; Yu et al. 2024 Lin et al. 2025b). Alternatively, other works generate data directly in the real world. Some require polictes trained on hundreds of human-collected demos (Zhou et al. 2024 Mirchandani et al.
2024). which still pose a bottleneck when transferring to new tasks and environments. Most simil lar our work methods that initialize from few demos using specially-deignee
policies. Manipulate-Anythinn foundation model policy for autonomous data collection, but performing multiple rounds of foundation model inference significantly hinders throughput, culminating in under 50 demos; additionally, environment reset is not considered. In contrast, we introduce a system that autonomously produces over 1000 demonstrations in the real world with minimal human intervention.
(Duan et al. 2024b) introduces a zero-shot
3 ROBUST IMITATION AND AUTONOMOUS PLAY
We formulate robot manipulation Controlled Markov Process (CMP). A CMP is represented as tuple .M- = (S, A, T), where S is the set of states, is the set of actions, T S x A + (S5) is the transition dynamics function.
In the imitation learning problem, for a given CMP M and a desired task, we have a dataset that perform that task, with trajectories т₃ {81, a1, 82, ... In practice, our robot does not have direct access to state 8, and instead receives visual observations o Oss)). consisting of two third-person RGB camera views from the left and right sides, as shown in Figure I We assume no significant occlusions or partial observability. The actions a are the 6-DOF pose of robot gripper and the gripper's binary open/close command. Given this demonstration dataset D, our goal is to learn policy π O + A A) for the task.
of N expert demonstrations D {πi T2 .2...... TN
As motivated in the introduction, today's standard imitation learning approaches require extensive human effort that is difficult to scale. Our paper directly addresses this problem. First, to produce effective behaviors with minimal human demo collection, we introduce an open-loop trajectory warping-based policy that generates action plans from few training demonstrations (Section [31)) much fewer than standard neural policies. Next, this policy serves as robust bootstrap for au- tonomous data generation, powering VLM-guided functional play procedure (Section 3.2) that produces demos for the downstream training of more expressive and powerful policy architectures.
3.1 TRAJECTORY WARPING WITH KEYPOINT CORRESPONDENCES
Towards spatially and semantically robust imitation against large variations in the initial state, Tether leverages semantic visual priors in the form of image correspondence matching algorithms to interpolate within and generalize beyond few demonstrations.
Our policy is designed to run open-loop, mapping from the initial state distribution to a full sequence of actions, and operates as follows. Every demonstration is represented by the initial frame, i.e. the camera observations at the beginning of the demo, and sequence of 3-D waypoints for the gripper motion. These waypoints, projected onto the image, identify important visual "keypoints" for that demo. To execute the policy starting from an initial observation of the scene, identify the best-matched demo based on the quality of keypoint correspondences, backproject those keypoints compute desired 3-D gripper waypoints, and warp the demonstrated trajectory based on those waypoints. We walk through these steps in detail below.
Demonstration Summaries: Image, Waypoints, and Keypoints. Our policy is non-parametric, and relies on accessing demonstrations at test time. For convenience, we first preprocess all demonstrations into concise summaries preparation for execution. This is a "training time" operation, it needs to be done once for each demonstration, and once summarized, the original demonstration can be discarded. For each demonstrated trajectory т; E DD we summarize the key information for Tether in tuple K; (o, W, K. a), where o is the initial image, i.e. two-view camera observations at the beginning of the trajectory. W is a "waypoint" sequence [w1, W2, .... WT] of task-critical 3-D gripper locations. In practice, we simply use the gripper locations from frames where the gripper open/close position toggles, following the convention for selecting critical frames from prior work (Johns) 2021 Mandlekar et al. 2023] Vecerik et al. 2224)), and which is applicable to diverse manipulation tasks; in Appendix[A.I] we also show that Tether works well with alternative waypoint extraction mechanisms. Next, (01;--., aM. is the full sequence of robot actions during the trajectory, i.e. 6-DOF gripper
3

---

Published as a conference paper at ICLR 2026
W
Wa
(left)
O |right]
Figure 1: Demonstration Summaries, Tether summarizes demonstrations into the initial frame,
action sequence (red), waypoints (blue), and keypoints.
Demo Summaries
Obs
Demo llett]
Demo K (right)
demos K
Correspondence Selection
demo k", waypoints W
Warping
Action plan
Obs (left)
Obs (right)
Robot Execution
Correspondence
Warping
During inference, Tether (left) computes correspondences (middle) and produces warped trajectory action plan (right).
Figure 2: Policy Inference.
positions and gripper state. Finally, for each of the T waypoint locations, we project them onto to identify visually important keypoints K = [k1, k2 ..., kr). We depict this process Figure
This remaining subsection outlines executing our policy for an observation visualized in Figure
Correspondence Matching and Source Demo Selection. To execute a Tether policy, we start by matching the current camera observations to the demos in D to find weighot demo. In particular, for each demo summary H;, we search the current images for correspondences for all keypoint locations K; in the demo images We do this separately for the left-image and right-image to find a set of corresponding pixels in each image K; K feftt rightel To find these 2D correspondences, we use tate--fftthe-art model (Zhang et al., 2024) built on DINOv2 (Oquab et al.. 2023) and Stable Diffusion (Rombach et al. 2022) features in our implementation.
We then backproject these correspondeecces K; using calibrated camera extrinsics to obtain sequence of target 3-D waypoints W, If the backprojections fail to intersect, then the match is deemed to have failed, i.e. demo an infeasible match for the current observation o. For the feasible matches, we rank the demos in order of dissimilarity to by computing the Euclidean distance between the original and target waypoints, i.e. (c) = IW; W,(o)]l2- The closest demo is selected as the source demo x* with its original gripper waypoints W*, original robot action sequence a", and the translated target waypoints for the current scene (o)...
Warping the Source Demo Trajectory. The "target waypoints" W*(o) above provide a scaffold for how to warp the robot trajectory for the current scene. However, we still need to fill in the fine-grained robot actions in between by warping the intermediate segments between waypoints.
Consider the segment (W,, Wc+1] between two consecutive waypoints from the selected source demo R² The target waypoints are [w0,, i2+1]. Denote the action sequence segment for this waypoint as a² We first compute the 3-D displacements of the two waypoints that mark the beginning and end of the segment, d, = Wi we and d++1 i W1+1- We now perform linear interpolation between those displacements and add the resulting displacements to the original action sequence a get the transformed action plan segment å, Concatenating these segments produces the full action plan
āā, ag, ...] to be executed in the scene.
4

---

Published as a conference paper at ICLR 2026
Target Task
Selected Task
Binary Success
Set Tasks
Planning
Policy UCB
Success Detection
o
Obs Action plan
Execution
Autonomous Play
Figure 3: Autonomous Functional Play. Our iterative procedure runs Tether for multiple tasks and uses VLMs for plan generation and success detection.
To prioritize preserving spatial relationships, we perform this linear interpolation in space rather than time. For waypoints W/; W1++1 define local 1-D coordinate frame mapping w2 to 0 and w to I. Then, the interpolation coefficient a for each a E the source action segment a is simply its coordinate in this frame. Geometrically, this can be thought of as projecting the gripper position a onto the line spanning Wi and W++1, then computing the projection's 's relative distance to w, and wt+1- Then, the corresponding displacement that must undergo when warped into the new scene is: da = (1 add, + adæ-1 The new action is thus + du.
While simple, we show in Section 4.2 that our correspondence-drir
trajectory warping performs remarkably well with as few 10 demos, across challenging manipulation tasks-inclune those with dfitt--disttiuution objects, millimeter-level precision, and complex contacts. Additional details and pseudocode are in Appendix.
3.2 AUTONOMOUS FUNCTIONAL PLAY WITH VISION-LANGUAGE MODELS
Now, we address the scalability issue of human data collection by using our Tether policy to set autonomous play in motion. This expands the data distribution starting from a few demos per task, so that the expanded, more diverse dataset can power larger and more flexible neural policy architectures.
To maximize the autonomy of our data generation, we apply our policy towards continuous functional play and design a set of tasks that facilitate natural resets and randomization: the end state of each task is valid start state for another task (e.g.. "place pineapple on table" leads into "place pineapple on shelf" or "place pineapple in bowl"), even in the event of failures. This approximately indefinitely composable structure is extension of forward backward tasks in prior reset-free
learning (Bysenbach et al., 2017 Sharma et al. 2021 Mirchandani et al. 2024), and promotes a set of reachable states that is effectively "closed" under the task distribution, while allowing previous tasks (and potential mistakes) to naturally randomize both relevant and background object locations and states for each task. This formulation is further illustrated by experiments Section 4.3
With these tasks, we run an iterative procedure where each step applies the policy to complete one task from our set. At high level, each iteration proceeds follows: we first query VLM with an image of the scene and ask for an appropriate task to attempt. Then, we run the corresponding Tether policy, record its execution, and evaluate it with another VLM query. These steps are visualized in Figure 3 and described below, along with pseudocode in Appendix.
Task Selection And Planning. each iteration, we select tasks based on both which demos we would like to add to our collection, and which tasks are actually executable. To prioritize demos to add, we maintain a running count of the number of successes for each task, and weight rare tasks higher. In particular,
sample the target task from softmax over the negated success counts.
However, this rare target task might not instantly executable, For instance, to attempt to "move object from shelf to table," the object of interest must have been placed on the shelf in the first place. that object could be in many other locations in the scene, it might only rarely reach the shelf the course of undirected play. To overcome this, query VLM to provide task plan, i.e. sequence of executable tasks that culminates in the target task, of which we attempt the first task within the current iteration, similar to receding horizon control. Our prompt and examples are in Appendix.
Success Evaluation. Having attempted a task, we determine the success of the resulting trajectory via VLM query. We follow the recommended practice (Team 2025), providing images of the
5

---

Published as a conference paper at ICLR 2026
Pineapple Pineapple Bowl Bowi from
Wiping Clath
Opening Cabinet
Apple
Pineagple Basket
Pineapple
Hanging Tape
Insarting Coffee
Figure 4: Evaluation Tasks. Our tasks involving moving fruits and containers with in-distribution
(orange) and out-of-distribution (green) objects, as well as challenging manipulation skills (purple).
pre- and post-execution scene state from left, right, and wrist cameras. While prior work (Duan iet al. 2024a) found that naively querying off-the-shelf VLMs results in bias toward false positives, have observed near-perfect performance in settings, possibly owing the multi-camera full-observability setup and the use of a VLM trained specifically for embodied reasoning tasks (Gemini Robotics-ER 1.5 TTea al. 2025 ).
Improving For and Through Play. There are a few additional considerations when deploying Tether. First, play benefits from injecting stochasticity to generate exploratory data search for improved policies. This also expands the set of potential actions, decreasing the chance of being trapped in states where all actions fail to induce environment transitions. Thus, rather than providing our policy with all demos, we first sub-select k demos and run our policy warp the closest one amongst them.
While we can select these demos randomly, there is second consideration: human demos could be of varying quality, in which case, play using various source demos could help identify consistently better demos to warp from, avoiding e.g. demos that involve non-robust fingertip grasps Thus, we select the top & demos by formulating multi-arm bandit problem: arms demos, and the reward for picking a demo is the binary success of the executed trajectory warped from that demo. For this upper confidence bounds (Garivier & Moulines] 2011), which balances exploring relatively less-tested source demonstrations with exploiting high-success ones.
problem,
4 EXPERIMENTS
We conduct experiments evaluating our policy design and autonomous play procedure. Specifically, we study (1) the robustness of our Tether policy to large, potentially out-of- -iit---iittiittoon variations in the initial state configuration (i.e., new poses or new object instances), and (2) the effectiveness of autonomous play in generating a stream of data for downstream closed-loop policy learning.
4.1 EXPERIMENTAL SETUP
We run all experiments the DOF Franka Emika Panda running 15 Hz and record two RGB views from calibrated ZED cameras. Across all experiments, we provide 10 demonstrations for each task. We run our semantic correspondence 1 A6000 GPU. In autonomous play, we use Gemini Robotics-ER 1.5 for task selection and success evaluation. Additional details are in Appendix.
Baselines And Ablations. We compare our Tether policy with recent imitation learning methods. oftte--f-the-aae
These baselines are representative of the state-of- of the-art art across various levels of data-efficiency. First, To (Black al., 2024) (with FAST (Pertsch et al.) 2025) is an open-source VLA that can operate zero-shot or finetuned with additional demos. Second, Keypoint Action Tokens (KAT) (Di Palo & Johns 2024) queries a L.LM for in-context action sequence generation, given a few demos (10 in the original work). Third, Diffusion Policy (DP) (Chi et al. (223) is general imitation learning algorithm typically trained with a few tens up to few hundreds of demos, depending on task complexity. Here, evaluate Fo zero-shot and finetuned with 10 demos, and provide DP and KAT with 10 demos as well, same as our approach. Additionally, we ablate the number of demos given to method, evaluating with 1, 5, and 10.
Tasks. We visualize our 12 tasks in Figure First, we have 4 tasks that involve moving fruits and containers table and shelf. We instantiate these tasks with soft pineapple toy and curved rigid

---

Published as a conference paper at ICLR 2026
In-Distributon Fruit and Containers
Success Rate
0.75
0.50
0.00
Pincapple from Shelf
Pineapple Boxl
Bow to Sheff
Bowl from Shelf
Out-of of Distribution Fruit and Containers
Success Rate
0.75
0.50
0.00
Apple Dowl
Stewwoery to Bool
Pinesople to Baaket
Pineopple
Challenging Manipulation Skills
1.00
Success Rale
0.75
0.00
Wping Cleth
Opening Cabinet
Insorting Coffee
Demos)
Demos)
Figure 5: Main Policy Comparison. We compare the Tether policy with baselines across 12 tasks.
bowl. The main challenges are the bowl, which requires careful orientation of the gripper to prevent slipping, and the shelf, which requires a horizontal orientation approach to avoid collision.
We start with in-distribution objects (the same pineapple and bowl from demos) to specifically evaluate spatial generalization, since our few demos do not fully cover the entire distribution of object positions. A Afterwards, we test with out-of-distribution objects to also assess semantic generalization: focusing the "Pincapple to Bowl" task, change the pineapple to apple (color change) or strawberry (size change) and the bowl a basket (appearance change) or cup (geometry change).
Next, we probe the limits of policy design in 4 challenging tasks involving deformation, sustained contacts, articulation, and precision. First, the robot picks up soft cloth and maintains consistent contact to wipe marks off whiteboard. Second, it grasps cabinet doorknob 0.5 centimeters thick (1/4 of gripper width) and applies stable contacts to fully open the tight hinge. Third, it places roll of tape small silver hook, 3 centimeters deep and visible in only few pixels. Fourth, it inserts K-cup pod into coffee machine, which demands precision with an error margin of millimeters.
We report success rates 10 trials. For each trial, randomize object positions and the same two camera views of the entire scene (in Figure D This naturally includes irrelevant locations in the majority of the image, requiring methods focus on relevant regions of the scene. The "Inserting Coffee" task is the sole exception: since the 8mm margin of eтror projects onto the camera views as pixels, we move the cameras closer zoom in the coffee pod and machine compartment.
4.2 ROBUST IMITATION
Tether policy outperforms alternative imitation learning approaches. In Figure 5 we find that given few demos, our policy surpasses baselines across all tasks. Diffusion Policy, being an end- to-end model trained from scratch without built-in priors, fails generalize from just 10 demos. Meanwhile, zero-shot To performs well on standard tabletop pick-and-place likely benefiting from its pretraining datasets containing similar behaviors, but fails with more complex tasks due to (1) incomplete command understanding (c.g., grasping the cloth but failing to locate the whiteboard) and (2) imprecise manipulation (e.g.. approaching the coffee pod but missing the grasp)
Next, two baselines failed to achieve any successes on our tasks. First, we observe that finetuned To collapses when trained only 10 demos, likely due often does not move at all, or misses the object. Second, our few-shot learning baseline, KAT, was not well-suited for extracting task-relevant features from cluttered scene, and when given human-
overfitting. When deployed, this model

---

Published as a conference paper at ICLR 2026
Successes Per Task
150
Bux26ses
100
Beginning
50
0
250
500
750
1030
1250
1500
1750
2000
Total Attempps
Bowl from Shelf to Table
Pinnappie from Shall to Table
Pineopple
Pineapple from Bowl Table
Pineopple to Shelf
Endl
Figure 6: Autonomous Play Statistics. around 26 hours of play, our method produces over 1000 cumulative trajectories across 6 tasks (left) and significantly expands data diversity, shown by the heatmaps of object poses at the heginning and end of play (right).
annotated inputs, it struggled with in-context learning from the complex multi colti-dimennsionl patterns in our tasks, caused by orientation changes, non-linear velocities, and wide object distributions. Additional details and experimental verifications for both baselines are Appendix.
Tether policy excels at both spatial and semantic generalization. First, our policy excels in tasks involving spatial robustness with in in-distribution objects, including the bowl, which requires accurate orientation and position avoid slipping. Second, our policy performs well even with out-of-distribution objects, attesting to the strong generalization inherited from visual correspondence: without the demo object being present at test-time, correspondence finds the most semantically similar object and pinpoints the relative region of the keypoint (c.g., center of fruit or rim of container). This capability is significant especially for grasping the strawberry, which is vastly different in appearance and 1/4th the size of the demonstrated pineapple, and cup, which has 1/2 the diameter of the bowl and just big enough to fit the pineapple, thus requiring precision along with generalization.
Tether policy succeeds even on challenging manipulation tasks. Our policy is effective at more difficult manipulation tasks despite their challenges with deformation, sustained contacts, articulation, and precision. Successes with small features of the scene like the cabinet knob, hook, and coffee machine compartment demonstrate the high accuracy of our semantic correspondences. Most notably, method achieves non-trivial success for the coffee insertion task without using the wrist camera; this task requires significant accuracy during grasping, since the deformable cylindrical pod crumbles if held just centimeter off, and insertion, since the difference in pod and compartment diameter is mere 8 millimeters. Additionally, fully opening the cabinet and wiping marks off the whiteboard show that trajectory warping can maintain consistent contacts with the scene, even without closed-loop adjustments. Lastly, accurate grasps of the soft cloth demonstrate our method's flexibility with deformable objects, in contrast with prior work (Wen al. 2022 Mandlekar et al. 2023 Zhu et al.) 2024) that rely on rigid objects for pose estimation.
Finally, while our Tether policy exhibits strong robustness to spatial and semantic variations in the initial state, even for challenging tasks, its open-loop nature remains a bottleneck, limiting reactivity
and recoveries during execution. Thus, it serves not as an independent policy solution, but bootstrap for generating experience and training more expressive policies, as we investigate below.
4.3 AUTONOMOUS PLAY
Given our policy's strong performance, we now run subset of evaluation tasks for autonomous play. Specifically, use the demos from Section 4.2 (10 per task) for moving the pineapple between table, shelf, and bowl, as well as moving the bowl between table and shelf. As described in Section 3.2 this task set is chosen that there is high certainty of at least one viable subsequent task that can be executed after previous tasks, even in their most common failure cases (e.g. object drops to table).
Tether runs across multiple hours without resets. We run autonomous functional play for sessions totaling around 26 hours. Success statistics for each task are shown in F Our policy generates 1085 successes from 1946 attempts across 6 tasks, averaging success every 86
8

---

Published as a conference paper at ICLR 2026
Bowl from Shell to Table
Bowl to Shelf
Pineapple from Bowl to Table
1.00
1.00
1.00
0.76
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
Success Rale
0.00
0.00
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Pineapple from Shelf Table
Pineapple Bowl
Pineapple to Shelf
1.00
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Total Attempts
Figure 7: Downstream Policy Learning Results. The stream of data generated by autonomous play consistently improves policy performance over time. Policies trained on equal in number the final set of successful fether-generater trajectories, are shown in black.
demos,
seconds and 1 attempt every 48 seconds, with cumulative success rate of 55.8%. We intervene total of 5 times, amounting to 0.26% of attempts and an average of once every 5.2 hours. Timelapse and intervention details are in Appendix.
We observe that these success rates are lower than those in Section 4.2 due to the uncontrolled nature of play. For instance, the bowl is sometimes tilted on its side due to imprecise placements or accidental pushes: these make grasps significantly harder, though our policy is still able to recover and fix the bowl. On rare occasions, play enters fallure mode where the bowl is flipped completely upside-down after dropping from the shelf; this state is generally irrecoverable with only one arm and causes the majority of our interventions. However, on two separate instances, the robot accidentally recovers by squeezing the bowl against the shelf and forcing it back upright. While such recoveries are not intended and occur purely by luck, they highlight the interesting nature of play: that at scale, coincidences may result in unexpected novel behaviors.
Tether produces reliable task plans and success evaluations. We evaluate our VLM-based planning and evaluation by annotating all 1946 attempts: we deem task plan correct if it can be feasibly executed given the current scene image, and an execution as successful it achieves the task's 's end state without disturbing task-irrelevant objects. Compared against the VLM responses, we achieve 95.2% accuracy for task planning and 98.4% precision (at 89.6% recall) for success evaluation; note that we prioritize precision as it's most important to minimize false positives that pollute the success data. These results validate the reliability of our VLM-based components for play.
Tether produces diverse trajectories. In Figure 6 (right), we visualize the diversity of keypoints from demonstrations and successful play trajectories. We see that while our demos sparsely cover the table and shelf, using them to seed play allows us not only interpolate between the demos but also expand on the edges of the distribution, such as the area around the cabinet.
Tether produces a stream of data that trains effective policies. Next, we train parametric policies on the generated data. While there are numerous algorithms for learning from suboptimal data, we adopt filtered behavioral cloning as a straightforward and effective approach. Integrating alternative methods that make fuller use of suboptimal trajectories remains key direction for future work.
After every 500 play attempts, we train Diffusion Policies on the cumulative successful trajectories for each task. We track their performance in Figure 7 Across all tasks, these policies progressively improve over time, with most eventually reaching near-perfect success rates. Thus, the data generated by Tether is consistently high-quality and effective for downstream policy learning. Diving deeper into the nature of this improvement, we see that as our ever-expanding datasets scale with more play, they increase in diversity and naturally build stronger coverage of the object pose distribution. Thus, policies trained on this data improve primarily on their spatial robustness to different object positions: earlier policies succeed only when the object in some specific locations, whereas our final policies are generally effective for any random object placement. Note that this robustness also extends to

---

Published as a conference paper at ICLR 2026
Success Rain
0.75
0.50
0.25
0.00
Pinnappin from Shelf
Pincapple to Shat
Pintapple from Rowi
Pinnopple Rool DP (141-202 Human Demos)
Row from Shelf
Riral In Shalf
Ours (10 Demos)
Figure 8: Comparison on Play Distribution. We compare Diffusion Policies trained on human-
collected demos with Tether policy on environment states encountered during autonomous play.
distractor objects that were involved in other tasks during play: for instance, policies that interact only with the bowl perform well irrespective of the pineapple position, and vice versa.
Additionally, we compare against policies trained on human-collected datasets, using the same number of human demos as the final set of successful trajectories produced by Tether (between 141 and 202, depending on task). seen in Figure 7 our policies generally achieve success rates similar their human-data counterparts, thus confirming that Tether indeed produces expert-level trajectories competitive with those experienced telcoperators. while requiring minimal human effort. On average across the 6 tasks, policies even achieve slightly higher success rates, which we hypothesize arises from two factors. First, the environment is naturally randomized when play is performed at scale, opposed to potentially biased randomization during manual resets. Second, Tether trajectories are produced by warping from small demo set, and thus its motions occupy relatively narrow (yet effective) mode of the expert distribution, whereas the hundreds of human- collected motions are relatively more multimodal and thus more difficult for the policy to capture.
Tether policy is essential for robust autonomous play, Finally, we investigate whether the Tether policy is only valuable when the number of source demos is very small, and if standard imitation learning approaches are more effective if we have more demos. To answer this, we simulate the result of plugging in the above human-date Diffusion Policies into play, replacing the Tether policy: for each the 6 tasks, using 20 randomly sampled play-induced initial states from our multi-hour play experiment, we compare Tether policy (with 10 demos) and Diffusion Policy (with 141 to 202 demos, depending on task). As seen in Figure 8] Tether policy consistently achieves higher success rates, whereas Diffusion Policies fail to generalize to the broader play state distribution (e.g. tilted bowl or entangled objects). Thus, our policy design is key to sustaining efficient and robust autonomous play, minimizing human intervention and maximizing throughput.
5 CONCLUSION
We have presented Tether, system for autonomous functional play with structured, task-directed interactions. We introduce a novel policy design with keypoint correspondence and trajectory warping that exhibits impressive spatial and semantie robustness, and deploy within VL. multi- task play procedure that produces over 1000 successful trajectories in 26 hours with minimal human intervention. This generated data, funneled downstream for filtered imitation learning, consistently improves the performance of neural policies, which ultimately reach near-perfect success rates. We believe that Tether demonstrates the potential for an alternative path in robot learning: one driven by scalable methods that perform and learn from autonomous interaction and experience.
Limitations. Our policy's 's strength in exploiting semantic correspondences is also weakness: recent advances in computer vision enable robust feature tracking despite textureless surfaces, deformable regions, lighting variations, and partial occlusions, yet the abstraction of an image into keypoints inherently makes the policy more susceptible to occlusions. In addition, our policy is deliberately designed for the low-data regime, and thus has built-in structural generalization and assumptions that are not suitable for larger datasets; in particular, open-loop execution limits its applicability to dynamic tasks that require reactivity, and trajectory warping struggles with complex motions that cannot be transformed from source demos. promising future direction could leverage the Tether policy as a strong prior, while remaining flexible to improving via imitation or reinforcement learning more data becomes available through play, thus enabling more effective seff-improvemennt
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We are grateful Jie Wang, Edward Hu, Junyao Shi, and Pieter Abbeel for their helpful feedback and discussions during the project. William Liang is supported by the NSF GRFP. This project was funded by NSF SLES 2331783, NSF CAREER 2239301, ONR N00014-22-1-2677, and DARPA TIAMAT HR001 12490421.
REFERENCES
AgiBot- TeBBot-Woldd-ce World-Contributorrs Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan. Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang. Lei Yang, Shukai Yang. Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems, 2025. URL (ttps::: /ttp///avvvvv/v. org/ abs 2250300665
Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipulation. In European Conference on Computer Vision (ECCV), 2024.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A laiio--lnguuge-acction flow model for general robot control. arXiv preprint arXiv:2410.24166 2024.
David Blanco-Mulero, Oriol Barbany, Gokhan Alcan, Adrià Colomé, Carme Torras, and Ville Kyrki. Benchmarking the sim-to-real gap in cloth manipulation. IEEE Robotics and Automation Letters, 100:)2981-298,, 2024. doi: 10.1 RA110/9/RA.22243330000
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.068// 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision carion--nnguge-actton models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 30t.:38/77//8.. 2023.
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? arXiv preprini arXiv:2312.12444, 2023.
Xuxin Cheng. Jialong Li, Shiqi Yang, Ge Yang, and Xiaolong Wang. Open-television: Teleoperation with immersive active visual feedback. arXiv preprint arXiv:240.001512 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241273666. 2023.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint 2rXiv:2402.10322 2024.
Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le. Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer
11

---

Published as a conference paper at ICLR 2026
Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ccola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homanga Bharadhwaj.. Homer Walke, Hongjee Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu. Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Sal Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch. Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang. Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto. Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero. Maximilian Du Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim. Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart' in-atin-Mart'in in. Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang. Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song. Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh- Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang. Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open x-embodiment: Robotic learning datasets and rt-x models, 2024. URL http://twww..../////23111100
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representa- tions for generalizable robot learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2018
Norman Di Palo and Edward Johns. Keypoint action tokens enable in-context imitation learning in robotics. arXiv preprint wwXiv:2403.19578 2024.
Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv preprint 0rXiv:2304.08742, 2023.
Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. Aha: A lisoon-laggaage-model for detecting and reasoning over failures in robotic manipulation. arXiv preprint
2024a.
Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. canipulltee-nything Automating real-world robots using lasion-language models. In 8th Annual Conference on Robot Learning, 2024b. URL https:://.........t...tt RRRl:tttss::/oeeeeiieeee http://pppeeeeeeeeeeeeee net.eiiwwwweeeeeeeeeeeeeeeeeeeee fformm?
Hd=2SYFDG4WRA
12

---

Published as a conference paper at ICLR 2026
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXiv:1711.06782, 2017.
Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. Moka: Open-world robotic manipula- tion through mask-based visual prompting. Robotics: Science and Systems (RSS), 2024.
Xiaolin Fang, Bo-Ruei Huang, Jiayuan Mao, Jasmine Shone, Joshua B. Tenenbaum, Tomás Lozano- Pérez, and Leslic Pack Kaclbling. KALM: Keypoint Abstraction using Large Models for Object- Relative Imitation Learning. In IEEE International Conference on Robotics and Automation (ICRA), 2025.
Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406. 10454, 2024.
Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switching bandit problems. In International conference on algorithmic learning theory, pp. 174-188. Springer, 2011.
Dylan Goetting, Himanshu Gaurav Singh, and Antonio Loquercio. End-to-end navigation with vlms: Transforming spatial reasoning into question-answerieg In Workshop on Language and Robot Learning: Language as an Interface, 2024.
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, and Dieter Fox. Rvt-2: Learning precise manipulation from few demonstrations, arXiv preprint arXiv:2406.08544 088445, 2024.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot /2307. 14535
skill acquisition, 2023. URL
Siddhant Haldar and Lemrll Pinto. Point policy: Unifying observations and actions with key points for robot manipulation. arXiv preprini S.Ki::2502.233991, 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke. Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong. Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. To.5 vision-language-ccioon model with open-world generalization, 2025. URL. | https: //arxiv org/abs/2504. 16054
Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025.
Edward Johns. Coarse-to-fine imitation learning: Robot manipulation from single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA), pp. 4613-4619. IEBE, 2021.
Tsung-Wer Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402. 10885, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.129.. 12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source lision-language-action model. arXiv preprint arXiv:2406. 092.00.99466.4 2024.
Yuxuan Kuang, Junjie Ye, Haoran Geng, Jiageng Mao, Congyue Deng, Leonidas Guibas, He Wang, and Yue Wang. Ram: Retrieval-basse affordance transfer for generalizable zero-shot robotic manipulation. arXiv preprint arXiv:2407.04689 2024.
13

---

Published as a conference paper at ICLR 2026
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pp. 71-91. Springer, 2024.
Mara Levy, Siddhant Haldar, Lerrel Pinto, and Abhinav Shirivastava. P3-po: Prescriptive point priors for visuo-spatial generalization of robot policies, 2024. URL https: httpps://arxiv.oom abs/
4412.06784
Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprini urXiv:2410. 1824477...77, 2024.
Kevin Lin, Varun Ragunath, Andrew McAlinden, Aaditya Prasad, Jimmy Wu, Yuke Zhu, and Jeannette Bohg. prnstraint- data generation for visuomotor policy learning, 2025a. URL
wttp:/orrgabba//ooooo 2508. 03944
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids, arXiv preprint drXiv:2502.20196. 2025b.
Yecheng Jason Ma. Vikash Kumar, Amy Zhang. Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301-23320. PMLR, 2023.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.
Marius Memmel, Jacob Berg, Bingqing Chen, Abhishek Gupta, and Jonathan Francis. Strap: Robot sub-trajectory retrieval for augmented policy learning. arXiv preprint 2XXi:24412.15882 2024.
Suvir Mirchandani, Suneel Belkhale, Joey Hejna, Evelyn Choi, Md Sazzad Islam, and Dorsa Sadigh. So you think you can scale up autonomous robot data collection? In Conference on Robot Learning, 2024.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint 22X::2203.116011 2022.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang- Huci Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn. Sergey Levine, and Brian Ichter. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. 2024.
NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da. Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dicter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhen- jia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. GrOOt nl: An open foundation model for generalist humanoid robots, 2025. URL
aattps://axxx wwt. wtg/ ht.gg//bb/5550 14734
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.071193. 2023.
Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint arXiv:21 2021.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- lision-lagguage-actton models, 2025. wtt.hhttt://aaxxt. http://awwiii..rrr////5//5000.00 09747
14

---

Published as a conference paper at ICLR 2026
Jean Piaget. Play. Dreams, and Imitation in Childhood. 1962.
Jianing Qian, Yunshuang Li, Bernadette Bucher, and Dinesh Jayaraman. Task-oriented hierarchical object decomposition for visuomotor control. arXiv preprint arXiv:2411.01284, 2024.
Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Chen, Angjoo Kanazawa, and Ken Goldberg. Language embedded radiance fields for zero-shot task-oriented grasping, 2023.
URL htt.httss///axiv.cooo wttp:ogggabbb///aa/ee 07970
Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, and Jeannette Bohg. Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning. arXiv preprint arXiv:2501.06994, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home. arXiv preprint arXiv:2311.1608 2023.
Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 34:18474-1844 18486, 2021.
Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre-trained object- centric representations for robotics from" what" and" where" foundation models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 15424-15432. IEEE, 2024.
Junyao Shi, Zhuolun Zhao, Tianyou Wang, Ian Pedroza, Amy Luo, Jie Wang, Jason Ma, and Dinesh Jayaraman. Zeromimic: Distilling robotic manipulation skills from web videos, 2025. URL https://
238777 23877
Sara Smilansky. The Effects of Sociodramatic Play on Disadvantaged Preschool Children. 1968.
Gemini Robotics Team, Abbas Abdolmaleki, Saminda Abeyruwan, Joshua Ainslie, Jean- -aptistt Alayrac, Montserrat Gonzalez Arenas, Ashwin Balakrishna, Nathan Batchelor, Alex Bewley, Jeff Bingham, Michael Bloesch, Konstantinos Bousmalis, Philemon Brakel, Anthony Brohan, Thomas Buschmann, Arunkumar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Christine Chan, Oscar Chang, London ceoapeelett-olpinng Jose Enrique Chen, Xi Chen, Hao-Tien Lewis Chiang, Krzysztof Choromanski, Adrian Collister, David B. D' Ambrosio, Sudeep Dasari, Todor Davehev, Meet Kirankumar Dave, Coline Devin, Norman Di Palo, Tianli Ding, Carl Doersch, Adil Dostmohamed, Yilun Du, Debidatta Dwibedi, Sathish Thoppay Egambaram, Michael Elabd, Tom Erez, Xiaolin Fang, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Ruiqi Gao, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Oliver Groth, Agrim Gupta, Roland Hafner, Steven Hansen, Leonard Hasenclever, Sam Haves, Nicolas Heess, Brandon Hemaez, Alex Hofer, Jasmine Hsu, Lu Huang. Sandy H. Huang, Atil Iscen, Mithun George Jacob, Decpali Jain, Sally Jesmonth, Abhishek Jindal, Ryan Julian, Dmitry Kalashnikov, M. Emre Karagozler, Stefani Karp, Matija Kecman, J. Chase Kew, Donnie Kim, Frank Kim, Junkyung Kim, Thomas Kipf, Sean Kirmani, Ksenia Konyushkova, Li Yang Ku, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Tuan Anh Le, Isabel Leal, Alex X. Lee, Tsang-Wei Edward Lee, Guy Lever, Jacky Liang. Li-Heng Lin, Fangchen Liu, Shangbang Long, Caden Lu, Sharath Maddineni, Anirudha Majumdar, Kevis-Kokitsi Maninis, Andrew Marmon, Sergio Martinez, Assaf Hurwitz Michaely, Niko Milonopoulos, Joss Moore, Robert Moreno, Michael Neunert, Francesco Nori, Joy Ortiz, Kenneth Oslund, Carolina Parada, Emilio Parisotto, Amaris Paryag, Acorn Pooley, Thomas Power, Alessio Quaglino, Haroon Qureshi, Rajkumar Vasudeva Raju, Helen Ran, Dushyant Rao, Kanishka Rao, Isaac Reid, David Rendleman, Krista Reymann, Miguel Rivas, Francesco Romano, Yulia Rubanova, Peter Pastor Sampedro, Pannag R Sanketi, Dhruv Shah, Mohit Sharma, Kathryn Shea, Mohit Shridhar, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Rachel Sterneck, Ian Storz, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Saran Tunyasuvunakool, Jake Varley, Grace Vesom, Giulia Vezzani, Maria Bauza Villalonga. Oriol Vinyals, René Wagner, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Chengda Wu, Markus Wulfmeier, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Jimmy Yan, Sherry Yang, Skye Yang,
15

---

Published as a conference paper at ICLR 2026
Yuxiang Yang. Hiu Hong Yu, Wenhao Yu, Wentao Yuan, Yuan Yuan, Jingwei Zhang, Tingnan Zhang, Zhiyuan Zhang, Allan Zhou, Guangyao Zhou, and Yuxiang Zhou. Gemini robotics 1.5: Pushing the frontier of generalist robots with advanced embodied reasoning, thinking, and motion transfer, 2025. URL
03342
Google Gemini Team. google-gemin//cookooo:: Examples and guides for using the gemini api. 2025.
Mel Vecerik, Carl Doersch, Yi Yang. Todor Davchev, Yusuf Aytar, Guangyao Zhou, Raia Hadsell, Lourdes Agapito, and Jon Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5397-5403. IEEE, 2024.
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models, 2024. hRttss::///axx www.aaiwwww..aaaseeeaaaeeeaaeeae abtrg/abs/23100 01361
Shengjie Wang, Jiacheng You, Yihang Hu, Jiongye Li, and Yang Gao. Skil: Semantic keypoint imitation learning for generalizable data-efficiem manipulation. arXiv preprint 2tt::25501.444000 2025.
Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. You only demonstrate once: Category- level manipulation from single visual demonstration, 2022. URL/attps://arxiv org/abs/ 22C1. 12716
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023.
Amber Xie, Rahul Chand, Dorsa Sadigh, and Joey Hejna. Data retrieval with importance weights for few-shot imitation learning. In 9th Annual Conference on Robot Learning, 2025. URL
https: http:///aa.vvvorgg../.. org/ wwwp/rt.///..//oo.///..//te.seet aa/aa//s////55 25009 225990016797777
Alan Yu, Ge Yang. Ran Choi, Yajvan Ravan, John Leonard, and Phillip Isola. Learning visual parkour from generated images. In 8th Annual Conference on Robot Learning, 2024.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03955 03955. 2024.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming- Hsuan Yang. Telling left from right: Identifying geometry-awe semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3076-3085, June 2024.
Haibo Zhao, Yu Qi, Boce Hu, Yizhe Zhu, Ziyan Chen, Heng Tian, Xupeng Zhu, Owen Howell, Haojie Huang, Robin Walters, et al. Generalizable hierarchical skill learning via object-centric representation. arXiv preprint arXiv:2510.21121, 2025.
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fin--grained bimanual ma- nipulation with low-cost hardware, 2023. URL
http:/bbs/20300 2304. 117055
Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Seyed Kamyar Scyed Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity. In 8th Annual Conference Robot Learning, 2024.
Zhiyuan Zhou, Pranav Atreya, Abraham Lee, Homer Walke, Oier Mees, and Sergey Levine. Autonomous improvement of instruction following skills via foundation models, 2024. URL
https://arxiv. org/ abs /200 .26655
Junzhe Zhu, Yuanchen Ju, Junyi Zhang, Muhan Wang, Zhecheng Yuan, Kaizhe Hu, and Huazhe Xu. Densematcher: Learning 3d semantic correspondence for category-level manipulation from single demo. International Conference on Learning Representations (ICLR) 2025, 2024.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 TRAJECTORY WARPING
Pseudocode. In Algorithm 1
we describe our policy method in detail.
Alternative Waypoint Extraction Mechanisms. In our main results, we perform waypoint extraction by identifying frames where the gripper open/close position toggles. However, Tether is agnostic to the specific choice of waypoint extraction method, which can be readily exchanged for alternatives.
For illustration, we propose the example task of "Pouring from Cup to Bowl," which is difficult because (1) it requires waypoint at the pouring location, which cannot be extracted via gripper change as the gripper is holding the cup, and (2) the keypoint used for corresponding the bowl does not lie at the gripper position, which hovering above the bowl. In this case, we instead query VLM to select keyframes from the demo video and point to the 2D pixel for the waypoint; then, we triangulate the pixels from both views compute the waypoint. Finally, we continue the original algorithm to run correspondence and warp the trajectory. We empirically verify this alternative mechanism in real world setup, where our method successfully identifies waypoints for the cup and bowl and achieves 90% success.
Correspondence Filtering. As explained in Section 3.1 our primary method of filtering out invalid demo correspondences is triangulation: if the backprojected correspondences do not intersect within a threshold of 10 centimeters, then we deem the demo as an infeasible match for the observation o.
However, we find in practice that incorrect correspondences may have a valid triangulation purely by coincidence, thereby passing the feasibility check. In the worst case, this can cause dangerous collisions. As our robot setup lacks built-in collision avoidance, we opt to build simple custom safety layer via the fast multi-view correspondence algorithm MAST3R (Leroy et al. (024). For both the demo and observation o, match each keypoint from its view to the same scene in the other camera view (left to right camera, and vice versa). We then compute the distance of the 3-D waypoint to the ray projected by this match, and if the distance differs between the demo and observation by over 10 centimeters, we deem this correspondence as infeasible as well.
Post- Warp Speed Adjustment. Another key consideration in implementing trajectory warping is the difference in relative waypoint positions Wi, Wt+1 between the demo and warped trajectory: if there is large position difference (e.g. objects closer in demo but farther apart in observation) and we keep the number of intermediate timesteps constant, then the resulting warped trajectory has a large velocity difference as well, which can be dangerous.
Thus, after the warping process described in Section 3.1 we recompute intermediate timesteps such that velocity remains the same between demo and warped trajectories. Specifically, we scale the number of intermediate keypoints T Tsegment by the are length L of the new action plan segment à relative to the are length L of the original segment a,, Teggment = (Ż/L)I L)T Then, we compute new intermediate actions following the relative distribution of the original intermediate actions: for original intermediate action a, we map timestep L/7 s/Tsegment to 8/ T. where & is the are length from we to a, Then, we interpolate along this function to map new timesteps 1/T t/Tuegmem to 8/ Ż, and compute the resulting intermediate action as the position with are length to Wt.
A.2 AUTONOMOUS PLAY
Pseudocode. In Algorithm
2
we describe our play procedure in detail.
Multi-Task Design. Our 6 tasks for autonomous play, plotted in Figure 6 are as follows:
1. Place pineapple from table to shelf.
2. Place pineapple from shelf to table.
3. Place pineapple from table bowl.
4. Place pineapple from bowl to table.
5. Place bowl from table to shelf.
6. Place bowl from shelf table.
17

---

Published as a conference paper at ICLR 2026
Algorithm Trajectory Warping with Keypoint Correspondences
1: Require: Observation Demos
2: for (o,, W,, a,) do
3: // Find keypoint correspendences between demo and observation
4: K, + findCorresponeences(K,, 0,
5:
// Backpro ject correspondences to compute waypoints
6: W, backprojet(Stereo(KK feff Kigright)
7: end for
8: // Select the closest-matchin demo
9: 1" < arg min; score;(o)
10: W W W", + , a,
11: // Warp each action segment
1 12: for t - do
13: // Compute waypoint displacements at start and end segment
d + th - W1 d-1 + u441 Wi+1 for a E ai do
15:
16:
17:
// Compute linear interpolation in space
18:
wi-1 wi
19:
(a wi) v/ |ull da (1 a)d: ad44
20:
à <- d.
22:
end for
23:
jā a E at]
24:
// Compute post-warp speed adjustment
25:
adjustSpeed(ā.)
26:
end for
27:
Output: Warped action plan = [ā], .2....]
Algorithm 2 Autonomous Functional Play with Vision-Language Models
1: Require: Tether Policy म, Task Library T, Demos (D.)ee, Success Detector S, Task Planner VLMptanner. Success Evaluator VIMeevauvtor
2: Hyperparameters: Demo subsample size
3: // Initialize generated trajectory set Go for each task f
4: (G, 0her
5: while TRUE do
6: Receive observation 0
7: // Select task ftarg to target
8:
frarg ~ Softmax(-lGel)
9:
// Generate plan for taarg attempt first task
(L....) = VLMptanner kaarg)
11:
1/ Select & demos for I using UCB
12:
D + TopK, (UCB(K.) K; E D:)
13:
// Run Tether to select deme n* and produce rollout
T = (D,,o)
15:
// Check success and update UCB
16:
if S(, x*, VI.Mevaluator) then
17:
// I successful, add to generation set G.
18:
G. Gr U (T)
1+1
20:
else
21:
r+- 0
22:
end if
23:
UCB(R") + UydacCUCB(UCB((").))
24:
end while
25:
Output: Generated trajectories (Gr) 1eT
18

---

Published as a conference paper at ICLR 2026
This set of tasks possesses the approximately indefinitely composable structure described in Section 3.2 regardless of execution success or failure, the pineapple will be at the table, shelf, or bowl, and the bowl will be at the table shelf. Then, for any of these combinations start state, there is at least one task from the list above that valid.
VLM Queries. Below, we present the VLM prompts and example responses from play procedure.
Analyza this robat remment, coptured from left and right viexs. Rased this please create an action plan that anda with the taak: <taak>, If the task 1s ready discupt
achieved
scene, you and then plan to achieve the task.
achieved anymore,
The fol lowing atomic actions are available: <actions>
Please provide your final answer in the following format: "yjson
"dascription"; "cdescription tha scene>", *actions";
"action": "<action execute>", Baetion_leitee". *elist letlers", "reasoning": "<reasoning actions*
Listing I: Task Plan Generation.
Json
ddscrription' "the scene shows a robot arm positioned zear a table covered with a checkared tableclath. the table, there A wooden shelf unit and cabinet are visible in the background.""
bowl containing toy pineapple.
Vactions";
"action": "pick up pineapple from bowl on Lable and place on table", "sction_lettes" "d", "reasoning": "The goal is to pick up the pinespple from the table and place it in the pineapple already inside the bowl. To make the Lask achieeable, we aust first diszupt the curcent state by removing pinespple
the
"action": "pick pinespple from table and place in bowl table", "action_letter letter"; "c", "reasoning"; "Acter removing pineapple
placing table, we execute the final task of picking up the pineapple from the table placing into
Listing 2: Task Plan Generation Example Response.
For this task, you will 1. see robot trying to perform the task of <tasks, You camera views of the same scene. Some cameras aze static and are mounted outaide of the mounted the robot and thus they moving during the
molt maltiple
and episode.
first images
episode (some time agol. The last 3 inages show multiple camera views from the current moment in the episode (as it new).
multiple cameza
start
Looking these images and comparing the start of the eplaode with current state did the robot successfully pezform the task *ctasks"? Make sure that the task is completed vithout disrupting other objects in the
please think step by and provide your reasoning. Then, provide your final answer in the lowing format:
19

---

Published as a conference paper at ICLR 2026
*ceasoning": creasoning for evaluationn *completed"; <true or false>
Listing 3: Success Evaluation.
jaonn
"reasoning": bS-ep- analyals:\nl. **titiaal State Analysis (Images 1-3) The pink bowl is located on the checkezed suzface (table areal in front of the wooden shelf unit. The middle shelf of the wooden unit visible and does An2. axterrent State Analynis (Imagen 4-61 :s2 The pink bowl is no longer on the checkered surface. Tastead, it The robct arm's gripper is positioned near the bowl on the shelf, indicating it has aither just planed
the
placed the middle ahelf the wooden unit.
the process releasing it. **Task Rvaluat ionser The Lask was 'pick bowl from table and place it on middle ahelf', The comparisen between the initial and current images confirms that the bowl has been successfully moved frem the table the middle shelf, No other objects appezr have been digrupted during this proceas.", completed";
Listing 4: Success Evaluation Example Response.
Correspondences for Success Evaluation As explained in Section 3.2 we simply query Gemini Robotics-FR 1.5 to determine success. However, before the release of this model, we required an additional mechanism that reduces false positive rates by double-checking the VLM evaluation with a correspondence-basee heuristic: from the final camera image Jinat of the source demo k* that produced this trajectory, we look for correspondences of its final keypoint fimal in the Tether execution's final image Ofinat- If one (or both) of the rays projected by the best-matched point from either camera view is far from the executed gripper position (beyond preset threshold of 10 centimeters), we mark the trajectory failure. Otherwise, if both the correspondence and VLM response are positive, then the trajectory is a success.
This correspondence--aeeee verification necessary avoid false positive evaluations from our initial experiments using GPT-4.1, but is now largely redundant. Following the procedure outlined Section 4.3 our post-hoc evaluation of Gemini Robotics-ER. 1.5 without correspondence based verification reaches 98.4% precision (at 89.6% recall), surpassing our previous results with GPT- 4.1 and the correspondence-baat based verification, which reaches 97.7% precision (at 87.7% recall). Similarly, Gemini Robotics-FRR 1.5 achieves 95.2% accuracy for task planning, surpassing our previous results with Gemini-2.5 Flsaalsh,,, which achieves 94.1%. Thus, while our play experiments were initially conducted with this mechanism and past VLM models, it is strictly more advantageous to simply adopt the most recent VLM advancements.
necessitating failures over 26 hours, the key failure mode of Tether caused by the bowl flipping completely upside-down. It is nearly impossible to recover from this state with only single arm, un-flipping the bowl requires of contact. Besides such cases, other failure modes that we discovered and resolved in previous iterations of play experiments include crashing into the furniture, which we minimize via a simple boundds check, and dropping objects to the floor, which we avoid by placing barriers on the boundary of the robot's 's workspace.
Failure Modes. While only had 5 entervention-nccessstttnngn
separate points
A.3 EXPERIMENTAL SETUP
Diffusion Policy (DP) Setup. For training and deploying all diffusion policies, we use the codebase provided by DROID Khazatsky al. 2024), following the original hyperparameters with a few modifications. Specifically, we set the batch size to 32, the shuffle buffer size to 1000, and training epochs to 600. For each task. we ablate the choice of camera views given to the policy: "Pineapple to Shelf" and "Pineapple from Shelf" use the left and right external cameras, while the rest use the wrist and right external cameras.
20

---

Published as a conference paper at ICLR 2026
Keypoint Action Tokens (KAT) Setup. Keypoint Action Tokens (KAT) receives series of teleop- erated robot demonstrations, encodes the initial visual observation and actions into a pair of input and output tokens, and feeds the demo tokens and current visual observation to a LLM to generate trajectory in the new scene. We find that KAT failed to achieve any successes on our tasks, despite ablating its three key components: keypoint tokens, action tokens, and the LLM used for queries. We review these attempted fixes below.
1. Keypoint tokens are extracted by extracting visual descriptors using DINO-ViT, extracting correspondences using Best-Buddy Nearest Neighbor matching between the descriptors of two initial observations in the imput set of demonstrations, and then finding the corresponding location of each selected visual descriptor in the new scene. However, since our camera views capture the entire workspace, we find that these visual tokens often fall on parts of the scene that are constant across demos but are not relevant to the demonstrated task (e.g.. cabinet in background); in contrast, locations that are crucial to track for the current task (e.g., pineapple) are not tracked by any of the descriptors. Thus, KAT may be susceptible to visual distractors. While the original paper experiments with distractors at test time, they are not present in the demos, and thus do not affect visual description extraction. To address this limitation, we manually select the relevant keypoints track in an image. For example, for the task of picking up pineapple and putting it into a bowl, we annotate keypoints on the location of the pineapple and bowl. However, we ultimately find that even with only the task-relevant keypoints marked, the output trajectory is not accurate.
2. In the original paper, KAT records actions at 4 Hz. We experimented with various frequencies between Hz and 15 Hz, but find that none work well. Additionally, we observe that after the frequency increases beyond 10 Hz, the model frequently outputs the same trajectory, regardless of the keypoint locations for the input visual observation.
3. KAT originally uses GPT-4-Turbo to generate the new trajectory. We tested GPT-4-Turbo, but found that it often outputs invalid action sequences inaccurate actions. We also tested with G Gemini-2.5-Fllshh and Gemini Robotics-ER 1.5 and find that while these models output validly formatted action sequences, the output action sequences are accurate behaviors.
In summary, believe that KAT's limitations are due to the failure of LLMs to handle multi- dimensional numerical patterns in our trajectories, caused by orientation changes, non-linear veloc- ities, and wide object distributions. To verify this hypothesis, we run additional experiments with KAT on two tasks with simpler trajectory distributions: lifting the lid handle of a coffee machine, and opening a drawer. KAT achieves 70% and 60% (while Tether achieves 90% and 100%), which confirms that it indeed works well on more local object-centric interactions. This validates that while few-shot imitation baseline succeeds on certain simpler trajectory distributions, Tether excels handling more diverse, challenging trajectories.
Finetuned To Setup. We finetune To using the official codebase and checkpoints provided by Physical Intelligence. We finetune the To-FAST-DROID checkpoint for 1000 steps on the set of 10 human collected demonstrations we provide to Tether. During training and evaluation, we set the language prompt to the corresponding action description used in our VLM queries during play. We observe that training the model on set of 10 demonstrations collapses the model, as it often does not move or misses the target object, while the model running zero-shot inference achieves nonzero success rates on the same tasks. However, to validate that this setup does indeed work for mo finetuning, we perform additional experiments finetuning mo Tether-generan generated data, with many more demos than the 10 human proanddrovided ones for the original baseline. Details and results are in Section A.5 below.
Tether-
A.4 EXPERIMENTAL RESULTS
Policy Comparison. In Table T report the exact figures from Figure along with standard errors.
Intermediate Tether Visualizations. We visualize intermediate steps of our Tether policy from the evaluations in Section [42 Specifically, include annotations for semantic correspondence, where the red dot indicates the correspondence match between source (demo) and target (scene), and the blue dot is the MAST3R match between viewpoints. Additionally, we include few visualizations of
21

---

Published as a conference paper at ICLR 2026
Method
Pineapple from Shelf
Pineapple Bowl
Bowl to Shelf
Bowl from Shelf
Tether (10 Demos)
1.0 + 0.0
1.0 + 0.0
0.9 + 0.09
1.0 0.0
Tether (5 Demos)
0.8 + 0.13
0.8 = 0.13
1.0 0.0
1.0
Tether (1 Demo)
0.8 + 0.13
0.9 + 0.09
0.7 0.14
1.0
Black et 2024
0.16
0.7 0.14
0.15
DP
Chi et al. 2023
0.0 0.0
0,0 0.0
0.0 - 0.0
0.1
Method
Apple to Bowl
Strawberry to Bowl
Pineapple to Basket
Pineapple Cup
Tether (10 Demos)
0.9 + 0.09 1.0 0.0
0.9 + 0.09
0.9 + 0.09
0.9 + 0.09
Tether (5 Demos)
0.9 + 0.09
0.9 0.09
0.9 0.09
Tether (1 Demo)
+ 0.09
1
0.9 0.09
0.5 0.16
Wog
Black et al.
2024
0.6 + 0.15
0.7 + 0.14
0.4 - 0.15
- 0.09
DP hi ct al.
2023
0.0 + 0.0
0.0 0.0
0.0 + 0.0
0.0 0.0
Method
Wiping with Cloth
Opening Cabinet
Hanging Tape
Inserting Coffee
Tether (10 Demos)
0.6 + 0.15
0.6 + 0.15
0.7 0.14
0.4 0.15
Tether (5 Demos)
0.0
0.3 i 0.14
0.14
0.13
Tether (1 Demo)
0.1 + 0.09
0.1 + 0.09
0.09
0.2 + 0.13
Black ct al.
2024
0.0 + 0.0
0.0 + 0.0
0.0 + 0.0
0.0 + 0.0
DP
hu et al.
2023
0.0 + 0.0
0.0 + 0.0
0.0 1 0.0
0.0 1 0.0
Table 1: Success rates and standard error measured over 10 trials.
source
target
source
target
Figure 9: Computed correspondence for the Pineapple to Cup task.
our trajectory warping. Note that the demonstration depicted is the one selected by our method as the most similar to the scene.
In Figures and and TO we visualize examples of successful semantic generalization: from the pineapple to strawberry and bowl to cup. These demonstrate the key ability of semantic correspondence to generalize beyond the demonstration, enabling interactions with out-of-distribution scenes.
In Figures 11 and we visualize examples of successes in hanging tape and wiping the whiteboard with cloth. The former shows the accuracy of correspondence even with small objects like the silver hook, and the latter demonstrates correspondence with deformable, non-rigid objects like the cloth.
Finally, in Figures 13 and 14 we visualize examples the full warped trajectory, with the source demo on the left, the warped result the right, and annotated waypoints along the trajectory.
22

---

Published as a conference paper at ICLR 2026
source
target
source
target
Figure 10: Computed correspondence for the Strawberry to Bowl task.
source
target
source
target
Figure 11: Computed correspondence for the Hanging Tape task.
23

---

Published as a conference paper at ICLR 2026
source
target
cource
target
Figure 12: Computed correspondence for the Wiping with Cloth task.
demo
Figure 13: Trajectory warping for the Pineapple Bowl task.
demo
warp
Figure 14: Trajectory warping for the Wiping with Cloth task.
24

---

Published as a conference paper at ICLR 2026
Figure 15: Environment states with the flipped bowl (left) pineapple pushed into shelf (right) that required manual interventions.
Method
Pineapple from Shelf
Pineapple Bowl
Bowl to Shelf
Bowl from Shelf
Tether
1.0 + 0.0
1.0 + 0.0
0.9 = 0.09
1.0 - 0.0
KALM
0.6 = 0.15
0.0 + 0.0
0.0 0.0
0.5 = 0.16
Table 2: Policy Comparison with KALM. Success rates and standard error measured over 10 trials.
We record a subsection of our 26-hour long autonomous play experiment. The timelapse, sped up by 100x, can be viewed github. io
Play Timelapse.
website at
reserrch
Play Interventions. Among our 5 interventions, 4 involved the bowl flipping upside-down, and 1 involved the pineapple being pushed too far back into the shelf. While the latter situation occurs many times during play, this single instance is edge case that could not be resolved autonomously via the Tether policy. Both situations are shown in Figure 15
A.5 EXTENDED EXPERIMENTAL RESULTS
Finally, we include additional experimental results regarding various aspects of our method.
Additional Baselines. To isolate and assess the importance of trajectory warping, we compare Tether with KALM (Fang et al., (225)), which also uses keypoint correspondences as input but produces actions via an imitation learning policy. Given 10 demos, our method outperforms KALM on our four pineapple-and-bor bowl tasks, shown in Table 2 In general, observe that KALM struggles with spatially generalizing to our large space of object poses, as its actions are predicted by an imitation learning policy, whereas Tether uses trajectory warping. Still, it does sometimes succeed at "Pineapple from Shelf" and "Bowl from Shelf." likely because the objects are randomly initialized on the smaller shelf; conversely, it fails to complete the other two tasks, where objects are randomized across the larger table. These results suggest that even with keypoint inputs, imitation learning falls behind trajectory warping in the low data regime, where few demos usually cannot densely cover the initial object distribution and thus require the policy to have more structured generalization (c.g. via warping).
Additional Environments and Tasks. To evaluate the applicability and generalization of Tether, we run two additional experiments for "Pineapple Bowl" in a mock kitchen setup and an office setup, with structural variation; in the former, we move the pineapple from the sink to bowl on the counter, and in the latter, we move the pineapple from the top of filing cabinet to a bowl inside a drawer. Our policy achieves 100% and 90% success respectively, thus validating that our method works well across different geometrically diverse environments, as well as spatial and semantic object variations.
Next, we assess policy's 's potential to semantically generalize to unseen objects for more difficult tasks, beyond those reported in Figure 5 (row 2). Specifically, we run our policy for "Lifting the Handle" of coffee machine: with 10 demos and evaluation new unseen machine, our policy achieves 80% success, thus demonstrating semantic generalization for a more complex task involving delicate articulation.
Finally, we evaluate our policy's 's ability to disambiguate between semantically similar objects: given 10 demos of "Pineapple to Bowl," "Strawberry to Bowl," and "Apple to Bowl," and evaluated in
25

---

Published as a conference paper at ICLR 2026
Method
Pineapple from Shelf
Pineapple Bowl
Bowl to Shelf
(10 human demos)
0.0 + 0.0
0.0 + 0.0
0,0 1: 0.0
To (50 Play-generates demos)
0.1 = 0.09
0.2 + 0.13
0.1 = 0.09
To (100 play-generated demos)
0.3 = 0.14
0.5 0.16
0.1 0.09
To (all play-generated demos)
0.7 + 0.14
0.8 0.13
0.4
mo ppte-trained)
0.5 = 0.16
0.7 0.14
0.0 0.0
Table 3: Downstream To Finetuning Results. Success rates and standard error measured over 10 trials.
multi-object setting with all three fruits present the table, Tether achieves 100%, 80%, and 100% success rates respectively (with no failures caused by targeting the wrong object). Thus, we confirm that in cluttered multi-object environments, Tether can still accurately identify the correct object to manipulate.
Downstream To Finetuning. Following similar setting Figure 7 we validate the effectiveness of our play-generated data in improving pretrained VLA policies like Tu- Results are reported in T In all three tasks, we observe consistent improvement as we include more data. We observe that while both baseline pre-trained To and finetuned with few demos often miss the object and become stuck and idle, mo finetuned with more demos is able to correctly move toward the object, and in case of misses, sometimes even successfully retry. This validates that Tether-generated data is also useful for improving the performance of large pre-trained policies.
26


Reviews:
[{"review_id": "9oldYDJiWX", "reviewer": "Reviewer_3dVV", "summary": "The paper proposed 1) a non-parametric table-top manipulation policy by warping source demonstrations, which has strong generalization performance, and 2) a framework for autonomously generating robot demonstrations with minimal human interventions. The proposed method is evaluated on 12 table-top manipulation tasks. The first part of the experiments shows the proposed non-parametric policy outperforms three baselines. The second part of the experiments shows the proposed framework can continuously gather demonstrations with minimal human interventions, and the downstream policy performance scales well with the autonomously-collected demonstrations.", "questions": "- The autonomous play framework proposed in Section 3.2 appears to be policy-agnostic. Can we use the same framework with different policies?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important problem - scaling data collection. Through the experiments, the paper shows that it enables collection of a large number of robot demonstrations with minimal human interventions, and the collected demonstrations are of comparable quality to human demonstrations in terms of downstream policy learning performance.\n- The paper is clear.", "weaknesses": "- The baselines for Section 4.2 are not fair comparison, and thus weakens the claim that the proposed Tether policy has superior generalization. Concretely, Tether is given 10 demonstrations, while $\\pi_0$ with FAST tokenizer is evaluated zero-shot. Diffusion policy, not meant for few-shot setting, is also disadvantaged by being trained from scratch with 10 demonstrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YWJ2x2HG60", "reviewer": "Reviewer_ZDhN", "summary": "This paper presents Tether, a framework for autonomous robotic play that aims to reduce dependence on large-scale human demonstrations for imitation learning. The method combines a non-parametric, correspondence-driven policy for robust generalization with a vision-language model (VLM)-guided autonomous play loop that continuously generates new training data.\n\nThe core insight is to leverage semantic image correspondences between demonstrations and novel scenes to warp existing robot trajectories. This approach enables a robot to adapt to a small set of demonstrations (as few as 10) in new environments and tasks without requiring retraining. The authors then deploy this policy in a self-improving cycle of multi-task play: the robot autonomously selects, executes, and evaluates tasks using VLM reasoning for both planning and success detection.\n\nOver 26 hours of autonomous operation, Tether produces over 1,000 successful real-world trajectories with negligible human intervention (~0.26% of runs requiring resets). The collected data are subsequently used to train downstream neural policies, which achieve success rates comparable to or better than policies trained on human demonstrations.", "questions": "See the points above. I am eager to see the paper get improved.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a clever and practical way to reuse a small number of demonstrations via keypoint correspondence-driven trajectory warping. This method leverages advances in visual correspondence (e.g., DINOv2 + Stable Diffusion) to enable generalization across large spatial and semantic variations without retraining.\n\n- Experiments on 12 diverse manipulation tasks—including deformable, articulated, and precision tasks—show consistent improvement over baselines such as Diffusion Policy, ε₀ (OpenVLA), and KAT. The approach exhibits impressive semantic generalization, successfully adapting demonstrations from, e.g., a pineapple to an apple or a bowl to a cup", "weaknesses": "- One major contribution, at the same time, can be a limitation as well, is the usage of correspondence. While correspondence-based warping enables generalization, its performance is tied to the quality and stability of keypoint matching. In cases with heavy occlusion, textureless surfaces, or large deformations, this component may degrade. It would be nice for the authors to include such discussions in the paper on which external disturbances the policy is robust to, and which disturbances it cannot handle.\n\n- Non-parametric vs. generalization claim: The paper argues that a non-parametric policy generalizes to unseen objects and layouts. However, this non-parametric methods always require access to demonstrations at inference time, which limits scalability and contradicts the claim of “extreme generalization”. It would be helpful for the authors to explain this better in the text.\n\n- Clarification question: How does the relatively coarse 10 cm correspondence threshold and the potential variability of VLM-based verification affect the reliability of results, especially for precision-sensitive tasks like coffee pod insertion with an 8 mm margin?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNcH6p8Vjm", "reviewer": "Reviewer_QvvQ", "summary": "This paper introduces Tether, a system for autonomous robotic manipulation that addresses the data scalability bottleneck in imitation learning through two main contributions: (1) a novel non-parametric policy that uses semantic visual correspondences to warp demonstrated trajectories into new scenes, achieving robust generalization across spatial and semantic variations with few demonstrations per task, and (2) a VLM-guided autonomous multi-task play procedure that enables the robot to continuously generate training data over extended periods (26 hours, producing 1000+ expert-level trajectories) with minimal human intervention. The key insight is that by leveraging strong visual priors from correspondence models rather than requiring massive demonstration datasets, the system can bootstrap from a few human demos to autonomously produce diverse, high-quality data that trains downstream neural policies to performance competitive with human-collected demonstrations, thereby shifting the scaling bottleneck from human time to robot time.", "questions": "- In the **Warping the Source Demo Trajectory** section, you describe interpolating 6-DOF gripper poses between waypoints, but it is unclear how the binary gripper open/close commands are handled during this warping process.\n\n- For the **Out-of-Distribution Fruit and Containers** evaluation, is this transfer zero-shot (e.g., no strawberry demonstrations provided at all), or were OOD demos included in the given demonstrations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles an important problem in robot learning by demonstrating that autonomous play can generate over 1000 expert-level trajectories across 26 hours with minimal human intervention, offering a compelling alternative to labor-intensive human demonstration collection.\n- The correspondence-driven trajectory warping policy demonstrates impressive robustness, outperforming strong baselines including vision-language-action models ($\\pi_0$) and LLM-based methods (KAT).\n- As evidenced in Fig.7, these tasks are reasonably complex that standard Diffusion Policy fails with limited data.", "weaknesses": "- While Fig.7 demonstrates that scaling to 2000 generated demonstrations improves Diffusion Policy performance, the paper lacks crucial comparisons with strong vision-language-action models like $\\pi_0$, $\\pi_{0.5}$ when provided with equivalent amounts of data. This comparison is critical because if VLAs can achieve high success rates with significantly fewer demonstrations (e.g., <500), it would diminish the practical value of scaling to 1000+ demonstrations per task.\n\n- The current system generates demonstrations for tasks predefined by human experts, rather than discovering novel tasks beyond the initial demonstration scope. Works like BBSEA (arXiv:2402.08212) demonstrate that foundation models can autonomously propose new learnable tasks in unknown environments. While the present work successfully scales data collection for given tasks, extending the framework to autonomously expand the task repertoire (e.g., discovering new object interactions or compositional skills not present in initial demonstrations) would strengthen the contribution and better justify the \"autonomous play\" framing.\n\n- The entire evaluation is conducted in one fixed environment configuration (table with two shelves and specific camera placements). While the method demonstrates robustness to spatial variations (object positions) and semantic variations (OOD objects) within this setup, testing across structurally different environments (e.g., different furniture arrangements, room layouts, or workspace configurations) would better validate the generalizability of the approach and strengthen claims about the method's broad applicability.\n\n- The paper employs VLMs for task planning and success evaluation but provides no ablation studies on these critical components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pvmm0vmUAH", "reviewer": "Reviewer_x1Cz", "summary": "This paper introduces **Tether**, a system that allows robots to learn from \"autonomous play\" instead of human-collected datasets. The Tether method consists of two primary components:\n\n1. A non-parametric manipulation policy that operates using \"correspondence-driven trajectory warping\". Given only a few demonstrations, this policy finds semantic keypoint correspondences between a demo pre-image and the current scene. It then selects the best-matched demonstration and \"warps\" its 3D trajectory to fit the new scene.\n\n2. An autonomous, multi-task \"play\" procedure that uses this policy to continuously generate new data by querying Vision-Language Models (VLMs) for high-level task selection, planning, and evaluation for success.\n\nThe authors demonstrate in real-world experiments that their policy, trained on only 10 demos, outperforms baselines like Diffusion Policy (DP) and VLA models ($\\pi_0$). The policy shows generalization to OOD objects and spatial configurations and succeeds on deformable and high-precision tasks.\nThe autonomous play dataset can further train a Diffusion Policy, which progressively improves and achieves competitive performance.", "questions": "1. In line 182, how are the keypoints \"identified\"? Where are the keypoints in Figure 1? Moreover, the waypoints are determined by locations where the gripper toggles. This heuristic seems specific to pick-and-place. How does this definition apply to tasks like \"Wiping with Cloth,\" where the critical part of the trajectory may not involve gripper state changes?\n\n2. How robust is trajectory warping for articulated objects? If geometry ratios differ (e.g., cabinet openings that are higher/wider/deeper), does the warping generalize? Can it achieve cross-object generalization like [2]?\n\n3. Human demonstrations often contain recovery from mistakes, creating data diversity in the strategy level, which trajectory warping may not capture. Is Tether primarily augmenting background/object pose? How does it compare with open-loop motion planning methods (e.g., RRT)?\n\n4. What are the requirements for the task set? Specifically, what are the criteria to ensure cyclic task graph? How to determine if a task need to be segmented into mutiple subtask?\n\n5. The policy finds the nearest-neighbor demo by computing correspondences against all demos in the set $\\mathcal{D}$. How does the inference time of this policy scale as the number of demonstrations grows?\n\n6. The core contribution of this paper seems to be the \"autonomous data generation\" instead of the policy. However, the paper mainly focus on the performace of the policy. How does its data generation compare to methods like [3,4]? \n\n[1] Haldar, Siddhant, and Lerrel Pinto. \"Point policy: Unifying observations and actions with key points for robot manipulation.\" CoRL, 2025.\n\n[2] Fang, Xiaolin, et al. \"KALM: Keypoint Abstraction Using Large Models for Object-Relative Imitation Learning.\" ICRA, 2025.\n\n[3] Mandlekar, Ajay, et al. \"Mimicgen: A data generation system for scalable robot learning using human demonstrations.\" CoRL, 2023.\n\n[4] Pan, Chuer, et al. \"One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies.\" CoRL. 2025.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Effective Warping Policy**: Using semantic correspondences to directly warp a full trajectory is a data-efficient approach, outperforming the baselines.\n\n2. **Strong Empirical Result**: The policy demonstrates generalization to OOD layouts and objects. The evaluation tasks include challenging ones involving high-precision and deformable objects.\n\n3. **Comprehensive Evaluation**: The paper successfully closes the loop. It shows not only that the policy can generate data, but also that the generated data is useful by training a downstream Diffusion Policy, which achieves good performance.\n\n4. **Clarity and Presentation**: The paper is well-written and easy to follow.", "weaknesses": "1. **Open-Loop Execution**: The policy appears to be entirely open-loop at the trajectory level. It computes correspondences, generates a full \"action plan. Without a straightforward adjustment to closed-loop execution, this may create limitations on task complexity and horizon.\n\n2. **Task Structure**: The authors claim the play method has \"approximately indefinitely composable\" structure. However, it is achieved on a deliberately designed, table-top task set. For tasks that are not easily invertible (e.g, washing dishes), the system may not be able to traverse a known, cyclic task graph. \n\n3. **Baseline Comparisons**: The policy takes keypoints as inputs, which is mostly compared with baselines taking raw pixel images as inputs. Such comparisons could not effectively evaluate the performance of warping vs. imitation learning. Although the author includes KAT as one of the baselines, it is not effectively reproduced. Some simpler keypoint-based policies [1,2] could be included as baselines.\n\n4. **Analysis of Failure Modes**: Given that the core idea of the paper is to make the robots play \"autonomously\", the analysis of the failure modes is more important, which is not covered in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMVW7cnCFP", "reviewer": "Reviewer_CA2Q", "summary": "This paper presents Tether, a system for autonomous robotic play that learns from its own interaction data without human supervision. The method combines a non-parametric correspondence-driven trajectory warping policy—which warps demonstration trajectories to new scenes using semantic keypoint correspondences—with a VLM-guided self-improvement loop that autonomously selects, executes, and evaluates tasks. Starting from only a few human demonstrations, Tether repeatedly performs tasks, identifies successes using a combination of geometric and language-based evaluation, and accumulates new high-quality trajectories for imitation learning. Experiments across 12 real-world manipulation tasks on a Franka Panda arm show strong few-shot performance, robust generalization to unseen objects, and continuous improvement through autonomous play. The system autonomously collects over 1,000 expert-quality trajectories in 26 hours, enabling downstream neural policies to match or surpass those trained on human-collected data.", "questions": "please address the concerns above", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Tether introduces an elegant and practical approach to autonomous play that avoids the dependence on large-scale pretrained policies or foundation models. The key technical idea, trajectory warping from visual correspondences, is conceptually simple but powerful, enabling robust generalization from as few as 10 demonstrations. The geometric formulation preserves spatial precision while providing semantic flexibility, allowing the method to generalize to unseen objects and scene layouts.\nThe system-level design is well thought out: integrating VLM-based task selection and success evaluation into a continuous play loop demonstrates genuine autonomy rather than mere offline augmentation. The experiments are impressively thorough, spanning spatial, semantic, and dynamic challenges, and even covering deformable and articulated manipulation. The real-world deployment shows convincing scale and stability, with near-zero human intervention over long periods. Finally, the downstream imitation learning results clearly validate that the autonomously generated data is useful and progressively improves learned policies, providing a strong empirical story of self-improving robotics.", "weaknesses": "1. The trajectory warping mechanism makes a strong simplifying assumption of linear spatial interpolation between matched waypoints, ignoring the coupled effects of gripper orientation, contact geometry, and dynamic feasibility. This can distort end-effector poses for curved or articulated motions such as “Bowl to Shelf” or “Wiping with Cloth,” yet the paper provides no analysis or constraint mechanism to prevent physically invalid interpolations.\n\n2. The correspondence-based matching assumes precise stereo geometry and reliable keypoint alignment, but the paper does not quantify how correspondence errors propagate to 3D waypoint accuracy. Since failed backprojections are simply discarded as “infeasible matches,” it is unclear how often this occurs or how sensitive performance is to visual conditions like lighting or occlusion.\n\n3. The autonomous play loop relies heavily on VLM reasoning for both task sequencing and success evaluation, but the paper treats these modules as black boxes. There is no quantitative assessment of VLM reliability, error rates, or false success detections, which makes it difficult to judge how robust the overall autonomy pipeline truly is.\n\n4. Lastly, while the downstream policy improvements are impressive, the analysis does not disentangle quantity from diversity of collected data. Without measuring coverage or novelty of the self-generated trajectories, it remains unclear whether improvement arises from meaningful exploration or simply repeated collection of similar trajectories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)