Flow as the Cross-domain Manipulation Interface
Authors: Mengda Xu
Zhenjia Xu
Yinghao Xu
Cheng Chi
Gordon Wetzstein
Manuela Veloso
Shuran Song
Venue: CoRL 2024 (2024)
Keywords: Robots
Learning
cross-domain
cross-embodiment
TL;DR: We present Im2Flow2Act, a scalable learning framework that enables robots to acquire manipulation skills from diverse data sources.

Abstract: We present Im2Flow2Act, a scalable learning framework that enables robots to acquire real-world manipulation skills without the need of real-world robot training data. The key idea behind Im2Flow2Act is to use object flow as the manipulation interface, bridging domain gaps between different embodiments (i.e., human and robot) and training environments (i.e., real-world and simulated). Im2Flow2Act comprises two components: a flow generation network and a flow-conditioned policy. The flow generation network, trained on human demonstration videos, generates object flow from the initial scene image, conditioned on the task description. The flow-conditioned policy, trained on simulated robot play data, maps the generated object flow to robot actions to realize the desired object movements. By using flow as input, this policy can be directly deployed in the real world with a minimal sim-to-real gap. By leveraging real-world human videos and simulated robot play data, we bypass the challenges of teleoperating physical robots in the real world, resulting in a scalable system for diverse tasks. We demonstrate Im2Flow2Act's capabilities in a variety of real-world tasks, including the manipulation of rigid, articulated, and deformable objects.

Flow as the Cross-Domain Manipulation Interface
Mengda Xu 1,2,3 Zhenjia Xu 1.2 Yinghao Xu
Cheng Chi 1.2
Gordon Wetzstein Manuela Veloso Shuran Song 1.2
Stanford University, Columbia University,
3 J.P. Morgan AI Research, 4 Camegie Mellon University https:// flow- aelowwattt..... .ett.githbb,
Abstract: We present Im2Flow2Act, a scalable learning framework that enables robots to acquire real-world manipulation skills without the need of real-world robot training data. The key idea behind Im2Flow2Act is to use object flow as the manipulation interface, bridging domain gaps between different embodiments (i.e., human and robot) and training environments (i.e., real-world and simulated), Im2Flow2Act comprises two components: flow generation network and flow- conditioned policy. The flow generation network, trained on human demonstration videos, generates object flow from the initial scene image, conditioned on the task description. The flow-conditioned policy, trained on simulated robot play data, maps the generated object flow to robot actions to realize the desired object move- ments. By using flow as input, this policy can be directly deployed the real world with a minimal sim-to-real gap. By leveraging real-world human videos and simulated robot play data, we bypass the challenges of teleoperating physi- cal robots in the real world, resulting in scalable system for diverse tasks. We demonstrate Im2Flow2Act' capabilities in variety of real-world tasks, including the manipulation of rigid, articulated, and deformable objects.
Keywords: Robots, Learning, cross-domain, cross-embodiment
1m2Flow. Task-specific generation via real cross-embodee embodiment video
Human@Real
Robot@Sim
(b) Flow2Act. Robot specific generation via simulated exploration data
"War aie fine dee boie""
Flow
Act
6DoF Play Tral.
(c) Im2Flow2Act. Real-robot execution without real-robot training data
Pick Place
Pouring
Cloth Folding
Drawer Opening
Figure 1: Flow as the Cross-domain Manipulation Interface. Im2Flow2Act, we utilize object flow to bridge the domain gap between both embodiments (human v.s. robot) and training environments (real v.s. simulation). Our final system is able to leverage both a) action-less human video for task-conditioned flow generation and b) task-less simulated robot data for flow conditioned action generation, results in language- conditioned multi-task system real-world manipulation tasks without the need real-robot training data.
Introduction
A key step for scaling robot learning is giving robots the ability to learn from various data sources beyond expensive real-world robot data [1, 2, 3, 4, 5]. Prior work has approached this problem from
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

two main directions: learning from cross eoosseembod videos [6. 7, 8, 9, 10, 11, 12, 13] and simu- lated robot data. While both approaches have shown promising progress, they each face significant challenges. Learning from cross-embodiment data is hampered by the large embodiment gap, while simulated data struggles with a significant sim-to-real gap and the complexity to build task-specific environment, especially for contact-rich manipulation tasks.
In this paper, we introduce Im2Flow2Act, a novel framework to learn real-world manipulation skills but without the need of real-world robot training data. Our key idea is to use object flow- the exclu- sive motion of the manipulated object, excluding any background or embodiment -as a unifying interface to connect human videos and simulated data, and achieve one-shot generalization for new skills in the real world. Our framework (Fig. 1) consists of two main components: a flow generation network (Fig. 1-a) trained on cross-embodiment demonstration videos and flow-conditioned policy (Fig. 1-b) trained on embodiment-speeci
cross-environment simulated data. During in- ference, the flow generation model generates the complete task-specific object flow based on the initial visual observation and task description (Fig, 1-c). Generating a complete object flow helps to minimize the visual appearance gap between the training and inference when learning from cross- embodiment demonstration. Conditioned on this generated flow, our task-agnostic policy generates actions to execute the task. We outline the benefits of using object flow as a unifying interface:
Expressive task descriptions: Object flow not only captures changes in object pose but also accounts for articulations and deformations. Its versatility enables it to represent wide range of objects and tasks, including rigid, articulated, and deformable objects. In Im2Flow2Act, we leverage this versatility by training a single manipulation policy for diverse tasks.
Embodiment agnostic: Object flow describes the change in the state of an object caused by an action rather than the action itself, making it independent of the agent's embodiment. Com- pared to prior work [14] utilizing uniform grid flows that also track the embodiment motion, our method uses object flow, making our framework more effective for cross-embodiment learning- Minimal sim-to-real gap: Compared to image-based representations, flow focuses motion rather than appearance, reducing the sim2real gap and aiding generalization. Unlike prior works utilizing flow for manipulation that rely on heuristic policy [15], realworld teleoperation data [14, 16]. or both [17], our closed-loop policy is entirely learned from simulated robot exploration data generated by a set predefined action primitives.
Interpretable: Unlike vector- based state representations, flow-based representation has intuitive physical meaning. This makes it good interface for human-robot interaction. For example, user can easily understand and select a flow when multiple flows exist for a task.
To utilize object flow as an interface to learn from diverse data, our system contains two components:
Flow generation network: The goal of the flow generation network (Fig. 1-a) is to learn high- level task planning through cross-embodiment videos, including those of different types of robots and human demonstrations. We develop langunge-conditiones flow generation network built on top of the video generation model Animatediff [18]. Compared to prior approaches [14, 15, 17] that train on the original flow space, we leverage the autoencoder (AE) from Stable Diffusion [19] to first compress the flow into latent space, and then train the model on that compressed representation, making our training process more efficient.
Flow-conditioned policy: The goal of the clw-conditioned imitation learning policy (Fig. 1-b) is to achieve the flows generated by the flow generation network, focusing on low-level execu- tion. The policy learns entirely from simulated data to build the mapping between actions and flows. Unlike most sim-to-real work that requires building task-specific simulation, our policy learns entirely from exploration data generated by a set of predefined primitive actions, which is easier to collect and more diverse. Since flow represents motion, common concept across both real-world and simulation, our policy can be seamlessly deployed in realworld settings.
Together, the flow generation network and the flow-conditioned policy form cohesive system. Our final system provides scalable framework for acquiring robot manipulation skills by bridging
cross-embodiment demonstration video and cost-effective simulated robot data via object flows as

---

"Phar the water Jroee the cup into the dowi"
(c) Closed-loop Manipulation Palicy
(a]
Task Description
Open-loop
(c-1) Temporal Alignment Module
Task Flow
(c-3)
Diflusion Action Head
Robot Action Seq.
(b)
Initial Frame w/ key points
Online Point
(c-2) State Encoder
Tracking
Key Points @ Current Frame
State Repr.
Vector
Next Frame
Closed loop Feetback
Cureent Frame
Figure 2: Im2Flow2Act Overview. Given a task description and the initial frame, the flow generator (a) generates complete object flow for the task (i.e., task flow). The closed-loop manipulation policy (c). takes in the task flow and the keypoint location at the current frame to infer the robot actions. Within the manipulation policy, temporal alignment module (c-1) is used compare task flow and keypoint location at current frame to infer remaining task flow (flow in color) for the diffusion head (c-3) to generate actions.
an interface. We achieve an average success rate of 81 % across four real-world tasks, including those involving rigid, articulated, and deformable objects without any real- world robot data for training.
2 Related Work
Flow-hased manipulation. Robot policies leveraging flows have demonstrated promising results in manipulating articulated objects [20, 21J and tools [22]. However, these approaches are limited to specific tasks. With recent advances in point tracking algorithms in computer vision, flows can be estimated in a video sequence through learning- based methods. Vecerik et al. [16] achieved few-shot learning through tracking flows but required explicit pose estimation and was still limited to rigid objects. To make flow applicable to more general manipulation tasks, ATM [14] learned flow- conditioned behavior cloning policy that can be applied to both articulated and deformable objects. However, ATM still required collecting in-domain robot data through teleportation in the real world. Yuan et al. [15] proposed heuristic flow-based policy but required manual positioning of the robot gripper on the object, making the overall system less autonomous. More recently, Bharadhwaj et al. [17] propose to learn a residual policy on top of the heuristic-based policy but still require collecting real-world robot data. Im2Flow2Act, we train a data-efficient and fully autonomous flow-conditioned policy from lask-less simulation datasets which does not require costly in-domain robot data collection and be transferred realworld in one-shot.
Learning from cross-embodiment data. A large body of work [23, 24, 25, 26] has studied leverag- ing cross-embodiment data to learn robotic policies. Prior works explore different directions, includ- ing visual pretraining [8. 27, 28], learning reward functions [7, 29, 30, 31], extracting affordances [32, 9, 33, 34), hand pose detection [6, 35, 36), and domain translation [37, 38, 39, 40, 41, 42, 43]. As most of cross-embodiment data lacks explicit action or the action is challenging to transfer due embodiment gaps, many prior works still require collecting in-domain robot data [44, 42, 12, 11, 45] to mitigate the large embodiment gap. A number of works [46, 14, 15] have learned video generation or flow generation models through cross-ebbdiimnt data. ATM [14] demonstrates promising re- sults in learning flow generation from human demonstrations. However, as ATM generates grid flow that inevitably captures embodiment motion, it requires costly real-world robot data to train along- side the human demonstration data. This ensures the generated flow aligns with the distribution needed for robot manipulation policies. Im2Flow2Act predicts only the flow of objects, indepen- dent of the embodiment. This approach allows us seamlessly learn from human videos without needing any in-domain robot data. Please see Appendix A for more related works.
3 Method
We aim build a framework that leverages object flow as a unified interface for robots acquire real-world manipulation skills but without the need of costly real-world in-domain robot data. Our key idea is to use object flows as a general and expressive interface to bridge:

---

GO000000
(0000000
d0
DINO
Sample
(b) AnimateDDif
C0000000
Fiter
as flw generator
10000000
Project
Sampled Points
the
Stepaise
Object-cnttic Task Flow
Task Descrigtion
Figure 3: Flow Generation Network. The flow generation network outputs object flow for the complete task conditioned on the initial frame and task description. The object of interest is first detected using Grounding DINO (a), then we sample grid points inside the bounding box as the initial keypoints. AnimateDiff (b) takes in keypoints and task description generates the future flow. We postprocess the generated flow through motion filter obtain an object-centric task flow.
Pros--Embodiment data: Transferring actions between different embodiments is often hindered by the embodiment gap. However, object flow encapsulates transferable task knowledge independent of embodiment-specne specific actions. To exploit this, we develop a flow generation network focused solely on extracting object flows from cross-embodiment videos.
coossEnnvronmmen data: In this work, cross-environnen refers to simulation and real-world en- vironments. Visual inconsistencies, such as differences in scene backgrounds and object textures, pose challenges in transferring learned policies from simulation to real world. Conversely, object flow captures motion cynamics-changes in object pose, articulation, and cntemattonn-oosiisse across both simulated and real world, providing an ideal interface for policy learning. Our flow- conditioned imitation learning policy is learned entirely from the diverse simulated exploration data generated by set of predefined primitive actions, which fosters an understanding of the relationship between actions and resulting object flows, further reducing the need for task-specific environments.
During inference, the flow generation network first generates a complete task flow at the beginning of the task, and then the policy executes the actions reproduce the generated flow.
3.1 Flow Generation Network
Our main idea to form permutation-invarin object flows into a sequence of structured rectan- gular flow images. This approach allows us to better utilize pretrained image and video generation models for more efficient training. Specifically, we use Grounding DINO [47] obtain the bound- ing box of the object of interest from the initial RGB frame, followed by uniform sampling within the box, resulting in rectangular flow Fo E R3*H*W The first two channels represent the u, coordinates of object keypoints in image space, while the third represents their visibility. For tasks involving multiple objects, we first obtain the bounding boxes of all relevant objects and perform uniform sampling within each. The number of keypoints sampled is proportional to the area of each bounding box. We then apply TAPIR [48] to the videos, generating the rectangular flow sequence Fi R x3xTxxxW with 7 steps in the temporal space. With this flow representation, we leverage flow generator network,
the iiffusion-based video generation architecture AnimateDiff [18] conditioning it on the initial RGB frame and task description generate object flow. The generated flow is processed by motion filters to retain only keypoints on the object, producing complete object-centric task flow. The flow generation pipeline is illustrated in Fig. 3.
Complete Object-Centric Task Flow. Im2Flow2Act generates a complete object-centric task flow, contrasting with the immediate next-step grid flows in ATM [14]. This design minimizes the embod- iment gap between human demonstration and robot execution. First, object-centric flows reduce mo- tion distractions from the agent. Grid flows predominantly capture the embodiment's motion (e.g., human arm), leading to robot-specific flows. As a result, ATM requires additional real-world robot data to close this gap, whereas our method directly learns flow generation from human demonstrations. Second, generat- ing a complete flow from the initial frame avoids visual gaps during execution. Immediate next-step flows are negatively impacted by the embodiment gap, humans often contact and occlude objects during demonstrations. This conditions the model visual input where the embodiment and objects are closely attached. However, during robot deployment, the gripper will contact objects, leading to
flow generation for the robot policy, which is trained with

---

out-of-distriee of-distribution visual input. Generating complete flow from the initial frame avoids this, as the emhodiment need not be in contact with objects.
High-prectnn gigh-precision flow critical for enabling robots to achieve fine-grained control over objects. However, generating object flow at high resolution introduces significant computational complexity and results in inefficient generation speed. Inspired by pre- vious work [19], we propose to compress the object flow into compact latent space with lower spatial dimension, followed by training the generative model within this latent space. StableDif- fusion (SD) [19] compresses input images using an autoencoder (AE) proposed in VQGAN [49), resulting in a well-distributed latent space by training on a vast amount of RGB images. In this work, we explore the idea of using AE to encode the flow. To leverage this well-structured latent space, fix the encoder (.)..) from the AE and finetune the pretrained decoder Da(-) to better adapt it the flow images. The latents IuT RCXTx H. for input flow can be obtained by FI:T = (F) E [1, where Ha and Wa denote the spatial dimensions, which are downsam- pled by factor of 8 compared to the input flow spatial dimensions H and W.
Compressing Flow into Latent Space.
Video Diffusion Models for Flow Generation. As we share the same latent space as StableDif- fusion (SD), we can utilize the pretrained SD to provide a strong content prior for flow generation. Therefore, we choose to inflate the SD network along the temporal axis for flow generation rather than training model from scratch. We then insert the motion module layer into SD proposed by Animatediff [18] to model the temporal dynamics for flow generation. The motion module performs self- attention on each spatial feature along the temporal dimension to incorporate temporal informa- tion The SD model with the motion module, learns to capture the temporal variations in keypoints' location (u, v) in the image space. We train the motion module layer from scratch but only insert LoRA (Low-Rank Adaptation) layers [50] into the SD model.
3.2 Cldd-Condt Imitation Learning Policy
Our imitation learning policy
8ts 8ts Pt) takes the object flow Forr for complete task (i.e.. task flow), the current state representation 8t, and the robot proprioception Pt as input. The output is sequence of actions {a....., a of length L, starting from the current time t, denoted as ag. The robot action a includes a 6-DOF end-effector position in Cartesian space and 1-DOF gripper open/close state. The state representation St is formed by keypoints location at current frame denoted as ft containing N keypoints' locations {(u,, " J in the image space at time t and the 3D coordinates of N keypoints zo E RNN3 at the initial frame.
During inference, the task flow F is generated once by the flow generation network at the beginning of the task, as described in Sec. 3.1. We leverage an online point tracking algorithm to obtain the fe during robot manipulation for the same set of keypoints. The policy consists following components: a state encoder & takes the N keypoints encapsulate in the Jt and their corresponding initial 3D) coordinates xo as input to generate state representation 8t, a temporal alignment module v which compares task flow and current keypoints location fe to infer remaining task flow, and finally, a diffusion action head [51] to output the robot action.
We briefly describe how we construct the training data and some decision choice (See supplementary for more details). For an episode with time duration of T", we run the point tracking algorithm to get location of keypoints on object being manipulated at each step, i., fo.r". We can get a dataset D in the form of trajectories T3 = {Poo og), .... (pT ar During the training, we randomly sample frames among forr to treated it as task flow Fo:r.
Training data:
The state encoder d(Jt.rn) is a transformer-based [52] network that processes all N keypoints in the ft and the corresponding initial 3D coordinates To to output a current tate representation $t, i.e., 8, = (((f: For each keypoint, its location (u², vf.)) at time is first encoded using 2D sinusoidal positional encoding and the initial 3D coordinate Zo,n is passed through a linear projection head. They are then concatenated together to form a unique descriptor for each point. The descriptors for all N points are passed into the state encoder, and we use CLS token [53] to summarize the current state. Since the points are permutation invariant, we do not add any positional embedding and only add learnable embedding for the CLS token.
State Encoder:
5

---

Temporal Alignment: During inference, the policy should locate the current task progress and be conditioned on the remaining task flow, rather than the complete task flow, to predict precise actions. To this end, we incorporate a temporal alignment module & to predict the remaining task flow given the current keypoints location and the complete task flow. Furthermore, the alignment module enables our method to take in a human demonstration as policy imput and learn from during inference (i.e., idea is similar to the skill alignment in XSkill [45] but we extend it into more complex flow domain.
execution), capability that ATM [14] lacks. The
Instead of predicting the raw remaining task flow, we conduct it in the latent space 2 to improve training efficiency. We construct transformer-based temporal alignment model (FooT St, Pt) to predict the latent representation of the remaining task flow 2, based on the complete task flow Forr, the current state 8t, and the robot proprioception Pt- For the alignment module supervision &,, another transformer encoder Ç to encode the ground truth remaining task flow fer into the latent space which is accessible in the training dataset D. We use L2 loss between 2t and ê, i.e., || 22 to supervise the alignment learning.
During inference, the policy can directly condition on 2t, i... p(a |2 St, Pt) to predict future actions. During training. the policy is conditioned on & such that the encoder & gets supervision. Although the whole system be trained in an end-to-end fashion, detach the & when computing alignment loss, i.c., [| adetachh zz|/2 for more stable training.
Diffusion Action Head:
Evaluation
We demonstrate Im2Flow2Act's capability across 4 tasks ranging from rigid, articulate, and de- formable objects, including pick-and-place pouring, open drawer, and folding cloth. The task de- scription can be seen in the leftmost column in Fig. 4.
4.1 Experiment Details
Training data: We collect robot exploration data in simulation and human hand demonstration in the realworld. (i.) In simulation, we collect data across rigid, articulated, and deformable objects using a URSe robot through a set of predefined random heuristic actions for exploration. The details for each exploration strategy can be found in the supplementary material. We trained one imitation learning policy with all exploration data. (ii.) In real world, we collect human hand demonstrations for each task. We mixed the data together to train the flow generation model.
Comparisons. We compare our method with the following baselines:l) ATM: ATM [14] trains a closed-loop flow generation model that outputs grid flow, and the manipulation policy takes in the immediate next few steps of flow prediction. This comparison tests the benefits generating complete task object flows when learning from cross-embodiment demonstrations. 2) Heuristic: A heuristic action policy selects object contact points and applies pose estimation method, such as RANSAC, the future object flow (requiring 3D flow) to infer robot actions, which is implemented in General Flow and Track2Act [15, 17]. We provide the ground truth 3D flows as heuristic method's input. This baseline examines the necessity of having
barming-based policy for translating flow into actions. 3) Grid Flow: Similar to ATM, we replace the initial object keypoints in Im2Flow2Ac with a set of uniformly sampled keypoints. 4) No alignment: To ablate our alignment module design, we removed the alignment model ya and the policy condition on the complete task flow.
Evaluation Setting. We evaluate Im2Flow2Ach and the baseline models in two scenarios: (1) demonstrationncooniiioonn execution and (2) ianguage-conditionn and simulated environment. In the simulations, we utilize a sphere robot to create cross-embodiment demonstrations. In the cinioossration-condiiiinno human demonstration video (or sphere demonstration in simulation) as input for the manipulation policy. This setting evaluates the robustness of the low-level action generation independent of flow generation accuracy. In the language-conditionee execution, we evaluate the full system by first gen- erating flow using learned flow generation network conditioned on the task description and initial frame. The manipulation policy then uses this generated flow for action inference.
execution under both real-world
setting, we use object flow extracted from a single

---

Initial
Generation
Execution
Online Tracking
Pouring:
pour
vaser in the green cup red bovel.
Pich&Place:
por cup
onto the red plate.
open the drowes
Folding:
the the diagonal corser
Figure 4: Task and results: We conducted evaluations on four tasks involving rigid, articulated, and de-
formable objects. The initial scene, flow generation visualizations, and online point tracking during inference were captured using the RealSense camera. Visualizations of the robot's execution were recorded from dif- ferent angle to provide clearer view of the robot execution.
Table Simulation Results (%)
conditioned execution Pick&Place Pouring Drawer
Language-conditioned execution
Cloth
Pick&Place Pouring Drawer Cloth
Im2Flow2Act
100
95
95
90
90
85
90
35
ATM[14]
/
/
/
50
30
85
30
Heuristic 17]
70
50
30
GridFlow
30
25
35
45
No alignment
80
85
90
90
4.2 Key Findings
Flows can effectively bridge different data source: Im2Flow2Act achieves best performance among all baseline in both simulation and realworld, as shown in both Tab and Tab 2. Further, the performance only drops 15% on average in realworld compare to in simulation. As shown in Fig. 4, our learned policy can largely follow the generated flow to complete the desired tasks across rigid, articulated and deformable objects. This suggests that object flow is good interface connect the both cross-embodiment and cross-environment data sources.
Learning-based policy is necessary for translating flow to actions: To map the generated flow to robot actions, we used an learned policy and alignment network. In contrast, prior works [17, 15] use heuristic-based policy to compute robot action from the flow. When compared to them, we provide the ground truth 3D flows and optimal grasping pose, which are not required by our method. As shown in Tab. I, heuristic-based policy shows good performance for rigid objects, such as pick&place and pouring in simulation. However, struggles when manipulating deformable and articulated objects in both sim and real the performance of heuristic methods heavily depends the camera view. For instance, in the task of opening drawer, if the center of the point clouds under camera view is not a good contact point, it will push the drawer back (Fig. 5). In the realworld (Tab. 2), we notice the heuristic-bart based frequently triggers the emergency stop from UR5e as UR5e does not have impedance control, therefore a small error in pose estimation can cause the task to fail. This suggests that learning based policy necessary for translating flow accurate and safe actions.
Table 2: Real-World Results (%)
Demoostrationnconnditonne exccution
Language-conditionee exccution
Pick&Place Pouring Drawer
Cloth
Pick&Place Pouring Drawer Cloth
Im2Flow2Act Heuristic[15, 17]
90
70
80
70
70
50
30
0
/
/
/
No alignment
55
80
60
/
/
/
/

---

Object-flow is crucial when learning from cross-embodiment demonstration: We evaluate the object-centric flow design by comparing our method with ATM. As shown Tab. 1, Im2Flow2Act outperforms ATM by an average of 30% across four tasks. The key reason ATM performs poorly when learning from cross-embodiment demonstrations that the visual input becomes out-of- distribution during robot deployment. During the training, ATM's takes in the visual imput with cross-embodiment demonstrations (human arm in the real world and sphere in simulation) and makes predictions, most of which are related to the embodiment flow. However, during inference, the visual input contains the robot arm instead of the human arm or sphere. As the model also generates the embodiment flow, the unseen embodiment causes the model to be out-of outoof-distribution. In contrast, Im2Flow2Act only generates the object flow and does not focus on the embodiment. The initial object keypoints also provide a strong condition for the model to generate object-centric flows. Therefore, our model is robust to the presence of different embodiments during robot de- ployment. We further compare the design decision on initial keypoint sampling by replacing the object-flow in method with the uniform grid flow. In Tab. 1, we observed that its performance is significantly drop, as the uniform flow also captures motion from the embodiment itself, similar to ATM. This highlights the superiority of object-flow over grid-flow for cross-embodiment learning.
The alignment module is necessary for leveraging unstructured exploration data: In Tab. 1, the performance of Im2Flow2Act without alignment only slightly decreased in the simulation. How- ever, we observed that the robot's exe- cution became unsmooth, involving many unnecessary actions such as sudden rota- tions. This suggests that the policy can- not align its behavior with the task flow based task progress and only attempts to find the closest trajectory in the explo- ration data. Further, in realworld, its per- formance drops significantly (Tab, 2), es- pecially in the pick-and-plae placeeace and pouring tasks, as the object flow yielded from human can differ from the flow in the simulation. It challenging for the policy to find similar trajectories in the la- tent space, leading to random trajectories as shown in Fig. 5. Im2Flow2Act with alignment does not suffer from this issue, as the latent space has finer granularity with the extra alignment supervision, leading to smooth interpolation.
lock
Hoariss:
draggel
CLD A NESSE
Figure 5: Typical failure cases of baselines. Even given the ground truth flows produced by human, the hcuristic pol- icy may produce inaccurate actions, for example, pushing the drawer back first. As for the No Alignment model, the cup may be randomly rotated along the trajectory, which is the same behavior the random exploration data.
4.3 Limitation and Future Work
Our system uses 2D flow the manipulation interface, enabling to utilize large-scale 2D videos. However, this approach is inherently ambiguous for representing 3D actions. For instance, a com- mon failure in pouring tasks arises when the robot's z-axis movements (in camera coordinates) lack precision. Additionally, 2D flow struggles with tasks involving out-of-plane rotation (e.g., screw- ing), a limitation potentially addressed by 3D flow [54, 15J. Our framework also assumes consistent object dynamics between simulation and real world, yet simulating deformable objects remains challenging. Consequently, we observe significant performance drop in the folding task when deploying the policy in the real world. Please see Appendix I for more discussion.
5 Conclusion
We introduce Im2Flow2Act, scalable learning framework that enables robots to acquire diverse manipulation skills from cost-effective cross- domsin data using object flow as a unifying interface. Our final system demonstrates strong real-world manipulation capability by outperforming all base- lines across various types of manipulation tasks. Our work paves the way for future research to seale up robotic manipulation skills from diverse data sources.
8

---

Acknowledgments
We would like to thank Yifan Hou, Zeyi Liu, Huy Ha, Mandi Zhao, Chuer Pan, Xiaomeng Xu, Yihuai Gao, Austin Patel, Haochen shi, John So, Yuwei Guo, Haoyu Xiong, Litian Liang, Dominik Bauer, Samir Yitzhak Gadre for their helpful feedback and fruitful discussions.
Mengda Xu's work is supported by JPMorgan Chase & Co. This paper was prepared for informa- tion purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates ("JP Morgan"), and is not product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended invest- ment research or investment advice, recommendation, offer solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. This work was supported in part by NSF Award #2143601, #2037101, and #2132519. The views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] Y. Zhu and A. Joshi. Viola: Imitation learning for vision-based manipulation with object proposal priors. In Conference Robot Learning. 2022.
[2] M. Seo, Han, K. Sim, S. H. Bang, C. Gonzalez, L. Sentis, and Y. Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation. In 2023 IEEE-RAS 22nd International Conference Humanoid Robots (Humanoids), pages 1-8. IERE, 2023.
[3] A. Toedtheide, X. Chen, H. Sadeghian, A. Naceri, and S. Haddadin. A force-sensitive ex- oskeleton for teleoperation: An application in elderly care robotics. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 12624-1 1662--16630. IEEE, 2023.
[4] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and Levine. Bridge data: Boosting generalization robotic skills with cross-domain datasets.
[5] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068117 2022.
[6] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. 2022.
[7] A. S. Chen, S. Nair, and C. Finn. Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos. In Proceedings of Robotics: Science and Systems, Virtual, July 2021.
[8] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, 2022.
[9] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos versatile representation for roboties. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13778-13790, 2023.
[10] C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. Tax-pose: Task-specific cross-pose esti- mation for robot manipulation. In Conference Robot Learning, pages 7792-1992. PMLR, 2023.

---

[11] K. Schmeckpeper, A. Xie, O. Rybkin, S. Tian, K. Daniilidis, S. Levine, and C. Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pages 708-725. Springer, 2020.
[12] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang. R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570-587. Springer, 2022.
[13] Y. Zhu, A. Lim, P. Stone, and Y. Zhu. Vision-based manipulation from single human video with open-world object graphs, 2024. URL
20321.
[14] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00M 00000,, 2023.
[15] C. Yuan, C. Wen, Zhang, and Y. Gao. General flow as foundation affordance for scalable robot learning. arXiv preprint arXiv:2401. 2024.
[16] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint arXiv:2308. 15975, 2023.
[17] H. Bharadhwaj. R. Mottaghi, A. Gupta, and S. Tulsiani. Track2act: Predicting point tracks from internet videos enables diverse zero-shot robot manipulation, 2024.
[18] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai. An- imatediff: Animate your personalized text-to-image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representations, 2023.
[19] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent difffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 00684-10695, 2022.
[20] B. Eisner, H. Zhang, and D. Held. Flowbol3d: Learning 3d articulation flow to manipulate articulated objects. In Roboties: Science and Systems (RSS), 2022.
[21] H. Zhang, B. Eisner, and D. Held. Flowbot++:
Learning generalized articulated objects ma- nipulation via articulation projection. Conference on Robot Learning, pages 1222-1244 1241. PMLR, 2023.
[22] D. Seita, Y. Wang, S. J. Shetty, E. Y. Li, Z. Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot Learning, pages 1038-1049. PMLR, 2023.
[23] A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V.P. Berges, T. Wu, Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelli- gence? Advances in Neural Information Processing Systems, 36. 2024.
[24] H. Niu, J. Hu, G. Zhou, and X. Zhan. A comprehensive survey of cross-domain policy transfer for embodied agents, 2024.
[25] R. McCarthy, D. C. H. Tan, D. Schmidt, F. Acero, N. Herr, Y. Du, T. G. Thuruthel, and Z. Li. Towards generalist robot learning from internet video: A survey, 2024.
[26] haoyu Xiong, H. Fu, J. Zhang, C. Bao, Q. Zhang, Y. Huang, Xu, A. Garg, and C. Lu. Robo- tube: Learning houschold manipulation from human videos with simulated twin environments. In 6th Annual Conference on Robot Learning, 2022. URL https: /ttpps://opee htttps://ppenreiiiee hetttewwwwweeeeeee.e.........eee forum? perummid=VDie 2iD=nDoXU.
[27] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre- pre-training for motor control. arXiv preprint vrXiv:2203.06173 2022.
10

---

[28] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual repre- sentation for robot manipulation. In Conference on Robot Learning, pages 892-909. PMLR, 2023.
[29] I.. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research, 40(12-14):411-44444 2021.
[30] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pages 537-546. PMLR, 2022.
[31] C. Ying, Z. Hao, X. Zhou, X. Xu, Su, X. Zhang, and Zhu. Peac: Unsupervised pre-training for cross-embodiment reinforcement learning, 2024.
[32] R. Mendonca, Bahl, and Pathak. Structured world models from human videos.
[33] M. Goyal, S. Modi, R. Goyal, and S. Gupta. Human hands as probes for interactive object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3293-3303, 2022.
[34] S. Liu, S. Tripathi, S. Majumdar, and X. Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282-3292, 2022.
[35] K. Shaw, S. Bahl, and D. Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, pages 654-665. PMLR, 2023.
[36] C. Wang, H. Shi, W. Wang, R. Zhang, I.. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788 2024.
[37] K. Schmeckpeper, O. Rybkin, K. Daniilidis, Levine, and C. Finn. Reinforcement learn- ing with videos: Combining offline observations with interaction. In Conference on Robot Learning, pages 3394.54. PMLR, 2021.
[38] Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
[39] Y. Liu, A. Gupta, P. Abbeel, and Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1118-112.. IEEE, 2018.
[40] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprini (TX.v:1112.04444/ 2019.
[41] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj. S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[42] M. J. Kim, J. Wu, and C. Finn. Giving robots hand: Broadening generalization via hand- centric human video demonstrations. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
[43] L. Y. Chen, K. Hari, K. Dharmarajan, C. Xu, Q Vuong, and K. Goldberg. Mirage: Cross- embodiment zero-shot policy transfer with cross-painting, 2024.
[44] C. Wang, L. Fan, Sun, Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long- horizon imitation learning by watching human play. In 7th Annual Conference Robot Learning, 2023.
11

---

[45] M. Xu, Z. Xu, Chi, M. Veloso, and S. Song. Xskill: Cross embodiment skill discovery. In 7th Annual Conference on Robot Learning, 2023.
[46] Y. Du, S. Yang, B. Dai, H. Dai, O. Nachum, Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
[47] S. Liu, Z. Zeng. Ren, Li, H. Zhang, J. Yang. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
[48] C. Doersch, Y. Yang. M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 10061-10072. 2023.
[49] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthe- sis, 2021.
[50] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li. S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021.
[51] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. C. Burchfiel, and S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023.
15607/RSS.2025 22223XXXX0026
[52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L.. Kaiser, and I. Polo- sukhin. Attention is all you need, 2023.
[53] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv: 2018
[54] S. Koppula, I. Rocco, Y. Yang, J. Heyward, J. Carreira, A. Zisserman, G. Brostow, and
C. Doersch. Tapvid-3d: -3: benchmark for tracking any point in 3d, 2024. URL https: / www.iiv.rg/aaba... 2407 05921.
[55] X. Zhang, R. Chen, A. Li, F. Xiang, Y. Qin, J. Gu, Z. Ling, M. Liu, P. Zeng, S. Han, et al. Close the optical sensing domain gap by physics-grounder active stereo sensor simulation. IEEE Transactions on Robotics, 2023.
[56] Y. Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang. Dexpoint: Generalizable point cloud rein- forcement learning for sim-to-real dexterous manipulation. In Conference on Robot Learning, pages 595-605. PMLR, 2023.
[57] C. Wu, X. Bi, Pfrommer, A. Cebulla, Mangold, and J. Beyerer. Sim2real transfer learning for point cloud segmentation: An industrial application case on autonomous disassembly. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4531-4540, 2023.
[58] Y. Huang, J. Yuan, C. Kim, Pradhan, B. Chen, L. Fuxin, and Hermans. Out of sight, still mind: Reasoning and planning about unobserved objects with video tracking enabled memory models. arXiv preprint arXiv:2309. 15278, 2023.
[59] S. Uppal, A. Agarwal, H. Xiong, K. Shaw, and D. Pathak. Spin: Simultaneous perception, interaction and navigation. CVPR, 2024.
[60] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ international conference intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
12

---

[61] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 969-977, 2018.
[62] J. Tobin. Real-world robotic perception and control using synthetic data. 2019. URL https: semant sepisseenticscchoo....
icscholar org/Corpus ID: 1 1999188944
[63] M. Sundermeyer, Z.-C. Marton, M. Durner, Brucker, and R. Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In Proceedings of the european conference on computer vision (ECCV), pages 699-715, 2018.
[64] M. Kaspar, J. D. M. Osorio, and J. Bock. Sim2real transfer for reinforcement learning without dynamics randomization. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4383-4388. IEEE, 2020.
[65] X. Yue, Y. Zhang, S. Zhao, A. Deggiovannn-Vinceneelee K. Keutzer, and B. Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2100-2110 10, 2019.
[66] R. Traoré, H. Caselles- Duupéé, T. Lesort, T. Sun, N. Diaz- and D. Filliat. Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer. In ICML Workshop on "Multi-Tass" and Lifelong Reinforcement Learning", 2019.
[67] A. A. Rusu, G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation, 2016.
[68] K. Kristinsson and G. Dumont. System identification and control using genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics, 2235): 1992. doi:10.1109/ 21. 21.179842.
[69] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[70] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[71] I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019.
[72] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.026433 2023.
[73] N. Ravi, Gabeur, Y.T.. Hu, R. Hu, C. Ryali, T. Ma. H. Khedr, R. Rädle, C. Rolland, L. Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 2024.
Appendix
A Additional Related Work
Sim2Real Transfer. To overcome the data collection challenge in the real world, a desired ap- proach is to train robot policy in simulation and deploy in the real world. Due the sim-to-real gap, various techniques have been employed to bridge this gap, including but not limited to using depth observation [55, 56, 57, 58, 59], domain randomization [60, 61, 62, 63, 64, 65]. knowledge
13

---

distillation [66, 67], and system identification [68]. Learning sim-to-real policy typically requires task-specific environment construction such as reward function design. In contrast, Im2Flow2Act learns the task information from real-world human video by learning a flow generation network. Our policy learn from simulated exploration data completely generating by a set of predefined prim- itive actions, which is easier to collect and further reduces the need for task-specific environment construction.
B Additional Experiment
a). Im2Flow2act cross-embodiment
b). ATM same-embodiment
c). ATM cross-embodiment
Time
Figure 6: Flow generation comparison during robot deployment. We compare Im2Flow2Act and ATM side by side in the pouring task. Im2Flow2Act (a) demonstrates its robustness in the presence of the URS in the visual input, generating high-quality object-centric task flows. contrast, ATM generates noisy flows (c) when learning from sphere demonstrations and only performs well (b) when learning from same-embodiment demonstrations. Notice, we only visualize the query points with high variance for ATM. As ATM only predicts the future N steps and the mug will not mug sub-figure Please refer to Fig. complete grid flow with object moving under the prediction.
under the prediction horizon, therefore there the case robot more close to the object. We visualize
flows the
In this section, provide additional qualitative and quantitative comparisons with the baseline ATM [14] in Pick&Place, Pouring and Drawer task. We used the official ATM repository to train its flow generation model and adapted its manipulation policy to diffusion action head. We re- move the RGB input into manipulation policy, similar to Im2Flow2Act. We additionally provide initial depth input same as Im2Flow2Act to ensure a fair comparison. Instead of training on more challenging exploration dataset as in Im2Flow2Act, we directly provide task specific data to train ATM manipulation policy. For ATM, we trained versions of its flow generation model: one with UR5 demonstrations (i.e., same embodiment) and one with sphere agent demonstrations (i.e., cross-embodiment demonstrations). Note that method is always trained with cross- coossemmbod demonstrations.
Demonstratioonconditinne execution
I Langunge-conditione execution Pick&Place
Plck&Place
Pouring
Drawer
Pouring
Drawer
Im2Flow2Act
100
95
95
90
85
90
ATM cross embodiment
/
50
30
85
ATM same embodiment
90
90
95
Table 3: Comparison with ATM Result (%)
In Fig. 6, we first compare the generated flows from both Im2Flow2Act and ATM side by side during robot deployment. shown in Fig. 6.a, Im2Flow2Act is robust. to the presence the UR5 in the visual input and generates high-quality flows, thanks to the choice of using complete object- centric flow. In contrast, ATM only generate high-quality flow (see Fig. 6.b) when trained with the same embodiment (i.e., trained with UR5 and inferred with UR5). It generates low-quality and
14

---

noisy flows (see Fig. 6.c) when trained with cross- cross-embodiment demonstrations (i.e., trained with sphere agent and inferred with UR5).
As mentioned updated main paper Sec. 3.1., the key reason ATM performs poorly when learning from cross-embodiment demonstrations is that the visual input becomes out-of-distribution during robot deployment. Compared Im2Flow2Act's object-centric flow, ATM generates grid flow. This design choice leads to the majority of the generated flow is for the embodiment. During the training, ATM's flow generation model takes in the visual input with cross- coos-embbdiment demonstrations (hu- man arm the real world and sphere in simulation) and makes predictions, most which are related to the embodiment flow. However, during robot deployment, the visual input contains the robot arm and gripper instead of the human arm or sphere. As the model also generates the embodiment flow, the unseen embodiment causes the model to be out-of-distribution.
In contrast, Im2Flow2Act only generates the object flow and does not focus on the embodiment. The initial object keypoints also provide a strong initial condition for the model to generate object- centric flows. Therefore, our model is robust to the presence of different embodiments during robot deployment.
We further evaluate the complete system of ATM with the two versions of the flow generation model and compare them with Im2Flow2Act. Notice, ATM system lacks the ability to condition on complete demonstration. As shown in Tab. 3, ATM's performance significantly drops when the flow generation model is learned from cross-embodiment demonstrations compared to when is learned from came-mmmod demonstrations. contrast, our system demonstrates strong capability when learning from cross-embodiment demonstrations.
B.1 Long Horizon Task with Multiple Objects
Get multiple bounding box
Dynamic keypoints sample based bounding box
Generated Flow
0000
Flow Generation Pipeline
"Put the blue cube into the drawer"
(b) Robot Execution
Online Tracking
Time
Figure 7: Long-Horizon Task with multi-objects: Im2Flow2Act is capable of generating flows (a) for long- horizon tasks involving multiple objects. Our pipeline starts with getting multiple bounding box for objects of interest and dynamically sampling the initial keypoints proportional the bounding box area. The sampled keypoints are then concatenated dition. Our manipulation policy (b) effectively follows the generated flow to successfully complete the tasks. For videos, please visit our anonymous website: hteps: http://i2ffowacccccc github. io/
a x W rectangular flow image as flow generation model con-
We conducted an additional experiment evaluate Im2Flow2Act's performance on a long-horizon task involving multiple objects. In this task, the robot needs to perform sequence of actions: first, open the drawer, then pick up the blue cube from the table and place into the drawer, and finally
15

---

close the drawer. The positions of the drawer and the cube are randomly initialized at the beginning of each episode. We collected 150 cross-embodiment demonstration trajectories (i.e., using a sphere agent) and trained our flow generation model accordingly and 100 URS demonstration to train the manipulation policy.
For tasks involving flow generation for multiple objects, similarly use Grounding DINO obtain the bounding boxes of all relevant objects. We then perform uniform sampling within each bounding box. The number of keypoints sampled within each bounding box is proportional to its area. To ensure total of H x W keypoints, we pad the final sampled keypoints as needed. The key idea is still to form structure flow image from permutation invariant keypoints.
The key reason Im2Flow2Act is capable of generating long-houe honghhoizzon flow is that our flow genera- tion model generates flow with temporal sub-sampling, rather than at the original flow speed. This design make it easy for the network to generate compact flow for long-horizon tasks. Addition- ally, it makes the system more robust when the speed of cross-embodiment demonstrations differs significantly from the robot's ' speed.
The same parameters were used for training and inference for both flow generation network and manipulation policy in the other tasks, with further details provided in appendix Sec. F.1 and Sec. F22
Demoonsrationn-omdittonnd execution
Language-conditioned execution
Im2Flow2Act
90
85
ATM
45
Table 4: Long Horizon Task with Multiple Objects Result (%)
We compared Im2Flow2Act's performance with ATM using the same training data for flow genera- tion and manipulation policy as Im2Flow2Act. As shown in Tab. 4, Im2Flow2Act successfully com- www.annttratioooonniiiiiooo
pletes long- nong-horizon tasks with 90% success rate in the and 85% success rate in the language- conditioned execution setting. In the language-conditioned execution setting, we visualized the generated flow and the robot execution in Fig. 7. As shown in the Fig. 7, Im2Flow2Act capable of generating complete object-centric task flows for long- horizon tasks with multple objects, and the robot execution closely aligns with the generated flows. ATM performance suffers from the out-of-distribution visual input into the flow generation model as we discussed in above section and our updated main paper Sec. 3.1.
exccution setting
B.2 Ablation Study on Initial 3D Keypoints
Demonstration-coddiiooed execution
Language-connitioned execution
Pick&Place
Pouring
Drawer
Pick&Place
Pouring
Drawer
Im2Flow2Act
100
95
95
90
85
90
Im2Flow2Act w.o. 3D
100
90
90
85
85
80
Table Ablation Study on Im2Flow2Act w./woo initial 3D keypoints Result (%)
We conduct an additional experiment to demonstrate the necessity of having initial 3D keypoints. Our experiments (see Tab. 5) show that the initial 3D keypoints improve policy performance when taking generated flow as input (i.e., language-conditione execution). The generated flow often contains some noisy keypoint movement, especially for keypoints around objects. The additional 3D keypoints help the network distinguish between noisy keypoints and those actually the objects, e.g., drawer handle.
16

---

Once the initial 3D keypoints have been identified, all remaining movement is relative to these initial 3D keypoints. Therefore, our system does not require 3D keypoints for all frames. Moreover, obtaining 3D keypointss for every frame in the real world involves additional operations, so we aim to keep our system simple and efficient.
B.3 Generated Flow in Simulation
Pick&Plaaee
Pouring
Open the drawer
Folding
Figure 8: Generated Flow simulation: We visualize the generated flow with high-variance for four tasks in simulations. We use the motion filter to filter the flow with high-variance. More details for motion filters can be found in appendix G.2. The keypoints has been down-sampled to provide better visualization. Note that in the real world, we train the flow generation model using human demonstrations.
In the simulation, we collect crosseembod demonstrations by replacing the URS with sphere agent and use these demonstrations to train the flow generation model. The learned flow generation model is used to evaluate Im2Flow2Act's performance in simulation under the language-conditioned execution setting (see Tab. 1).
B.4 Ablation on pretrain Stable Diffusion
The quantitative results of whether to use a pre-trained Stable Diffusion model to train the flow generation model are presented Appendix Sec. this section, we visualize the training loss for both approaches. As shown in Fig. 9, leveraging the pre-trained U-Net from Stable Diffusion leads to faster convergence and lower loss. This suggests that fine-tuning pre-trained U-Net is beneficial for flow generation.
epoch loss
Pretrain Unet Unet from scratch
0.08
0.06
0.04
0.02
epoch
100
300
Figure 9: Unet Pretrain vs Scratch: The training loss converges much faster for pretrain Unet. The visualized loss is not heing smoothed.
17

---

B.5 ATM flow generation additional visualization
In this section, provide the visualization for complete grid flow prediction from ATM and also the filleted object flow from predicted grid flow in Fig. 10. The ATM predicted the grid flow for the future 16 steps based on the current state. The bounding box we use to to filter the object flow is the same as the one we used for Im2Flow2Act. Compared to Fig. 6, the robot is more close to the object in Fig. 10. From Fig. 10 b1 and b2, the ATM can generate high-quality flows if learning from same embodiment demonstrations. However, generated flow is completely noise if learning from the cross-embodiment demonstration (See Fig. 10 el and c2). In contrast, Im2Flow2Act can generate high-quality flow when learning from cross-embodiment demonstrations. Notice, increased the number of grid key points from 32 in the original ATM paper to 400 to ensure fair comparison. The ATM policy also takes in the complete flows from 400 grid keypoints input, which is significantly larger than 128 object key points in our method.
a). 2m2flo222t
b1). ATM same embodiment
cl). ATM cross embodiment
b2). ATM same embodiment filtered by object bounding box
<2). ATM cross embodiment filtered by object bounding box
Figure 10: Flow generation comparison during robot deployment (grid flow and filterd with object bound- ing box) We compare Im2Flow2Act and ATM side by side in the pouring task. ATM generates noisy flow when learning from crosseemooliment demonstrations (cl,e2). In contrast, Im2Flow2Act generated high-quality flow even when learning from gross-embodiment demonstrations (a). Notice, visualized flow from Im2Flow2Ar after processed by motion filters. The initial keypoints is uniformally sampled from the bounding box and we do not manually select the initial points.
C Data Collection
In Im2Flow2Act, collect two types of data: simulated robot exploration dataset and real-world human demonstration videos. Note that we do not collect any real-world robot data. Below, we detail the data collection process.
C.1 Simulated exploration Data
We use MuJoCo as our simulation engine and have constructed three types of exploration environ- ments featuring rigid, articulated, and deformable objects. An UR5e robot explores these environ- ments using set of predefined random heuristic actions.
18

---

In the rigid environment, the robot can interact with five different objects, including toy car, a shoe, a mini-lamp, a frying pan, and mug. At the beginning of each episode, one object is randomly selected and placed on the table. The robot first picks up the object and starts executing 6DoF random trajectories. random trajectory constructed as a 3D cubic Bézier curve with four key waypoints: the start point Po. two control points P1 and p2. and the end point p3. The start point is where the robot first picks up the object, and the end point is randomly selected within the robot's reachable world coordinates. The two control points are obtained by adding curvature to the line defined by po and p3- Specifically, for the first control point pl, we first calculate the one-third point m. between pu and Ps. i.e., mi Pu (Pa_ro) We then twist m, in 3D space to obtain the first control point by adding random offset (curvature), i... P1 m2 + e, where set each dimension of c be uniformly sampled from (-0.05, 0.05). Similarly, we obtain the second control point by calculating the two-thirds point and adding some random offset. Once we obtain the points po- P1. P2, and Py, we define the cubic Bézier curve by (1 - P) + 3(1- - t) + - P + P3, where & E [0, 1]. We equally sample (k- 16) waypoints along this curve, which defines the translation for the robot's 's exploration trajectory. We add target object orientation (random sample from (-5
Rigid:
R We
5: g for each dimension) when the object arrives at the final point pa- We interpolate between the initial orientation at po and the target orientation at P3 and attach them to the k waypoints. For each episode, we sample two consecutive 3D cubic Bézier curves, allowing the robot to interact with random objects and build the correspondence between flows and actions. The trajectory ends with a random rotation or place actions.
We construct various types of drawers, none of which are the same as those tested in the real world. In each episode, one drawer is selected and placed on the table. The robot explores these articulated objects by opening the drawer to different extents, not necessarily fully open. To open a drawer, a random contact point is selected on the drawer handle within a range of (-0.07, 0.07) from the center of the handle. This contact point is then passed to our predefined open- ing primitive. The primitive places the gripper on the contact point and initiates the drawer's s opening by pulling perpendicular to the drawer's surface. A random threshold, ranging from (0.07, 0.12), is sampled to define the success criterion for each episode. The robot stops execution once the drawer handle has moved beyond this threshold, indicating a successful opening.
Articulated:
Deformable: At the beginning of the episode, a cloth is randomly placed on the table. The robot has the option to grasp either the left or right corner of the cloth. We use the predefined grasp primitives to achieve this behavior, which receive the grasping pose as input, similar to the one we use in the rigid dataset generation. Once grasped, the robot begins random folding (not necessarily folding diagonally). We have defined nine different folding targets, and the robot randomly selects one to start folding. These nine different folding targets is formed by the combination of (0.08, 0.15, 0.28) on and y axis of the grasping point coordinate. During this process, we also vary the folding trajectory by lifting the cloth to different heights. We use the predefined move heuristic to move the end effector to a random height. We first calculate the distance d between the folding corner and the folding targets. We then randomly sample the lifting height between (d/ /4/4,0 d). Finally, we move end effector towards to the folding target and open the gripper.
In total, we collect 4800 random exploration trajectories. Once all data is collected, we use an iterative procedure to obtain object flows. For each collected episode, we begin by sampling uniform grid (30x30) keypoints on the initial frame and track all points using Tapir [48], similar to the approach in ATM [14]. We then apply moving filters and SAM (Segment Anything Model) filters which we will explain in details at section G below to obtain keypoints on the object. However, as the initial sampling is uniform, there may be few points located on the objects. To address this, we sample additional points around the existing keypoints based on the SAM output achieve denser distribution of object keypoints. Specifically, for segments containing keypoints after filtering, we sample proportionally based on the number of filtered keypoints in that segment. In total, we sample 900 points on each object. Finally, we run the point tracking algorithm again on the newly sampled object keypoints to obtain dense object flow.
19

---

C.2 Real-world Human Demonstration Video:
We collect in-domain human demonstration videos for four tasks: pick & place, pouring, opening drawers, and folding cloth. We use a RealSense camera to record the demonstrations at 30 FPS. The descriptions below also serve as detailed task deseriptions. Im2Flow2Act, we use the human videos to provide the system with task information. We define the robot base the origin of the world coordinate system and use the Right- Right-Handed Coordinate System.
Pick & Place: Pick up green cup and place it into a red plate. The red plate is randomly placed on one side of the table, while the cup is placed randomly in the middle of the table.
Pouring: Pick up a green cup and hover over a red bowl, then rotate the cup towards the bowl.
The initial placement of the cup and the bowl is the same in the pick & place task.
Drawer Opening: Fully open drawer that is randomly placed on one side of the table. The pose of the drawer, including its position and orientation, can vary. The drawer is not rigidly attached to the table.
Cloth Folding: Fold 33 em x 33 em cloth from one corner (either lower left or lower right) to the diagonal. The cloth is randomly placed in the middle of the table.
Once the human demonstration is collected, for each episode, we use Grounding DINO to obtain the object bounding box the initial frame and uniformly sample the keypoints inside, where we set H = W 32 as sampling parameters. We then run point tracking algorithm to track the keypoints across frames in the episode. To construct the training dataset, we uniformly sample 32 frames from each episode to form the task flow.
D Ablation study
We conducted an ablation study in simulation to evaluate the impact of using pretrained StableD- iffusion (SD) versus training it from scratch on the flow generation model. In this ablation study, we still use pretrained AE (auto-encoder) from the SD but trained the U-Net from scratch instead of incorporating LoRA layers. To ensure fair comparison, deploy the same flow-conditioned
Table Ablation study Results (%)
Pick&Place Pour Drawer Cloth
Pretrain U-Net U-Net From Scratch
90
85
90
90
90
95
30
policy for both the pretrained SD and the training scratch as the manipulation policy. We collect data for four tasks the simulation by substimting the UR5e robot with sphere robot to create a cross- embodiment scenario. Both networks were trained for the same number of epochs and evaluated within the same initial state. Based on the results shown in Tab. 6, the choice of pretraining had minor impact on Im2FlowAAct's final performance. This suggests that: i. Althouugh StableDiffusion was initially designed for image generation, directly using its pretrained weights for flow generation does not impact its performance. Utilizing LoRA can lead to better training efficiency compared to training from scratch. ii The latent space encoded by the AE from the pretrained SD might benefits the diffusion model's 's learning process.
Compared to the results using ground truth flow in simulation, observed that for tasks like pick & place, pouring, and drawer opening, the success rates by taking generated flow as input are very close. This further demonstrates flow generation network's capabilities. However, the success rate for cloth folding drops significantly. We attribute this to the way we constructed the cloth folding simulation environment. Grasping deformable objects like cloth is challenging, so we attached a cube (See Fig. 11) the corner of the cloth to help the robot to lift the corner by grasping the cube. During data generation and inference, we render only the cloth, not the cube, to mimic realistic cloth folding behavior. We made the cube's dimensions the xy-plane very small (lem lcm) to emulate the width of a gripper grasping actual cloth. Since the output from diffusion
20

---

Tiny Cube enable grasprig
Reasonable grasping point
Close the gripper
Fail to grasp the cube
Figure 11: Policy rollout in Deformable Environment. We attach a tiny cube (lem x lem 9cm) to enable robot grasp deformable objects. The generated flow guides the policy toward an appropriate grasping point; however, fails grasp the tiny cube, resulting in task failure.
model typically exhibits multimodal distributions and our training dataset contains considerable randomness, the learned flow generation model produces reasonable flows for the folding task, but may not direct the policy grasp the cube precisely. Moreover, the flows after motion filtering exacerbate the issue. We found that in most cases, the robot can reach the corner and attempt the grasp (See Fig. 11), but it fails accurately grasp the cube. In contrast, we observed a reasonable success rate in real-world experiments because, in practice, the robot can grasp the cloth anywhere along the corner to executing folding.
E Experimental Details
E.1 Real-World Setup
We UR5e robot equipped with a WSG-50 gripper. During inference, the URSe receives end- effector space positional commands from 2.5 Hz policy. We limit the end-effector's speed to less than 0.2 m/s and restrict its position to at least cm above the table for safe execution. A RealSense D415 depth camera is mounted at the table to capture policy observations, including the initial depth image for the initial point cloud Zo and the real-time RGB images used for online point tracking. The RealSense camera records at 720p and 30 Hz. We downsample the resolution to 256x256 at Hz for the online point tracking algorithm to process. Additionally, we use smartphone positioned at different angle better record robot execution. desktop with 24 GB NVIDIA RTX 4090 runs the flow generation network and fow-conditioned imitation learning policy inference and online point tracking algorithm.
21

---

E.2 Real-Worle Evaluation Protocol
In this section, we describe the details of the evaluation protocol in the real world.
1) Initial State: When evaluating our system, we do not match the background scene setup those recorded in the human demonstrations such that we also test the generalization capability of flow generation network. For the initial position of objects, they are placed with roughly the same distribution as in the human video demonstrations.
2) Success Metric: Below are the details of the real-world success metric for all four tasks:
Pick & Place: The robot needs to pick up the cup and place it into the red plate. We do not require the mug to be centered on the plate. We consider one episode successful if: i. the robot can steadily place the mug onto the red plate; ii. the plate does not move more than 5 cm on the table during the robot's manipulation process.
Pouring: The robot needs to pick up the cup, hover it near the bowl, and exccute pouring actions. We consider one episode successful
the robot demonstrates the complete pouring behavior by rotating the cup more than 30 degrees; ii at least half of the cup overlaps with the red bowl the x-axis in terms of world coordinates.
Drawer Opening: The robot needs to fully open the drawer without colliding with its surface. Since the drawer is not rigidly attached to the table, it may slightly slide during the robot's s execution. An episode is considered success if: i. the robot gripper does not hevaily collide with the drawer surface and push the drawer back more than cm; ii. the drawer fully opened;
Cloth Folding: The robot needs to fold the cloth from one corner to the diagonal opposite corner. We consider an episode successful if: i. the robot folds the corner as indicated by the generated flow; ii the corners within threshold once the robot completes the execution. Due to the large sim2real gap for deformable simulation, set this threshold as 7 cm.
E.3 Evaluation Procedure
Im2Flow2Act with/without alignment: For each task, we first record the initial frame with differ- ent initial states for 20 evaluation episodes. We then query Grounding DINO on the object of interest obtain the bounding boxes in all initial frames. Using the bounding box. initial frame, and the task description, we generate the object flow (lask flow) for all episodes and store them as a buffer on the disk. With all ingredients for policy inference set, start policy evaluation for each episode by manually matching the initial states to be close to pixel-perfect within the mounted RealSense camera and load the corresponding generated flow from the previously saved flow buffer.
Heuristic- based policy: To obtain ground truth future point clouds for the objects, we first record human demonstrations for 20 evaluation episodes. We store both RGB and depth images during this process. For each evaluation episode, manually match the initial states to be close to pixel- perfect and obtain the open-loop action sequence by estimating object pose transformations between the initial frame and future frames for each time step in human demonstrations. We use the same motion filters in Im2Flow2Act to ensure fair evaluation. Furthermore, we provide the maximum available points (without downsampling) from the task flow. We manually check the transformed point cloud by overlapping the transformed initial frame point cloud and future frame point cloud ensure the transformation largely correct under the noisy conditions of the real-world depth camera.
F
Training Details
F1 Flow Generation Network
Condition Injection. The text condition are passed into the CLIP [69] text encoder to obtain text embedding. We also input the initial frame and the initial object keypoints Fo into the model to en- sure the generation process considers the spatial relationship between objects and the scene. We use the CLIP image encoder to encode the initial frame, yielding p2 embedding for each patch from
22

---

the last ViT [70] layer. The initial keypoints Fo are encoded using fixed 2D sinusoidal positional encoding. All conditions are injected into the denoising process through cross-attention.
Training Details. For the rectangular flow image, we set the spatial resolution to H W = 32 and T = 32. generating flow for 1024 keypoints over 32 steps. We finetune the decoder from StableDif- fusion for 400 epochs with a learning rate of 5e - 5. To obtain these keypoints, we uniformly sample them from the bounding box provided by Grounding DINO. For training AnimateDiff, we insert the LoRA [Low-Rank Adaptation) with rank of 128 into the Unet from StableDiffusion and train the motion module layer from scratch with learning rate of x 10-1 for 4000 epochs using AdamW [71] optimizer with weight deacy X 10-2, betas (0.9, 0.999) and epsilon 1 x 10 We load the pretrained (openai/clip-vit-enn
weights from CLIP [69] to process the initial frame and freeze them during the entire training. Zero-initialized linear layers used to process the patch em- bedding and the initial keypoints embedding before passing the conditions into the cross-attention layers.
F.2 Flow-Conditioned Imitation Learning Policy
Training Data Format: A training sample consists of (pt, ft. at. Fo:T), where Pt is the propriocep- tion data, a sequence of actions at..... aL+L of length L, and fi contains the locations (u, v) of N 128 object keypoints in the image space at time I. We set the object flow (i.e., task flow) horizon T' = 32, which matches the output of the flow generation network. The action sequence length is set to 16. The N keypoints are randomly selected from all available keypoints for every training sample during the training process. To construct the task flow Fo:r, we randomly select T frames from the episode length 7' to which the training samples belong. This allows the policy can be trained with diverse task flow as the flow generation model is trained with human data which typ- ically has different execution pace compared to robot. Further, constraint the length of task flow For be much shorter than the episode length T" to ease the training difficulty of flow generation model otherwise it needs to generate long- horizon task flows. To ensure the task flows are complete, we include both the first and last frames of the episode in the task flows.
We project the keypoints' initial 3D coordinates, Xo. into 192-dimension vector using a linear layer. We also encode keypoints' locations (u, v) in image space into another 192- dimension vector, using fixed 2D sinusoidal positioning. These two vectors are concatenated to form the descriptor e, with a total dimension of 384. We then pass all keypoints' descriptors into the state encoder é, which is a transformer with 4 encoder layers. It outputs a state representation of dimension 384 using a CLS token.
State Encoder:
Temporal Alignment: As discussed in the main paper, during training, we first encode the remain- ing task flow fer into 2, E Z. This process involves encoding the keypoints at each time step fer into St.2 via the state encoder. Next, we encode the future state representation 8t.7 into the latent space through the encoder &, which is implemented as a transformer with 4 encoder layers. We use fixed ID sinusoidal positional encoding to preserve temporal information in the state representation SEr before feeding them into E. The Temporal Alignment model is implemented as a transformer with 8 encoder layers. We also add fixed ID sinusoidal positional encoding to all inputs and utilize a CLS token for making predictions.
We use the diffusion policy [51] our action head. We use DDIM sched- uler with 50 training diffusion steps and 16 inference steps.
Diffusion Action head:
We train the policy for 500 epochs with learning rate Ic-4 using AdamW with weight deacy 1x 10- betas (0.9. 0.999) and epsilon x 10-8
G Inference Details
In this section, describe the details of the inference process, which includes Grounding DINO, motion filters, and online point tracking.
23

---

G.1 Grounding DINO
For each task, we begin by using Grounding DINO to identify the object of interest. We manually provide the keyword to the model; however, this process could potentially be automated using a large language model to find the desired object in the task description. Specifically, we employ the grounding- dino-base model to extract the object's bounding box. The keywords used for the pick & place and pouring tasks "green cup". For drawer opening, the keyword is "yellow drawer". and for cloth folding, it is "checker cloth". The input images are processed at resolution of 480x640.
G.2 Motion Filters
We use motion filters process the object flow (i.e., task flow) generated from the flow gener- ation model. As explained in the main paper, the initial keypoints are constructed by uniformly sampling within the bounding box. This approach inevitably yields keypoints that are not on the object, specifically, keypoints that fall on the background. To address this, we deploy several filters simultaneously to remove these background keypoints. Additionally, we implement depth filters to eliminate keypoints that lack depth data from noisy real-works depth image.
Moving Filter: In the training set, keypoints sampled on the background remain static in the image space, as only the object is moving. Therefore, we deploy moving filter during inference time to remove keypoints whose movement in the image space (256x256) is below a certain threshold. We find that this filter effectively eliminates most background keypoints. In real-world experiments, we set the threshold as 20 for pick & place, pouring, and drawer opening tasks, and as for cloth folding.
SAM Filter: To further remove points after applying the moving filter, we deploy the Segment Anything Model (SAM) [72]. Specifically, we first resize the initial frame to 256x256 and pass it through SAM to obtain the finest segmentation. We then iterate through the keypoints and filter out those where the area the located segment exceeds a threshold. We use a high threshold value of 10,000 for all tasks to prevent filtering out keypoints on objects with rich textures.
Depth Filters: Real-world depth images often noisy and contain many "holes." We filter out keypoints where the depth value is missing (i.e., the value is zero).
We randomly select N=128 keypoints which is the same number we used for training after applying motion filters as the policy input.
G.3 Online Point Tracking:
We utilize the online point tracking function from Tapir [48] to track the filtered keypoints during inference. We resize the visual observations to 256x256 and run the online point tracking at 5Hz.
H Simulation Cross Embodiment Demonstration Details
As we described in the main paper, we collect sphere agent demonstration to mimic the human demonstration in the real-world. In this section, we provide details for collection process. We use the same set of pre-defined heuristic action primitive collect the sphere demonstration. Please refer the appendix Sec. for details.
PickNplace: At the start of each episode, both the mug and the target plate are randomly positioned on the table. The mug's 's initial position is randomly sampled along the x-axis between (0.42, 0.76) and along the y-axis between (-0.1, The target plate's 's initial position is randomly sampled along the x-axis between (0.42, 0.65) and along the y-axis between (0.24, 0.32). The robot first executes the grasp primitive, lifting the mug to a randomly selected height between (0.15, 0.20) meters above the table. After lifting the mug, the robot performs the move primitive, positioning the mug above the target plate. The hover position is randomly sampled from (0.1, 0.25). Finally, the robot places the mug onto the plate and opens the gripper.
24

---

Pouring: The initial position distribution for the mug is the same as in the PickNplace task. The initial distribution for the target bowl is the same as that for the target plate in the PickNplace task. The robot follows the same procedure as in PickNplace before moving the mug, including the primitives used and the random parameters sampled. After lifting the mug, the robot moves near the target bowl and hovers the mug at a position sharing the same x-axis value. The robot then executes the pouring primitive, which rotates the mug clockwise around the y-axis. The rotation angle is randomly sampled from (66 16
Drawer: The drawer's initial pose is randomly sampled at the beginning of each episode. Its initial position is randomly sampled from (0.45, 0.62) on the x-axis, (-0.4, -0.32) on the y-axis, and (0.05, 0.18) on the z-axis. The drawer's initial rotation is sampled from {8 2 The robot executes the open primitive as described in Appendix Sec. C.
Folding: The cloth is randomly placed on the table at the beginning of each episode. We describe the initial position using the corner pinned with cube. The corner is randomly placed between (0.32, 0.44) on the x-axis and (-0.32, 0.32) on the y-axis. Values beyond 0.44 on the x-axis often lead to unstable simulations due to IK errors. The cloth is randomly rotated between (- - } Instead of randomly sampling from the nine predefined folding points as described in Appendix Sec. C, the robot will only fold the cloth diagonally.
The initial position distribution for the drawer is the same as in the Drawer task. However, the sample range on the z-axis is modified to (0.05, 0.12) to ensure the cube remains within the camera frame when placed into the drawer. The cube's initial position distribution is the same as the mug's s in the PickNplace task. The robot executes the open primitive to open the drawer, the grasp primitive to grasp the cube, and finally, places the cube into the drawer. The input parameters for all primitives are the same as those used in PickNplace, with modified sample range for the hover position before placing, (0.1, 0.15), to keep the cube within the camera frame.
Long Horizon:
I Additional Limitation
The system assumes the camera viewpoint is calibrated between simulation and testing environ- ments, with actions visible from the camera. Moreover, using flow as an action abstraction over- looks some action details, limiting performance in dexterous tasks like in-hand manipulation. Our flow generation model relies on accurate object detection, which could be improved with advanced object detection/segmmenttion models [73]. Finally, we focus on "manipulation tasks" rather than locomotion, where manipulation is defined by changing object states through interaction. However, in scenarios where object state changes on touch screen"), our method may not apply.
reflected in object motion (e.g., "touching button
25


Reviews:
[{"review_id": "zw4qqMG0rH", "reviewer": "Reviewer_WWt7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper learns a text-conditional flow generation model to predict object-centric flow given task description and initial observation, and learns a policy network that utilizes the generated flow to predict actions that finish the task.", "summary_of_recommendation": "The flow representation is interesting but seems to have some inherent insufficiencies for more general policy learning. I would like to hear authors' insight on this.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "I would like the authors to answer the two questions in the review part. And here are some more detailed questions:\nL140: can grounding dino help locate one of two identical objects by description of location? how to get  object of interes? Is it specified?\nL160: what does it mean by slightly finetune?\nL185-187: how to select initial keypoints? Why do we need 3D keypoints for initial observation but only 2D keypoints for the future frames? Why not using 2D or 3D for all the frames?\nL234-238: would like to see the performances with flow generation model trained only on simulation or real-world data separately. It would help validate the idea that connecting simulation and real-world data with flow representation helps policy learning.\nSupp L31: “opening the drawer to different extents”. Are you using programmed actions for play data? Would like a clarification on how to get actions in play data for all the environments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5LapIDe69", "reviewer": "Reviewer_aaJx", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a control policy that uses flow as its action representation. This enables policy learning from both robot and human demonstrations. The authors conduct real-world experiments to verify the effectiveness of the method and compare against heuristics-based policies.", "summary_of_recommendation": "I find this idea of using flow as action representation quite interesting. But I have many concerns, I will increase my score if my concerns are addressed.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Flows are inherently noisy, lack real groundtruth (e.g. Optical Flow), and may not contain full information of an intended action. This poses many concerns about this approach: \n    - Does this method work on tasks that require high precision, such as insertion type of tasks?\n    - What about tasks that require rotation actions, such as screwing? The flow will be difficult to imagine.\n    - What about certain actions that require only the moving of robot arm without objects? For example, reaching a certain button. \n    - How to deal with the inherent randomness of flows? There are at least two types of randomness: 1) object movements are non-determinant, which can results large changes in flow for the same robot actions. 2) The output of off-the-shelf flow models is also noisy. \n\n2. The proposed method seems to have a strong assumption: Every task action can be represented by a single object, and that object can be correctly detected. This limits the types of tasks to which the proposed method can be applied. For example, tasks that require tool use usually involve the interactions of multiple objects. Locating the key object alone can be very difficult.\n\n3. The authors mentioned the ATM paper (any point trajectory modeling for policy learning) but did not thoroughly address the difference between Im2Flow2Act and ATM. ATM also proposes a flow-like action representation but does not use object-based flow. Based on question 2, this could even be an advantage for ATM for being a more general framework. \n\n4. The proposed method generates the entire task flow and proposes a temporal alignment module to determine the current task progress. Why not just predict the flow for the next few timesteps? Is there a particular reason for generating the complete flow? Moreover, generating a complete flow seems to be infeasible considering the complex interaction of different objects in a long horizon.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2szO06HRW", "reviewer": "Reviewer_GbPT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents Im2Flow2Act, a general framework for imitation learning on cross-domain and cross-embodiment demonstrations with image inputs. The authors design new flow-prediction and flow-conditioned policy modules and show the effectiveness of their methods on different real-world tasks.", "summary_of_recommendation": "I think the main contribution of this paper is a new flow-generation network and a new flow-conditioned policy network, but it lacks enough experiments to show the supeority beyond previous designs. If the author can add enough experiments to show the superiority of their method compared to previous works such as ATM, GeneralFlow, and Track2Act, I will imporve my score.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Why should we use Stable Diffusion for flow generation? To my knowledge, the latent space of SD is \"image features\" from internet-level pictures that mainly focus on the color, spatial structure, texture, etc., rather than \"flow features\" that consist of a matrix of coordinates. I think these two feature spaces are totally different. The author should give more explanation on this, and add an experiment to show the superiority of using SD (with Animatediff) models over using other generative models such as VAE, DiT, and so on.\n2. Why did the author only fine-tune the decoder of the VQGAN?\n3. The author only compares their model to some ablation versions of their own, ignoring comparisons to other baselines such as ATM, Track2Act, and GeneralFlow. The author should add these baselines in their experiments to show the superiority of their method.\n4. Why did the author design a \"full flow prediction network\" and an \"alignment network\" instead of directly training a single network that predicts the future flow at a given current observation?\n5. Is it necessary to incorporate the 3D point cloud of the object? This will bring burdens for the application since we need additional operations to get the 3D point cloud. The author should add experiments to show it is necessary to use the point cloud as part of the state.\n6. This work only considers object flow. What about the tasks that require considering two objects together? I think restricting the flow on a single object rather than the full image space (as in ATM) is a more restricted idea.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a framework for imitation learning across different domains and embodiments using image inputs. Reviewers generally agree that the paper addresses a significant problem and presents an interesting framework. However, concerns are raised about the originality of using flow as an interface, as it has been explored in previous works. The reviewers also highlight the lack of comprehensive experimental comparisons and sufficient baselines to validate the effectiveness of the proposed networks. Technical details need further clarification, especially regarding the motivation behind certain design choices and the practicality of flow generation using Stable Diffusion. The paper is well-written, but there are strong assumptions and limitations, such as representing tasks with single object flows, that need to be addressed. 
Following the rebuttal, two of the three reviewers engaged in discussions with the authors, and the majority of their concerns were addressed, with both leaning towards accepting the paper.
Decision: Accept