General Flow as Foundation Affordance for Scalable Robot Learning

Authors: chengbo  yuan
chuan  wen
tong  zhang
yang  gao
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-04 14:19:47

Abstract: We address the challenge of acquiring real-world manipulation skills with a scalable framework. We hold the belief that identifying an appropriate prediction target capable of leveraging large-scale datasets is crucial for achieving efficient and universal learning.
Therefore, we propose to utilize 3D flow, which represents the future trajectories of 3D points on objects of interest, as an ideal prediction target. 

To exploit scalable data resources, we turn our attention to human videos. We develop, for the first time, a language-conditioned 3D flow prediction model directly from large-scale RGBD human video datasets. Our predicted flow offers actionable guidance, thus facilitating zero-shot skill transfer in real-world scenarios.
We deploy our method with a policy based on closed-loop flow prediction. Remarkably, without any in-domain finetuning, our method achieves an impressive 81\% success rate in zero-shot human-to-robot skill transfer, covering 18 tasks in 6 scenes. 

Our framework features the following benefits: (1) scalability: leveraging cross-embodiment data resources; (2) wide application: multiple object categories, including rigid, articulated, and soft bodies;
(3) stable skill transfer: providing actionable guidance with a small inference domain-gap.

Main paper: 
 introduction: 
 We aim to develop a new framework that enables scalable learning for robot manipulation. With more data and larger model training in the future, this framework has the potential to progressively enhance the capabilities of robots, i.e., the scaling law that has been observed in LLMs [1] . Inspired by the LLMs training paradigm [2] , we believe that two key elements contribute to their strong generalization abilities: (1) a vast training dataset with a small inference domain gap, such as all texts from the internet in LLMs, and (2) a foundational prediction task, such as text-token prediction in LLMs. How can we translate these elements into robot learning?Confronted with the challenges of collecting real-world robot data [3, 4] , we pivot towards largescale human video datasets. These data resources guarantee scalability and a small inference domain-gap (no simulation-to-reality problem), key ingredients for effective generalization. Moreover, human manipulation data provides a vast, real-world resource rich in diverse physics interactions and dynamic behaviors that closely align with robot manipulation. The next step is to identify a proper prediction target. We propose affordance for this role. Rooted in Gibson's theory [5] , affordance concentrates on the actions associated with an object, remaining neutral to manipulators. This characteristic positions affordance as a cornerstone in humtn-to-robot skill transfer.What affordance format will lead to a foundation prediction target that is universal for object categories and provides actionable guidance for robot manipulation? In this paper, we propose General We propose General Flow as a foundational affordance. Our framework uses general flow affordance as a bridge representation for human-to-robot skill transfer. Trained solely on RGBD human video datasets, our system achieves an average success rate of 81% on 18 real-world robot manipulation tasks, highlighting a pathway for scalable robot learning.Flow as Foundation Affordance (as shown in Figure 1 ) to achieve this goal. This affordance elucidates the future trajectories of 3D points on the object of interest. Take the task of 'open Safe' as an instance (in the middle of Figure 1 ): the general flow represents future positions of points on the safe. Then, a robot can gain a resilient motion primitive for the opening skill by following the door's flow.Previous works [6, 7, 8, 9] have attempted to extract 3D flow representations from either simulation or real robot data. However, these approaches suffer from domain transfer gaps [6] or have limited scalability [7] . In contrast, general flow leverages real-world, scalable data resources, i.e., human videos, thereby eliminating both the sim-to-real domain gap and the dependence on burdensome robot data collection. The diversity of backgrounds, objects, and human behaviors in these videos also significantly enhances the robustness of real robot execution. We term our affordance "general flow" due to its capability for universal robot learning: (1) scalability: leveraging scalable human data resources; (2) wide application: multiple object categories, including rigid, articulated, and soft bodies. (3) stable skill transfer: providing actionable guidance with a small inference domaingap, even sufficient for zero-shot execution.In this paper, we first develop pipelines to extract 3D flow labels directly from RGBD human video datasets for model training. We find prediction of dense flow in real-world scene point clouds remains a challenge, primarily due to the variability of trajectory scales and the need to enhance robustness in zero-shot scenarios. To address these issues, we employ scale-aware strategies in the model aspect, complemented by augmentation techniques that focus on embodiment occlusion (human hand and robot arm) and query point sampling (3D points on objects of interest), thereby boosting zero-shot stability.Implementing a straightforward heuristic policy derived from closed-loop flow prediction, we evaluate our approach on a Franka-Emika robot in a real-world setup. Without any in-domain finetuning, our system accomplishes stable zero-shot human-to-robot skill transfer. Fueled by the rich actionable guidance of general flow affordance, with only one model, our system notches an impressive 81% average success rate in 18 diverse tasks across 6 scenes, covering multiple categories of object types like rigid, articulated, and soft bodies. These findings highlight the transformative potential of general flow in spearheading scalable general robot learning. 
problem formulation: 
 The affordance [5] typically consist of functional grasp and subsequent motion [10, 11] . While robust functional grasping has been extensively explored [12, 13, 14] , a general method for providing prior guidance for post-grasp movements remains a challenge, which is the primary focus of our paper.Understanding the semantics and geometry of post-grasp motion for accurate predictions is complex. For example, in the "open Safe" task, the model must infer that the door's point trajectory rotates around an implicit axis, while points on the base remain static. Additionally, it must note that the trajectories of closer points to the axis require a smaller predicted scale or length. To handle these, we introduce "general flow" as an affordance that provides actionable guidance for downstream manipulation:Definition of General Flow Given a perception observation S (from any embodiment) and a task instruction I, for N q 3D query points Q ∈ R Nq×3 in space, the general flow F ∈ R Nq×T ×3 represents the trajectories of these points over T future timestamps.In this work, we use point clouds from real-world RGBD camera streams as our perception state S. Our model processes natural language instructions I, scene point cloud features P s ∈ R Ns×6 (comprising N s points with XYZ + RGB attributes), and N q spatial query points Q ∈ R Nq×3 (comprising N q points with XYZ attributes). The aim is to predict a trajectory set, or "flow", denoted as F ∈ R Nq×T ×3 . For the i-th query point p i ∈ R 3 , its trajectory is defined as F i ∈ R T ×3 , with the absolute position at time t represented asF i t ∈ R 3 for t = 1, 2, • • • , T . Initially, F i0 is set to the input position of the query point p i . We term our affordance "general flow" to emphasize its broad applicability across different embodiments (from any embodiment to any robot embodiment, e.g. from human to Franka-Emika) and object categories (e.g. rigid, articulated and soft bodies). 
embodiment-agnostic flow prediction: 
 We propose a framework for general flow prediction that is agnostic to specific embodiments, which is outlined in Figure 2 . We first design pipelines to extract flow labels from RGBD human video datasets. To manage various scales of trajectories and account for real-world noise, we integrate key designs that enhance the model's scale-awareness and robustness in predictions. For the first time, we develop a language-conditioned 3D flow prediction model directly from large-scale RGBD human video datasets.We extract general flow labels from two types of cross-embodiment datasets. For rigid and articulated objects, we utilize the HOI4D dataset [15] to train our general flow prediction model. This extensive RGBD video dataset includes 16 categories and 800 objects, encompassing 44.4 hours of recording. It provides comprehensive 3D labels, such as active object segmentation, 3D pose, and camera parameters. To further explore general flow in soft object manipulation, we collect RGBD videos for the "fold clothes" task using the RealSense D455 camera, comprising 6 types of clothes, 30 trajectories, and 605 extracted clips. From HOI4D Datasets [15] : Utilizing the detailed 3D labels from these datasets, we first randomly sample points within the active object and then calculate its future position using ground-truth pose and camera parameters.From Collected RGBD Videos: We first perform Human-Object-Interaction (HOI) segmentation [16, 17] to obtain the active object mask. Points are then sampled within the mask, and the future 2D trajectory is tracked using the Tracking Any Point (TAP) models [18] . The 3D label of the general flow is computed through back projection in both the spatial and temporal dimensions.To reduce the effect of noise in the annotations and the pipeline, multiple filtering techniques are employed. Additionally, we retain the hand mask for potential use in training augmentations. More details are shown in Appendix B.We observe enhanced performance when predicting relative displacements rather than absolute positions. I.e., we predict∆p i t = F i t -F i t-1 instead of F i t .The illustration of the model architecture is shown in Figure 3 . Following the description in Section 2, we first use CLIP [19] encoder to convert instructions into semantic features. Then their dimensions are reduced via MLPs (to d I dimensions) to align with point features. We first concatenate aligned text features with point cloud RGB+XYZ features, and then concatenate the features of scene points and query points, forming merged point cloud features P M ∈ R (Ns+Nq)×(3+3+d I ) . The merged features are processed through a PointNeXt [20] backbone with a segmentation head to extract geometric features. Since humans may perform different action trajectories for the same task and scene, we utilize a VAE [21, 22] to capture this multimodal property. The features of query points serve as condition variables for a conditional VAE, generating the final predictions. We leave more details of architecture in Appendix C.1.A primary challenge in real-world flow prediction is the significant variance in trajectory lengths across different query points within a task. For instance, in the "open Safe" task, the trajectories of points on the door are substantially longer than those on the safe body. To address this, we apply the Total Length Normalization (TLN) to uniformly rescale trajectories. We define trajectory scale for each query point p i as L i = T t=1 ||∆p i t ||. For the original prediction target {∆p i t | t = 1...T }, we define the normalized target {∆n i t } as:EQUATIONThen the VAE predicts the scale L i and normalized trajectory ∆n i t separately. Our ablation study demonstrates that TLN yields the best performance compared with other normalization methods (Appendix F.1). The final loss function comprises trajectory prediction lossL traj = 1 Nq i,t ||∆n i t -∆n i t || 2 , scale regression loss L scale = 1 Nq i,t || Li -L i || 2 ,and KL-divergence loss L KL of the VAE [21] . To minimize cumulative error, we also incorporate an MSE lossL acc = 1 Nq i,t || F i t -F i t || 2for the recovered accumulative shift. Thus, the total loss is expressed as:EQUATIONIn The collected general flow dataset is divided into training, validation, and test sets in an 80%, 10%, 10% ratio, with no identical object instances across sets. We adopt three types of models as our baselines (demonstrated in Appendix D).(1) 2D model: ResNet [23] , ViT [24] , R3M [25] . ( 2) Point clouds affordance model: VAT-MART [10] .(3) 3D model: PointNeXt [20] , which is an improved version of the PointNet++ [26] used in Flowbot3D [6] and ToolFlowNet [7] . We refer to our model as "ScaleFlow" in subsequent discussions. We trained multiple versions of the PointNeXt-based models, each with different PointNeXt backbone sizes.We use 3D Average Displacement Error (ADE) and Final Displacement Error (FDE) in centimeters [27, 28] as evaluation metrics. For VAE-dependent models, metrics are averaged over 10 samplings.The results in Table 1 demonstrate that our model achieves superior performance on all metrics, even with fewer parameters. We show the visualization of flow prediction in Figure 4 . Moreover, our system, trained at scale, demonstrates notable emergent properties such as robustness to hand occlusion, language-driven semantic control, resilience to label noise, and scale spatial adaption. We demonstrate these properties in Appendix E.  
zero-shot real world manipulation: 
 In this section, our goal is to demonstrate the efficacy of general flow as an actionable guidance [29] for downstream robot manipulation. We choose one of the most challenging scenarios: zeroshot human-to-robot skill transfer in the real world. Utilizing only one prediction model for all tasks paired with a straightforward heuristic policy derived from closed-loop flow prediction, we achieve an impressive 81% average success rate. This success spans categories including rigid, articulated, and soft objects, and covers 18 tasks across 6 distinct scenes.Here we present our heuristic policy based on close-loop flow prediction (Algorithm 1 in Appendix). We use a RealSense D455 RGBD camera positioned behind the Franka-Emika Arm to capture an ego-view stream. The robot's static base during manipulation acts as a reference for the FastSAM model [17] to segment the robot. More prompt points could also be employed via a designed GUI.Post-segmentation, we reconstruct 3D scene point clouds and select query points near the gripper. These query points, treated as a miniature rigid body, together with the scene point clouds and the text instruction, are fed into our to predict the general flow.The next step involves solving the corresponding transformation, an ICP (Iterative Closest Point) problem, based on the general flow prediction. We employ the SVD (Singular Value Decomposition) algorithm [30] to align the robot arm's end-effector transformation with the predicted flow. The choice of SVD is due to its robustness in handling noise and outliers, as well as its computational efficiency. Once the aligned transformation is computed, we use the Deoxys library [31] as an impedance controller in the operation space to execute the transformation accurately. We leave more details of deployment system and policy derivation in Appendix G.3 and Appendix G.4For real-world experiments (Figure 5 and Appendix G. As general flow affordance guides post-grasp motion, we manually position the robotic arm for task initiation. This can be replaced with automatic methods, as demonstrated in Ko et al. [32] . For storage furniture with handles, we evaluate the performance of both opened and closed grippers. Each task undergoes 10 trials, and the success rates are recorded. Discussions about the quantitative success criteria of each task can be found in Appendix G.1. We also discuss the real-world baseline model [11] in Appendix G.2.For result analysis, we focus on the following keywords and ask the following questions:• Transfer Ability: Does general flow facilitate stable zero-shot human-to-robot transfer?• Segmentation Error: How robust is the system against robot segmentation errors?• Novel Shape: Can this model generalize to the shapes of new categories that are significantly different from the training instances?• Grasp Manner: Is this object-centric system robust to variations in grasp manner?• Diverse Setting: How well can general flow adapt to diverse scenes and setting?Figure 5 presents a comprehensive overview of our analysis distribution. Following this, we delve into a detailed quantitative and qualitative examination to address these questions.Stable Zero-Shot Skill Transfer Our results, presented in Table 2 , demonstrate that using general flow as a bridge enables our framework to achieve stable zero-shot human-to-robot skill transfer. An impressive 81% success rate in such challenging settings underscores the strong transfer ability of general flow in cross-embodiment robot learning. To our knowledge, this is the first flow-based work to reach such a level of zero-shot transfer performance in real-world experiments. For tasks with success rates below 60%, we meticulously analyze the reasons and propose feasible future solutions in Appendix G.6. We also discuss the inference latency of our system in Appendix G.5.Robustness to Segmentation Error Our findings reveal that random hand mask augmentation during training significantly enhances the model's robustness to errors in the segmentation maps of FastSAM [17] . Figure 5 (a) illustrates this advantage with two examples. Notably, even with almost failed robot segmentation (as in the "open safe" task), our method still predicts meaningful flow to facilitate task completion in a closed-loop manner.Generalization to Novel Shape To probe the boundary of general flow's generalization capabilities, we experiment with a "Box" category, which can be approximated as an atypical design of "Safe". For comparison, we also test a conventional "Safe". Figure 5 (b) presents these instances. Surprisingly, the success rate for manipulating "Box" is even higher than that of the ordinary one (100% vs. 90% for the "open" task, Table 2 ), attributed to the structure of "Box" allowing more trajectory deviation without losing gripper on the door. This underscores the strong generalization capacity of general flow methods.As general flow is an embodiment-agnostic and object-centric method, it is expected to be resilient to variations in gripper position and grasp manner. To test this, we conduct manipulations using two storage pieces (a Refrigerator and a Drawer), leveraging their handles for different grasp methods. Figure 5 (c) displays these different execution manners. Our model successfully completes tasks regardless of the gripper's state.Handling Diverse Scenes and Directions We examine the extent to which general flow can handle changes in scene and direction. We distribute our tasks across six diverse scenes and perform scenebased prediction, eliminating the need for clean segmentation of the manipulated object. We also vary the direction of movable objects during our experiments. Figure 5 (d) showcases the most challenging example in this regard. We find that our heuristic policy successfully pushes a Toy Car in different directions to a certain extent. 
related work: 
 Embodiment-Agnostic Framework for Scalable Robot Learning We discuss more related works of real-world general robot learning in Appendix H. To leverage large-scale, cross-embodiment data resources [33, 34, 35, 15, 36] , multiple embodiment-agnostic frameworks [37] are proposed for robot learning. Prior works [25, 38, 39, 40] employ large-scale visual pre-training to develop embodied-aware pretrained representations, but these demonstrate limited generalization [41, 42, 43] . Alternative approaches seek to derive action signals from video generation [44, 45, 46, 47, 48, 32, 49] . Affordances extracted from simulators [10, 50, 51, 52, 53] are another focus, yet they struggle with the significant sim-to-real domain-gap, particularly in 3D environments. Recent efforts [54, 11, 55, 56] attempt to directly acquire geometric-aware structured information in human video but require in-lab training or suffer from unstable performance. Instead, we leverage 3D flow-based affordance to achieve reliable zero-shot solutions.Keypoints and Flow for Robot Learning Systems Previous studies use flow as action descriptors for robot learning [57, 58, 59, 60, 61] . However, these approaches either depend on embodimentspecific data, limiting their scalability [9, 7] , or are simulation-based, facing significant sim-to-real domain-gap due to imperfect real-world RGBD point cloud generation [6, 8] . Wen et al. [62] , Bharadhwaj et al. [63] utilizes flow as a prediction target for pretraining, but only operates in 2D flow and requires in-domain finetuning. In this paper, we extend the work of Seita et al. [7] into a more general version both in data resources and downstream applications. We acquire 3D flow prediction directly from RGBD human video datasets and achieve stable zero-shot skill transfer. 
conclusion: 
 In this paper, we introduce General Flow as Foundation Affordance for scalable robot learning. For the first time, we develop a flow prediction model directly from large-scale RGBD human video datasets and successfully deploy it with a heuristic policy for stable zero-shot human-to-robot skill transfer. Our framework marks a stride in achieving scalability, wide application, and stable skill transfer concurrently. We believe our work paves the way for innovative research in scalable general robot learning. We comprehensively discuss the limitations of our framework in Appendix I. We then proceed to extract general flow labels. Initial query points are selected within the masks of the objects of interest. Addressing segmentation noise, we maintain only the overlapping masks from the previous, current, and subsequent frames, using a homography matrix for projection onto the current frames. These points are chosen randomly, and their future trajectories are calculated based on ground-truth poses. We project all data back to the initial frame using the camera parameter labels. To correct for camera shake in the extrinsic parameter labels, we identify trajectories with shifts under 0.02 cm, compute their average as the camera shake, and deduct this from all points. 
 
 Appendix: Given the constraints of current RGBD Human-Object Interaction (HOI) datasets, which are either small in scale [34] or lack semantic richness [36] (mainly limited to pick & place actions), and considering the notable scarcity of resources for soft objects, we opt to collect our own RGBD videos. Our collection of RGBD videos for the "fold Clothes" task, captured with a D455 depth camera, includes 30 rollouts of 6 different types of clothing, resulting in 605 extracted clips. We plan to release this dataset in the future.Echoing the process used in HOI4D, we maintain a duration of 1.5 seconds for each clip, with intervals of 0.15 seconds. Initially, we apply HOI segmentation [64] to acquire masks for hands and active objects. Utilizing HOI detection results from [16] , we input bounding box outputs into FastSAM [17] for refined results. We retain only the masks with a confidence level above 0.5 for subsequent processing. After segmentation, we randomly sample 1024 points on the active object and employ co-tracker [18] to track their future positions in 2D pixel space. We exclude trajectories affected by occlusion, disappearance, or breakdown of depth values midway, and project the remaining trajectories back to 3D space and the first frame of the clip to derive the final general flow labels.Our pipeline functions automatically, without the need for manual intervention. As general flow captures the geometric dynamics of the physical world, extending beyond mere object-centric interactions, our system effectively manages noise factors such as segmentation and point sampling errors (e.g., selecting query points on non-target objects due to segmentation mistakes), particularly during large-scale training.While the main body of our paper covers the bulk of our design, we offer further details in this section. The alignment width for CLIP [19] text features is set at 6, aligning with the dimension of the original point cloud features (RGB+XYZ). In the conditional Variational Autoencoder (VAE) [21] segment, we utilize a 2-layer Multilayer Perceptron (MLP) to encode the latent variable. This is followed by another 2-layer MLP that functions as the VAE decoder. We employ separate 2layer MLPs for scale and normalized-trajectory prediction, each featuring a hidden dimension of 512. For ScaleFlow-B, our backbone configuration mirrors that of PointNeXt-B [20] . In ScaleFlow-L, we increase the backbone width from 32 to 64. Conversely, for ScaleFlow-S, PointNeXt-B is substituted with PointNeXt-S, and the CVAE utilizes a simpler 1-layer encoder and decoder, each with a hidden dimension of 384. The loss function parameters are configured as β 1 = 25 and β 2 , β 3 = 1, with a focus on enhancing scale prediction. We maintain the latent variable dimension at 16. For more detailed information, please consult the configuration files in our code repository.In light of the complex environmental challenges encountered in zero-shot real-world deployments, we propose two technical augmentations to boost zero-shot generalization robustness. The fundamental concept involves simulating various states of hand occlusion and distributions of query points, compelling the model to adapt to diverse conditions:• Hand Mask (HM) Augmentation: We encounter occlusions from human hands in our training data while facing occlusions from robot arms during deployments. Therefore, it is crucial to enhance the model's resilience to embodiment occlusions. To achieve this, we manipulate the presence of points on the hand in the input scene point clouds. We choose one of three rules, with probabilities p h1 =0.5, p h2 =0.2, and p h3 =0.3: (1) deleting all hand points; (2) keeping all hand points; and (3) sampling a random anchor point on the hand and retaining only points that are more than 12cm away from the anchor. These three rules are designed to simulate situations where the gripper experiences no occlusion, full occlusion, and partial occlusion during robot execution.• Query Points Sampling (QPS) Augmentation: Different downstream applications may require varying query point sampling methods. Consequently, our model must be adaptable to various query point distributions. We achieve this by augmenting the training process. In each training iteration, we select a subset of available query points using one of two rules, based on probabilities p s1 =0.7, p s2 =0.3: (1) complete random sampling; (2) randomly selecting an anchor query point and then choosing a specific number of points closest to the anchor. The complete random sampling ensures that all points are covered, whereas anchorbased sampling maintains a structure more aligned with the format used in downstream action guidance generation.For skills such as 'open safe', where most query points are static (e.g., points on the safe's body), direct model training leads to a strong bias towards predicting stationary trajectories. This results from a scale imbalance in our datasets. To mitigate this, we implement scale rebalance across the dataset. First, we employ the K-Means algorithm to cluster each data point's general flow by scale L i . As a result, we obtain N r clusters of 3D points. We represent the original point ratios of each cluster as {r i | i = 1..N r }. Except for the cluster with the largest number of points, we perform resampling for all other clusters. The resampled distribution is given by: ri = e ri/τ Nr i=1 e ri/τ(3) which is smoother than the original distribution. By default, we set τ to 1.We utilize 1.5s video clips as our training data and set the time steps of general flow to 3 for all data sources. The dataset is divided into training, validation, and test sets in an 80%, 10%, 10% ratio, resulting in 51693, 6950, and 6835 clips respectively, with no identical object instances across sets. Each sample consists of 2048 scene points sampled in an 80×80×80 cm 3 cube space around the center of the flow start points using the furthest point sampling (FPS) algorithm (a 40 cm perception range suffices for most tasks). During training, we randomly sample 128 query points, while for validation, 512 points are randomly sampled. Standard augmentation techniques for point cloud prediction [20] , including random rotation, shifting, scaling, coordinate normalization, color jittering, and feature dropping, are applied.In each training iteration, we randomly select 128 trajectories from the available flow labels. Additionally, to boost robustness in downstream zero-shot prediction, we implement technical augmentations as described in the main paper. For scale rebalance, we set the number of clusters to 4 and the default temperature to 1. The training process utilizes the AdamW optimizer with a learning rate of 0.001 and a weight decay of 0.0001. We incorporate 10 warmup epochs, followed by a cosine scheduler for 200 epochs. The training for ScaleFlow-B can be completed within 10 hours using an Intel(R) Xeon(R) Gold 5220R CPU and a single NVIDIA GeForce RTX 3090 GPU.Given that our datasets include ground-truth labels for object parts, we distribute 512 query points across each part equally during testing to enhance evaluation effectiveness. It's important to note that we do not use any part labels in model validation and real-word testing.We adapt three types of relevant work to our setting:• 2D Models: To investigate the importance of 3D geometry information, we employ pretrained ResNet [23] and Vision Transformer (VIT) [24] models from the timm [65] library as feature extractors. We finetune these models, combining their 2D visual features with aligned text features and processing them through an MLP for direct flow regression. We also evaluate the performance of the R3M representation [25] . Both finetuning and frozen modes are considered for R3M.• VAT-MART [10] : This model, originally designed for predicting affordance with single contact points, is adapted to our setting. We only utilize the 3D trajectory prediction branch of VAT-MART, replacing its task indentifier with aligned text features while keeping the rest of the model unchanged.• 3D Backbones: FlowBot3D [6] and ToolFlowNet [7] share a similar problem setting with ours. They originally used plain PointNet++ [26] for flow prediction in simulation without language supervision. For fair comparison, we implement an improved version, replacing PointNet++ with the stronger PointNeXt [20] backbone as a geometric feature extractor. The extracted features, combined with aligned text features, are then processed through an MLP for general flow regression.We preserve the architecture from the original repository, with the sole modification of incorporating text features before the prediction MLP to transition the model into a multimodal version. We derive aligned text features from the original CLIP features. The dimension of these text features is set to 32. For ResNet [23] and Vision Transformer [24] , we utilize the standard 'ResNet18' and 'VIT-B-224' versions, respectively. The default pretrained weights are loaded using the Timm library [65] .For R3M [25] , its 'ResNet18' version is employed. In our architecture, all MLPs dedicated to the final flow prediction consist of 2 layers, featuring hidden dimensions of 512 and 256, respectively.To ensure a fair comparison, scale rebalance, HMA augmentation and QPS augmentation are applied to all baselines. In this section, we demonstrate the notable emergent properties of general flow such as robustness to hand occlusion, language-driven semantic control, resilience to label noise, and scale spatial adaption.We first test robustness to hand occupancy in inputs across all 3D models, which are denoted as ADE-H and FDE-H (evaluations that include hand points in the model's input). As shown in Table 3 , with appropriate augmentations, our model are robust for hand occupancy.language-driven semantic control Through large-scale training, our model not onlyaptures rich semantic information but also becomes adeptly controllable through language modality. As depicted in Figure 8 (a)(b), our model demonstrates the capability to predict varied flows for identical input point clouds when provided with different instructions.Resilience to label noise Furthermore, it is remarkably robust to label noise. Figure 8(c)(d) showcases two instances of this resilience: despite severe label noise (notable deviation in "open Safe" and near-static in "pickup Toy Car"), our model accurately predicts the correct trend.Scale spatial adaption Additionally, our model could perform scale spatial adaption through scalable training. It dynamically adjusts its prediction scale in response to the spatial relationships of objects, such as ending on the table and scaling up for longer distances, as seen in Figure 8 (e)(f). All these emerging phenomena reveal the benefits of large-scale training. Table 5 : Result of the real-robot ablation study. We take the success rate as the evaluation metric.• w/o robustness augmentation: these variants omit three types of technical augmentations (Scale Rebalance, Hand Mask Augmentation, query points sampling Augmentation) to determine their impact on the model's prediction accuracy in our benchmark.The results of the ablation study are summarized in Table 4 . For all variants except those without robustness augmentation, there is a noticeable degradation in model performance. In regards to the ablation of the three technical augmentations, it is evident that they do not detrimentally affect benchmark performance. Notably, the hand mask augmentation even significantly enhances indomain prediction, which is an interesting observation.We also investigate the role of designed augmentations in enhancing the robustness of zero-shot transfer via real-robot ablation study. We select two representative tasks ("open Safe" and "close Drawer") for evaluation. The ablation comparisons included:• w/o Scale Rebalance: Utilizing the original flow label without rebalancing based on scale cluster results.• w/o HM Augmentation: Setting p h1 = 1.0, which means erasing all points on the hand throughout the entire training process.• w/o QPS Augmentation: Setting p s1 = 1.0, which means relying solely on random sampling for training query point selection.The results in Table 5 indicate that each augmentation significantly contributes to robust zero-shot execution.Notably, hand augmentation markedly affects tasks with substantial embodiment occupancy and occlusion, such as "close Drawer". Query point sampling augmentation emerges as particularly influential. Currently, the PointNeXt [20] architecture inherently couples the extraction of query point features, leading to a reliance on query point sampling augmentation in our framework. We anticipate that future advancements in disentanglement architecture within 3D learning will address this issue thoroughly. In our real-world experiment (Figure 10 ), we select 8 objects, including rigid, articulated, and soft bodies, as featured in our human video resources. We manually define multiple tasks and their corresponding success conditions for each object, resulting in a total of 18 distinct tasks. For a complete listing of these tasks, please refer to the content in the main paper (Section VI.B). For "Refrigerator" and "Drawer", we refer to them as "Storage Furniture" in the instructions. "Box" is also referred to as "Safe" since it can be seen as an approximation of an atypical design of "Safe". These objects are arranged into 6 scenes, as depicted in Figure 10 . For objects that are movable, such as "Mug" and "Toy Car", we randomly adjust their positions and orientations to add variability. It is worth noting that our experimental setup more accurately mirrors practical real-world scenarios compared to previous studies like Eisner et al. [6] . Our setup features diverse scenes and eliminates the necessity for clean object segmentation (left of Figure 11 ). This resemblance underscores the robustness and stability of our system in real-world scenarios.The criteria for successful task completion varied. For "pickup" and "push" tasks, moving the object in the correct direction by more than 15cm is deemed successful. The "Putdown" action is deemed successful if the object is ultimately placed on the desktop and the orientation of the object is appropriate (for example, the mouth of a mug facing vertically upwards). For the "open" task of the revolute articulation structure, an opening of 80 degrees is considered a success. For its "close" task, bringing the object to within less than 5 degrees of the fully closed state is regarded as successful. 5 cm (to fully open or close state) is used as a criterion for prismatic structure. The "fold" is considered successful if one end of the garment reaches the other end.To the best of our knowledge, Bahl et al. [11] is the only open-source work that shares a similar setting with ours, which involves learning a low-level affordance model directly from real-world human videos and could perform zero-shot prediction. We deploy this model in our experimental environment, using 'safe' and 'drawer' as text prompts for the grounding module [66] . Right of Figure 11 shows the visualization of the affordance prediction. Although it provides semantically meaningful predictions to some extent, it is limited by: (1) inadequate generalization for accurate motion direction prediction; (2) providing only 2D guidance without depth information; and (3) significant disturbances caused by embodiment occlusions. Given that the predicted post-grasp trajectories do not offer sufficient 3D guidance for closed-loop execution, we refrain from further robot execution trials.The heuristic policy based on close-loop flow prediction is demonstrated in Algorithm 1. Figure 7 shows a snapshot of our real-world deployment setup. We use a RealSense D455 RGBD camera to capture point cloud streams at a resolution of 1280 × 720, which is lower than the 1920 × 1080 resolution used in HOI4D [15] . Consequently, we opt for 0.01cm voxel downsampling during deployment, as opposed to the 0.02cm used in model training. The calibration parameters for our camera-robot system are (qw=0.911, qx=-0.015, qy=0.410, qz=-0.032) for orientation and (x=-0.265, y=0.260, z=1.095) for position. This configuration mimics a human ego-view manipulation perspective, beneficial not only for minimizing inference domain-gap but also aligning with practical applications in mobile robots. We utilize the RealSense camera's ROS driver for data acquisition. The robot's base, static during manipulation, serves as a prompt for the FastSAM [17] model for robot segmentation. For enhanced accuracy, more prompt points or customized models [37] can be employed. Post-segmentation, we reconstruct 3D scene point clouds and select query points within 10cm of the gripper. These, along with the scene point clouds and the text instruction, are fed into our prediction model (ScaleFlow-B in our experiments) to obtain the anticipated general flow. We then apply the SVD algorithm [30] to obtain a robust transformation aligned with the predicted flow. The robot arm is driven by the Deoxys library [31] to follow the derived SE(3) transformation in a closed-loop manner, achieving a 0.4s inference latency (0.05s without FastSAM).In practical applications, we note that the 6DoF controller for operational space in Deoxys lacks the necessary control precision for minute distances. Consequently, for trajectories shorter than 5 cm, we adopt a strategy of consolidating all steps into one and scaling this unified step to 5 cm in length. This method significantly enhances control accuracy over shorter distances, boosting the system's overall effectiveness and efficiency. Approximately 25% of predictions activate this workaround, which is considered acceptable given that the majority of tasks do not demand high levels of dexterity. Future improvements could include more precise controllers and calibration. The loop rate for our ROS system is set at 20 Hz. For safety, we manually confirm each planning step, although we find this almost unnecessary, as all experiments proceed with continuous pressing and confirmation without any delays.This section offers an in-depth explanation of our heuristic policy derivation. We begin by obtaining the gripper pose from the Deoxys API and projecting it into the camera's coordinates using calibration parameters. We select points within a 10cm distance from the gripper. Utilizing these points, we predict general flow and proceed to derive a 6DoF end-effector motion plan in camera space. For point clouds k t and k t+1 , each containing N points and representing adjacent timestamps, our objective is to identify a 6DoF transformation with rotation R and translation T that fulfills the following criteria:EQUATIONwhere w i denotes the regression weight inversely proportional to the distance d i between the i-th query points and the gripper position, with β set to 1. We solve Equation 4 using the SVD algorithm [30] for robust results. The acquired transformation T = ( R, T ) is then projected back into the robot's coordinates and adjusted to the gripper's coordinates for controller execution. • "Push Toy Car" (50% success rate): The Toy Car's direction requires a sophisticated semantic understanding. To improve our model's capability in this area, additional data collection is essential. Due to the toy car's small size, integrating a wrist camera could also help mitigate significant occlusion problems and enhance performance.• "Open Drawer" (30% grasping, 40% pulling): The mixture of prismatic and revolute structures in the HOI4D [15] datasets leads to a slight tendency towards including rotational components in predictions. Its negative impact is amplified in our fabric cabinet with high friction. The leather handle on the drawer also poses a challenge, often slipping from the gripper. Future enhancements could include using larger datasets with more robust language semantics [67] and redesigning the gripper.• "Open Laptop" (50%): The laptop's thin lid often results in poor or incorrect RGBD point cloud generation. Utilizing point clouds fused from multiple camera views could ameliorate this issue.Failure case videos are available in https://general-flow.github.io. In conclusion, most of these problems are addressable in future deployments. We systematically summarize these limitations and potential improvements in the next section.Research in general-purpose robotic manipulation in real-world settings is constantly evolving, with a focus on integrating Large Language Models (LLMs) for high-level planning [68, 69, 70, 71, 72, 73] and exploring direct actionable guidance through LLMs [74] , albeit with challenges due to overlooked physics dynamics. The development of large models for direct low-level control [75, 76, 77, 78] faces scalability issues due to intensive data requirements [3, 4] . This highlights the need for a training framework that achieves a balance between actionable output and scalability.In this work, we achieve this through a scalable robot learning approach based on general flow prediction.The diversity and volume of our data are currently limited, which restricts our model's ability to provide adequate guidance for complex tasks. A future direction could involve utilizing larger RGBD datasets [79] or RGB datasets [35, 80] combined with depth estimation techniques [81, 32] to address these issues.We currently employ a Conditional VAE to capture the multimodal distribution of human behavior in our dataset. However, this architecture may lack sufficient expressive power when dealing with more challenging scenarios, such as larger and more diverse datasets. To address this limitation, the Diffusion model [82] could be a potential solution.In addition, the manual grasping process could be replaced with heuristic policies such as those described in AVDC [32] , or through more advanced and automated methods like DexFunc [14] , MoKa [13] , and ManipVQA [83] in the future. It is also feasible to expand the representation from object-centric affordance to universal keypoints motion, similar to what is described in ATM [62] .The initially adopted ICP-based heuristic policy in a zero-shot manner might limit the completion of tasks, particularly those requiring contact-rich manipulation. Future studies could finetune policies based on general flow predictions within a few-shot imitation learning setting, such as Track2Act [63] , or use general flow as an action constraint [29] for sample-efficient reinforcement learning, akin to VRB [11] .We extend our gratitude to the following codebases for their support in the development of this work:• The model training framework and the 3D backbone are based on the codebase from Qian et al. [20] .• For HOI4D point-cloud data processing, we adopt methods from Liu et al. [15] .• For Hand-Object-Interaction (HOI) detection, we utilize the 100DOH tools from Shan et al. [16] .• FastSAM (Zhao et al. [17] ) is used for all segmentation in this work.• The co-tracker from Karaev et al. [18] is employed to track points in pixel space.• Implementations of ResNet, Vision Transformer, R3M, and the VAT-MART baseline are adopted from Bao et al. [27] , Nair et al. [25] , and Wu et al. [10] .• The ros-perception and data stream acquisition are based on Shridhar et al. [84] .• We inherit the SVD transformation solver directly from Zhong et al. [85] .• The impedance controller for the end-effector is adopted from the Doexys library (Zhu et al. [31] ). 


Decision: This paper proposes using 3D flow as a more general affordance representation for scalable robot learning. All reviewers recognize the importance and relevance of this work and are generally positive about the experiments and analysis presented.

Before the rebuttal, a major concern among the reviewers was whether the method could genuinely be considered "foundational." Although the definition of a foundation model may vary, the reviewers suggested that a more diversified and in-depth evaluation across a larger set of novel scenes might be necessary. Additional concerns included the need for manual grasp initialization, reliance on RGBD data, and the lack of detailed comparisons with previous methods.

During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns by providing further clarifications on their method’s generalization capabilities and better positioning their work within the existing literature. The final ratings were one Strong Accept, one Weak Accept, and one Weak Reject. The reviewer who rated the paper as Weak Reject did not engage in the discussion during or after the rebuttal. From my perspective, the authors provided sufficient clarifications to address this reviewer’s concerns.

Overall, this is an interesting paper that offers a new perspective on affordance representation. However, the authors should carefully revise the paper to ensure it is fairly positioned within the broader literature.

Reviews: review: title: title: Official Review of Submission178 by Reviewer 7196
paper_summary: paper_summary: The paper presents a method for robotic manipulation using a language-conditioned 3D flow prediction model. The key idea is the use of 3D flow, which represents future trajectories of 3D points on task-relevant objects. The model is trained on relatively large-scale RGBD human video datasets and achieves zero-shot skill transfer to robots, with an 81% success rate across 18 tasks in 6 scenes. The approach leverages the scalability of human video data and incorporates several methods of enhancing robustness and stability.
main_review: main_review: This paper presents a novel approach to robot learning by using RGBD human video data to train a language-conditioned 3D flow prediction model. The proposed method, which predicts future trajectories of 3D points, demonstrates impressive zero-shot transfer to robots, and provides promising results with real-world tasks. The paper's analysis provides strong evidence for the method's performance and robustness. However, the requirement for grasp initialization, complexity of the architecture, and reliance on RGBD data are significant drawbacks that may limit scalability. 

**Strengths** 

- **Potential for Scalability:** The framework utilizes actionless RGBD human video data, potentially addressing the challenge of data scalability in robot learning.

- **Generalization:** The model demonstrates zero-shot skill transfer across multiple tasks and scenes. 

- **Robustness:** The use of augmentations such as hand mask augmentation and query point sampling augmentation improves the model's robustness to occlusions and diverse conditions. 

- **Comprehensive Evaluation:** The paper includes thorough experimental evaluations and ablation studies, providing detailed insights into the system's performance and properties. 

**Weaknesses**

- **Grasp Initialization Requirement:** The need to initialize with a grasp limits the range of possible tasks and scenarios, as evidenced by the lower success rate in tasks like pushing. 

- **Complexity:** The architecture is complex and not fully justified by the ablation studies, leaving room to question the necessity of all components. 

- **Inference Speed:** Although it is addressed in the appendix, the inference speed, particularly the latency due to segmentation, could be a bottleneck in real-time applications. 

- **Scalability Limitation:** The reliance on RGBD data, although mentioned as a limitation by the authors, does restrict the scalability to environments where such data can be consistently captured.
questions: questions: - Can you expand on your claim in Section 3.2 that using a VAE helps capture multimodal distributions, considering that a Gaussian prior and regression loss typically assume a unimodal distribution? For example, given the same point cloud and task description, if the flow distribution is multimodal, would you not observe averaging behavior between these modes in your predictions?

- Please clarify the generalization capabilities of your model. Are all object categories and tasks in evaluation seen in HOI4D or your custom dataset? Generalization from "Safe" to "Box" does not seem sufficient to claim generalization to novel object categories, as is claimed in Section 4.3.

- Could this method be extended to enable grasp prediction by adding query points not only to objects but also to the agent itself? Or does this expand the domain gap too much for zero-shot generalization?
limitations: limitations: 3
review_summary: review_summary: The paper introduces a promising approach to robot learning through the use of 3D flow prediction, enabling learning from actionless videos, a potentially impactful contribution to the field. The authors acknowledge several limitations, but some concerns remain. The initial results are promising, yet the dataset is relatively small, limiting the model's generalization capabilities. Additionally, the current heuristic method for transferring predicted flow to policies requires manual grasp initialization, which constrains task diversity. Scalability concerns exist due to the reliance on RGBD data, though future work incorporating monocular depth estimation methods may address these issues. Despite these challenges, the paper makes a significant contribution and deserves to be accepted, with several potential opportunities for further improvement.
title: title: Learning a general object flow prediction model
paper_summary: paper_summary: The paper introduces a method to learn a language conditioned object flow prediction model from RGBD data. This model is afterwards exploited for robot action generation. The authors claim an 81% success rate in Zero-Shot transfer.
main_review: main_review: The authors propose a method to learn a language-conditioned object flow prediction model, called ScaleFlow. The authors claim this model could be benefitial to learn robot skills directly from video. 

To achieve this goal, the authors:
- Generate a labeled dataset, combining available online datasets (HOI4D) and collected RGBD videos.
- Train a language-conditioned flow prediction network. To do so, they combine CLIP features with PointNeXt from ToolFlownet.
- Design a controller to generate robot actions given the predicted object's flow.

The paper is reasonable easy to read and they motivate their goal of learning a foundation model of object's flow as useful representation to scale from videos. 

**Strenghts**
- The ultimate goal of building an object's flow prediction model is sound for robotics as a middle way to learn robot skills from videos.
- The authors show impressive zero-shot transfer. This is particularly impressive given the small size of the used dataset.


**Weaknesses**
- It is hard to clearly find what are the novel contributions in the work and what is not. For example, ToolFlowNet already predicts the flow of the objects. The paper could benefit from clearly stating in the introduction on the shoulders of which previous work they are building their method. For example, they could reframe some paragraphs in the introduction stating what the community has been able to do until now and what is missing that they propose. Without this clarification, it is hard to measure the real contributions of the work.
- Also, related with the literature, previous works, such as ScrewNet [1] or FlowBot 3D also propose a category-free, general object motion prediction model. The authors could also clarify how their method differs from these previous approaches and what are the benefits of their approach with respect to them.
- The authors should clearly specify if the zero-shot transfer is happening in arbitrary novel objects or in objects belonging to the training set. If they test in only train, it would be nice to provide a table comparison on success rate on object's out of the training dataset.

In summary, despite the task of learning a foundation model for general object motion prediction is a highly relevant one, the paper should focus on:
1. Cleary state how their general method is different from previous approaches.
2. Present an extensive evaluation on objects in-demonstrations and out-of-demonstrations to show the generalizable capabilities of the presented model.

[1] Jain, Ajinkya, et al. "Screwnet: Category-independent articulation model estimation from depth images using screw theory." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
questions: questions: - Beyond the big picture of building a foundational model for object flow's, what are the technical contributions of the work?
- The Zero-Shot transfer is pretty impressive. Did you test it in object's that were not in the training dataset? 
- If not, how well does the model perform when you try to generate object's flows for objects that were not in the dataset? If the model does not adapt to novel objects, calling the model a foundation model might be an overclaim, and the paper would benefit from reducing the overclaims.
limitations: limitations: 1
review_summary: review_summary: After the rebuttal period, the authors correctly answered my concernes. Thus, I have updated my score to weak accept.
title: title: A well-motivated idea and a clean method, but need to rejudge its significance.
paper_summary: paper_summary: The paper aims to increase the scalability of zero-shot human-to-robot skill transfer in real-world language-conditioned manipulation tasks. It proposes to predict the 3D flow of RGBD videos as a general guidance for the downstream manipulation policy. To achieve robust flow prediction, scale-aware strategies and augmentation techniques are employed. Through experiments, the method demonstrates more accurate flow predictions over baseline models and high success rate in zero-shot human-to-robot skill transfer in real world manipulation tasks.
main_review: main_review: The paper presents good motivation and intuition of why predicting the 3D flows benefits zero-shot human-to-robot skill transfer. The flow learning pipeline is clear and simple enough. The scale-aware prediction target and the augmentation techniques are natural and make sense. The authors have done adequate experiments to demonstrate that the model can predict better 3D flows in different scenarios than baseline architectures. The real world manipulation experiments show that the predicted flow can be easily mapped to a downstream policy.

My main concern is the "generalist" and "foundation" of the 3D flow in robot manipulation, as is emphasized by the authors. 
1. The 3D flows are a coarse estimate of object affordance and inevitably comes with noise and inconsistency, which makes it not a suit for tasks requiring higher accuracy like contact-rich manipulation (e.g. insertion). For articulated objects, a policy based on 3D flows may break the object because 3D flows do not reveal the articulated property.
2. There is no natural way of getting the query points automatically. In the real world experiments, the authors manually select these points near the gripper, which largely affects its generalization ability. 
3. To use the predicted flows, the authors assume that the gripper is already in a good manipulation state to the robot (i.e. grasping), making it harder to directly apply to common tasks. Also, how to manipulate an object can depend on how the object is being grasped, but the predicted flow does not consider the grasping state (https://sites.google.com/view/task-oriented-grasp/).
4. It is unclear how robust the flows are to the change of the camera views.
5. It is unclear if the 3D flows can be used in tasks requiring obstacle avoidance (https://interactive-language.github.io/).
6. The authors utilize a VAE to capture the multimodality property, but there is no experiment showing that the model learns it. Can the model predict all possible flows in such tasks like "pushing T" (https://diffusion-policy.cs.columbia.edu/)?
7. The authors mention ATM (https://xingyu-lin.github.io/atm/) in the limitation section, which has a similar idea of predicting flows. More discussions on the difference is necessary, preferably with experiments. The authors need to clarify the benefits of predicting a 3D flow than a 2D one, and the potential limitation of the former compared with the later.
questions: questions: Please see the bullet points in Review. The authors should proof the generalist of the method in various scenarios, or limit their claims to the extent that the method actually applies to.
limitations: limitations: 2
review_summary: review_summary: I list the paper as weak accept because the idea is well-motivated and the method is clear and works in enough scenarios. However, the experiments are not sufficient to show whether the 3D flows are general and foundational, which requires more work. I will raise my score if the authors are able to address large part of my concerns.