RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation
Authors: Yuxuan Kuang
Junjie Ye
Haoran Geng
Jiageng Mao
Congyue Deng
Leonidas Guibas
He Wang
Yue Wang
Venue: CoRL 2024 (2024)
Keywords: Hierarchical Retrieval
Affordance Transfer
Zero-Shot Robotic Manipulation
Visual Foundation Models
TL;DR: This work proposes a retrieve-and-transfer framework for zero-shot robotic manipulation, dubbed RAM, featuring generalizability across various objects, environments, and embodiments.

Abstract: This work proposes a retrieve-and-transfer framework for zero-shot robotic manipulation, dubbed RAM, featuring generalizability across various objects, environments, and embodiments. Unlike existing approaches that learn manipulation from expensive in-domain demonstrations, RAM capitalizes on a retrieval-based affordance transfer paradigm to acquire versatile manipulation capabilities from abundant out-of-domain data. RAM first extracts unified affordance at scale from diverse sources of demonstrations including robotic data, human-object interaction (HOI) data, and custom data to construct a comprehensive affordance memory. Then given a language instruction, RAM hierarchically retrieves the most similar demonstration from the affordance memory and transfers such out-of-domain 2D affordance to in-domain 3D actionable affordance in a zero-shot and embodiment-agnostic manner. Extensive simulation and real-world evaluations demonstrate that our RAM consistently outperforms existing works in diverse daily tasks. Additionally, RAM shows significant potential for downstream applications such as automatic and efficient data collection, one-shot visual imitation, and LLM/VLM-integrated long-horizon manipulation.

RAM: Retrieval-Based A Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation
Yuxuan Kuang 1.2+ Junjie Ye Haoran Genga.3, Jiageng Mao', Congyue Deng",
Leonidas Guibas He Wang", Yue Wang'
University of Southern California, "Peking University, SStanford University "Equal contributions
Abstract: This work proposes serrieve-and-rraste framework for zero-shot robotic manipulation, dubbed RAM, featuring generalizability across various ob- jects, environments, and embodiments. Unlike existing approaches that learn manipulation from expensive in-domain demonstrations, RAM capitalizes on a retrieval-based affordance transfer paradigm to acquire versatile manipulation capabilities from abundant out-of-domain data. First, RAM extracts unified af- fordance scale from diverse sources of demonstrations including robotic data, numan-object interaction (HOI) data, and custom data to construct comprehen- sive affordance memory. Then given a language instruction, RAM hierarchically retrieves the most similar demonstration from the affordance memory and transfers such out-of-domain 2D affordance to in-domain 3D executable affordance in a zero-shot and mubodiment-agnostee manner. Extensive simulation and real-world evaluations demonstrate that our RAM consistently outperforms existing works in diverse daily tasks. Additionally, RAM shows significant potential for down- stream applications such automatic and efficient data collection, one-shot visual imitation, and LLM/VLM-integraee long-horizon manipulation. For more details, please check our website at https: /ywuuank.kkkkkk com/RAM/.
Keywords: Hierarchical Retrieval, Affordance Transfer, Zero-Sio Shot Robotic Ma- nipulation, Visual Foundation Models
Open the drawer plaase.
Hlerarchlenl Retrieval
Afordance Lifting
Action
The Pipeline of RAM.
(b) Zero-Shot Manipulation.
(c) One-Shot Visual Imitation.
(d) LLM/VLM Integration.
Figure 1: (a) We extract unified affordance representation from in-the-wild multi-source demonstra- tions, including robotic data, HOI data, and custom data, to construct a large-scale affordance memory. Given language instructions, RAM hierarchically retrieves and transfers the 2D affordance from memory and lifts to 3D for robotic manipulation. (b-d) Our framework shows robust generalizability across diverse objects and embodiments in various settings.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
A longstanding goal in robot learning is to develop generalist robot agent capable of performing diverse robotic manipulation tasks on common household objects in real-world settings. Crucially, such an agent must learn to generalize to manipulate unseen objects in unseen environments with unseen embodiments. Traditional approaches to achieving this goal often involve manually collect- ing extensive expert demonstrations through telcoperation, scripted policies, and similar methods, followed by imitation learning from these datasets [1. 2]. However, these methods are costly and labor-intensive, requiring significant human effort and prior knowledge of various objects and tasks. As result, the scarcity of usable real-world data challenges the generalization of policies trained on these datasets to unseen objects or environments.
On the other hand, apart from real-world and synthetic robotic datasets [1, 2, 4, 5, 6, 7, 8] that agents can directly learn manipulation policies from, there exist vast amounts of out-of-domain data rich with actionable knowledge, ranging from Hand-Object Interaction (HOI) data [9, 10, 11, 12], to Internet-scale videos of daily activities, Al-generated videos [13], and even sketches [14]. Despite efforts to leverage these data sources for robotic manipulation [15, 16, 17]. learning from out-of- domain samples remains elusive due to significant domain shifts. Addressing this issue requires rethinking how to unify and utilize the actionable knowledge hidden within these heterogeneous and often noisy data sources. To that end, we identify that the key is to represent the actionable knowledge as transferrable affordance, i.e., 'where' and 'how' to act [18, 19]. By efficiently extracting and transferring these affordances from diverse data sources to our target domain, we can overeome dopain- hurdles and data scarcity. In addition, the fast development of generalizable visual foundation models (VFMs), trained enormous Internet images, offers streamlined solution to seamlessly connecting diverse data realms with our target domain, facilitating the distillation of rich affordance from these data sources to enable generalizable zero-shot robotic manipulation.
Therefore, we introduce RAM, Retrieval-based Affordance transfer approach for generalizable zero-shot robotic Manipulation. RAM utilizes a retrieve-and-transffe paradigm for everyday robotic tasks in zero-shot manner. First, extract 2D affordance from diverse data sources like robotic datasets, HOI datasets, Internet images, etc., construct comprehensive affordance memory. Upon receiving monocular RGBD observation of an unseen object in an unseen environment, RAM employs an effective hierarchical retrieval pipeline, which selects the most similar demonstration from the affordance memory and leverages VFMs to transfer the pixel-aligned 2D affordance to the target unseen domain. convert the 2D affordance executable robotic actions, we further develop sampling- affordance lifting module. This module lifts the 2D affordance to a 3D representation that includes a contact point and a post-contact direction, which is directly executable by various robotic systems using off-the-shelf grasp generators [20, 21] and motion planners [22, 23].
Extensive experiments in both simulation and the real world show that our method surpasses existing works by a large margin ($4.3), thereby confirming its effectiveness and superiority leveraging large out-of-domain data. Notably, our method is bebboddimen--gnosttio and data-efficient quaalite that prior methods are struggling with-which is substantiated by abundant experiments conducted across various robotic platforms. Furthermore, we showcase the versatility of our retrieval- based framework through its applications in several key areas in $4.5, such as automatic and efficient data collection, one-shot visual imitation conditioned on human preference, and seamless integration with L.LMs/LLMs for long-horizon tasks with free-form human instructions. In summary, the key contributions of our proposed RAM are three-fold:
We propose retrieval- based affordance transfer framework for zero-shot robotic manipulation, significantly outperforming prior works, both in simulation and the real world. Our key insight is to transfer affordances in 2D, followed by a module to lift them into 3D for direct execution.
We propose a scalable module for extracting unified affordance information from diverse out-of- domain heterogeneous data for retrieval. This module can potentially generalize to other robotic tasks beyond the manipulation considered by this work.
Our pipeline enables variety of intriguing downstream applications, including policy distillation, one-shot visual imitation, and LLM/VLM integration to facilitate future research.

---

2 Related Works
Affordance for Robotic Manipulation. Visual affordance, which indicates where and how to interact with diverse objects from visual inputs, plays important role in robotic manipulation thanks to its simplicity and interpretability. Point cloud-based methods [18, 19, 24, 25] focus on learning point-wise score map as the representation of affordance and utilizing it to regress end-effector poses. However, these methods face severe sim-to-real gaps due to the depth noises of RGBD cameras in the real world. Other series of works such as HOI-Forecast [15] and VRB [16] aim to learn affordance from human videos in an end-to-end manner. However, the scarcity of training data diversity hinders their generalization unseen tasks and objects. Compared to these existing works, our method leverages the unified affordance representation extracted from diverse data sources and is thus generalizable to a wide range of unseen objects and environments in the real world.
Zero-Shot Robotic Manipulation. Due to the issues mentioned above, how to manipulate diverse kinds of objects in a zero-shot manner remains challenging topic. Existing works [26, 27, 28, 29, 30, 32, 33, 34, 35] attempt to solve this problem by leveraging the reasoning abilities of LLMs/VLMs. However, they heavily rely on pre-programmed heuristic action primitives to execute low-level tasks like "grasping handle". Other works [36, 37, 38) leverage intermediate representations as conditions to the policy to achieve data-efficient imitation learning and can generalize to in-domain tasks without direct demonstrations. However, these methods still require collecting in-domain demonstration for test-time training and thus cannot be generalized to in-the-wild domains. Compared to these works, our method is training-free, neglecting the need for heuristic policies or test-time training.
Learning from Demonstrations. Recent years have witnessed the rapid progress of imitation learning (IL). Multiple teleoperation systems and algorithms [39, 40, 41, 42] have shown great capabilities to imitate human demonstrations smoothly. There are also methods [43, 1, 2, 44) that leverage VLMs or co-train data priors to reduce the requirement for demonstrations. Other series of works [45, 46, 47, 48, 49, 50, 51, 52, 53, 54] learn manipulation from demonstrations based on retrieving in-domain demonstrations for IL or direct trajectory replay. However, these methods fail to generalize the wild without in-domain demonstrations. Recent work Robo-ABC [55] tries to leverage CLIP [56] and Stable Diffusion [57] to retrieve and transfer contact points, but it is limited to HOT and table-top grasping scenarios and cannot perform object manipulation in the 3D space. Our method, instead, not only makes the most of existing diverse out-of-distribution data sources to guide affordance transfer in 3D but also generalizes to a diverse set of in-the-wild tasks a zero-shot manner.
3 Method
To address the in-domain data scarcity, the core of RAM involves transferring out-of-doman knowl- edge to our unseen target domain. To achieve so, we introduce a retrieve-and-trame that capitalizes on visual affordance. In contrast to existing works [16, 55] that only predicts 2D affordance in the pixel space, RAM generates executable 3D affordance 3 (c3D 33D) where
approach
,D 3D E R are the 3D contact point and the post-contact direction respectively.
First, RAM extracts affordance information from diverse out-of-domain data sources, constructing an affordance memory M ($3.1) in source domain S. Then, given monocular RGBD image (IT D) from the novel target domain T along with a language instruction L, RAM leverages a three-step hierarchical retrieval procedure ($3.2) to identify the most similar demonstration from M. Leveraging VFMs, we establish image-pair correspondences to transfer the 2D affordance from the demonstration in source domain S to our target domain T ($3.3). Finally, through a sampling-bass affordance in the target domain is lifted back to 3D, resulting in the 3D affordance AaD ($3.4), which can be directly executed through grasp generators [20, 21] and motion planners [22, 23].
method, the 2D
3.1 Affordance Memory
To achieve generalization across objects, environments, and embodiments, RAM opts for unified affordance representation that can be casily acquired from diverse data sources in a scalable manner,
3

---

constructing a comprehensive affordance memory, M, that encompasses a wide range of skills and objects. This memory integrates subsets of demonstrations from real-world or synthetic robotic data MR. HOI data Mи, and custom data Mc. To unify affordance information from these varied sources, each entry in the affordance memory includes an object-centric RGB image 1 which the initial frame of interaction, set of 2D waypoints C (ci 2 -2 :....) indicating the contact point c and post-contact trajectories, along with task category T expressed in natural language. Therefore, the constructed affordance memory can be denoted as:
M MM Mи U Mc = ((1,T,C0
= (c c 2D ...
(1)
For different data sources, employ different strategies to extract and annotate affordance:
Robotic Data. In the real world or sim- ulators, obtaining camera parameters and robot proprioception straightforward. This allows us seamlessly project the end- effector's 3D position onto 2D image plane. For affordance extraction, first frame of a rollout for the image TS, en- suring the object is unobscured. Then, we identify the frame when the robot's 's gripper closes, and use the end-effector's ' 2D position this moment as the contact point cg Sub- sequent frames used to trace post-contact trajectories. In our experiments, utilize DROID [2] dataset for robotic data subset construction.
start with the
(a) Robotic Data
(b) HOI Data
(c) Custom Data
Figure 2: Affordance annotation of demonstrations from various data sources. We extract unified affor- dance information automatically (for robotic and HOI data) or with minimal human effort (for custom data).
HOI Data. Human demonstrations, compared to robotic data, are often considered more intuitive and efficient. For affordance extraction, we begin by identifying and grounding video segments where relevant events occur. Similar to the approach with robotic data, we select the initial frame from each segment to serve S Then, the contact point and post-contact trajectories are determined by averaging the hand keypoints across frames. We found this simple strategy can generate reliable hand waypoints. We utilize the annotations of video segments and hand keypoints from the HOI4D [12] dataset to construct the subset while those annotations can also be easily acquired through action segmentation [58] and hand-object segmentation [59] techniques.
Custom Data. Beyond the automatic collection of affordance information from robotic and HOI data, we also consider custom data that are manually annotated with minimal human effort. This allows to tailor manipulation strategies for novel objects by augmenting the affordance memory with newly annotated demonstrations. The annotation process can be realized by selecting the start and end points on an RGB image, followed by automatic interpolation between the two points.
Notably, our affordance memory built scalably and semi-auttmaticallyy featuring manipulation customization with minimal human intervention. Putting all subsets from different data sources together, easily establish affordance memory M spanning diverse objects, environments, and embodiments, enabling method's 's generalizability across various settings.
3.2 Hierarchical Retrieval
To correctly manipulate a new object, humans often turn to their memory for similar scenarios for guidance. Inspired by how humans think and Retrieval Augmentedd Generation (RAG) in LLM inference [60]. we proposed three-step hierarchical retrieval pipeline effectively retrieve the most similar demonstration from the agent's affordance memory M in coarse-to-fing manner, and leverage the demonstration as a hint to guide the manipulation.
Task Retrieval. Based on the language instruction L (e.g., "Please open the drawer to find some utensils."), we leverage text encoders (e.g., CLIP [56]) or language models (e.g., GPT-4 [61]) to retrieve the task T' that the instruction falls within ("open the drawer") and extracts the object name N ("drawer")) If the memory doesn't 't contain the queried object, the model will reason the object geometry and propose task (or multiple tasks) that potentially shares similar affordance.

---

Semantic Filtering. Then in the retrieved tasks, we can get the source RGB images { } of all demonstrations. For each source image, we can calculate its joint similarity with the target observation image 17 and the object name N, in the form
similarity cos(C (19), CLIP.(I"))
(OOCCIIP.PP)) CLIP.
where C.) and CLIP,(-) are CLIP [56] visual and text encoders, and cos(-) measures the cosine similarity between two embeddings. Then, we set a threshold to filter out demonstrations that have very low similarities whiic can be caused by a lack of intended objects, low-quality segmentation masks, low-light environments, and so on on-to improve the robustness of subsequent geometrical retrieval and affordance transfer.
Geometrical Retrieval. Large amounts of previous research work [62, 63, 64] reveal the emergent correspon- dence from large-scale unsupervised visual foundation models [65, 56, 57, 66]. The core concept is that the deep dense feature maps produced by visual foundation models contain rich geometrical and semantic information that can be used for dense keypoint correspondence matching, However, illustrated in [67, 55], these visual foundation models often struggle to understand the orientation of instances. Therefore, it is essential for our method to find a demonstration where the object oriented similarly to the target image. To this end, after task retrieval and se- mantic filtering, calculate Instance Matching Distance (IMD) [67] using Stable Diffusion feature maps to per- form geometrical retrieval from the remaining demonstrations to find the one with the most similar viewpoint. More details can be found in the appendix. By leveraging the hierarchical retrieval, we can retrieve the most suitable demonstration for manipulation, both semantically and geometrically. Experiment results show that performs better than single-stage retrieval pipeline, found in Table 3.
Tisk Retrieval
Semaatic Filtering
Goumetrica Retrieval
Open the dravez
Figure Illustration of our hierarchical retrieval pipeline.
3.3 2D Affordance Transfer
After getting the most similar demonstration from the affordance memory, aim to transfer the 2D affordance from the source domain S to the target domain T in generalizable way. Given the source image 1 and its contact waypoints C, leveraging dense feature maps by visual foundation models, we first perform per-point correspondence matching to obtain the corresponding waypoints in the target image. We then employ RANSAC algorithm [68] to remove outliers of target waypoints and fit line the 2D space. In this way, we obtain the 2D contact point 2DD which is the first transferred waypoint, and the post-contact direction 2D, leading the 2D affordance A (c2D, 72D). By establish robust affordance transfer from S to T.
doing
3.4 Sampling- Based Affordance Lifting
The 2D affordance A = (c 22D) obtained from the affordance transfer step cannot be directly used in the 3D space. Therefore, we propose simple yet effective sampling-bas basedd method to lift the affordance to 3D, which can be executed by the end-effector in the 3D space.
Based the contact point 2D and the partial point cloud generated from the depth map D. we back-project the 2D contact point to 3D to get the 3D contact point 3D using depth D. Then we crop the point cloud around 3D to acquire the local geometry the contact area. Based on the cropped point cloud, we first estimate the normal vector of each point and then perform K-Means algorithm [69] Figure 4: Affordance transfer and lifting.
Affordance
Affuntazce Sumpling
Aftentance
cluster these normals and get Top-K cluster centers. Then we back-project these cluster center normal vectors into the 2D space through camera parameters and
5

---

Object
AVG
Task
C
C
C
0
0
P
P
P
Where2Act [18]
2
34
2
54
2
68
2
0
/
/
/
20.50
VRB" [16]
8
62
56
16
66
4
12
10
18
28
44
60
30.77
Robo-ABC* [55]
20
58
22
60
30
46
30
28
26
40
54
66
60
41.54
RAM (Ours)
38
68
32
76
32
50
66
54
38
46
56
72
64
52.62
Table Success rates of object manipulation for different methods the simulation environment. O, C, and P stand for Open, Close, and Pi Pickup, respectively. denotes necessary modification for 3D adaptation. RAM consistently and significantly outperforms baseline methods in most tasks.
select the one with the least included angle with the 2D post-contact direction ,2D, In this way, we can obtain the 3D affordance _A3D_ = (c , -3D) for robotic manipulation.
To map the 3D affordance concrete robot actions, we sample dense grasp proposals on the cropped point cloud leveraging off-the-shelf dense grasp generators [20, 21] and select the closest grasp from @D, After grasping, we can utilize position control to move the end-effector in the direction of 3D or leverage impedance control in the real world based on the real-time impedance force feedback as in [70] to adjust the end-effector and safely perform the actions.
4 Experiments
4.1 Experimental Setup
To verify the effectiveness of the proposed method, we conduct extensive evaluations in both simulation and real-world settings.
In simulation, we adopt IsaacGym [71] as the simulator, GAPartNet [5], and YCB [72] datasets as object assets, and a flying Franka Panda gripper for manipulation. We collected over 70 objects in 10 categories and evaluated them on kinds of tasks, including Open, Close, and Pickup, comprising 13 different tasks. For each task, we conduct 50 experiments. The camera viewpoint is fixed, and the objects' positions and rotations are randomly initialized within certain range. For Pi Pickup tasks, several distractor objects also randomly placed. A manipulation success is defined by whether the DoF of interest (articulation joint height) exceeds threshold.
In the real-world setting, we conduct experiments that involve interacting with various real-world household objects. We employ Franka Emika robotic arm with a parallel gripper and utilize an on-hand RealSense D415 camera to capture the RGBD image. addition, we also utilize Unitree BI I robot dog equipped with Zl arm and RealSense D415 camera to show our method's cross-embodiment nature [73, 74], which is covered in $4.5.
4.2 Baseline Methods
We compare our method against three baselines, namely Where2Act [18], VRB [16], and Robo- ABC [55], where they represent training-based, 2D affordance-based and sstrive-andd-rrnnee methods respectively. Where2Act [18] and VRB [16] are trained on vast amounts point clouds and egocentric HOI images, respectively, while Robo-ABC and our method are training-free.
Note that VRB and Robo-ABC only predict 2D affordance in either contact point and direction (VRB) contact point only (Robo-ABC). To adapt them to the 3D space, we faithfully conduct necessary modifications. For Robo-ABC, we feed it with our collected affordance memory and use the proposed 2D affordance transfer module to generate a 2D trajectory. Subsequently, we follow the same procedure in our method lift 2D affordance to 3D for both methods. detailed discussion of baseline implementation can be found in the appendix.
4.3 Results and Analysis

---

For simulation and real-world settings, we adopt Success Rate (SR) as the major evaluation met- ric. Results of simulation experiments are shown in Table 1, from which we can see that our method outperforms all baselines in the vast majority of tasks, yielding an average success rate of 52.62%. Compared to our method, Where2Act [18] largely fails in the opening tasks that require precise contact point and grasp pose. VRB [16] also suffers from providing precise contact points for grasping, leading to suboptimal performance in opening and picking up tasks. Our reimplemented and improved Robo- ABC [55] yields the second-best performance. We further note such a performance of Robo-ABC significantly depends on our affordance transfer and lifting modules.
Object
AVG
Task
P
P
P
Robo ABC* [55] RAM (Ours)
4/5
4/5
50.0
3/5
2/5
3/5
A/5
4/5
5/5
66.7
Table 2: Success rates of object manipulation for different methods in the real-world environment. RAM yields a favorable real-world performance.
We also compare our method with Robo- ABC [55] in the real world on 6 tasks with 5 rollouts each, as shown in Table. 2. Results show that our method also outperforms the base- line due to our more effective retrieval pipeline. It also shows our method's 's cross-domain nature and easy sim-to-real transfer. Real-world videos can be found in the appendix.
RAM (Ours)
VRS
Haamired Scone
Apart from quantitative studies, we also show some qualitative results on affordance transfer with respect multiple data sources in Fig. 5. Visualization results show that our method can establish a more reliable affordance transfer, leading to more robust performance.
Open the disbwnbeer Openthe calteet
Open fhe drower
Open rrierwooee
Figure 5: Qualitative comparison of generated 2D) affordance. Source demonstrations retrieved by RAM are shown in the first row.
4.4 Ablation Studies
Probing deeper into our framework designs, we perform extensive ablation studies on various components. All studies are conducted on Open tasks of drawers, cabinets, and microwaves, with 20 episodes each. The metric Distance to Mask (DTM) reported along with SR as in [55], which measures the distance of the transferred contact point to the object handle in pixel space.
Ablation on Hierarchical Retrieval. As shown in Table 3, the removal of any submodules in the hierarchical retrieval pipeline results in degradation of SR and DTM. We further observed the deactivation of geometrical retrieval leads to a significant SR drop from 38.3% to 26.7%, showing the importance of geometrical alignment in affordance transfer.
Cross-Task Transfer. Although our affordance memory con- struction pipeline enables scalable skill acquisition from various data sources, it is still valuable to probe how RAM performs
Ablation Method
SR
DTM
w/o Task Rirvl.
31.7
4.99
w/o Sem. Filtering
6.69
w/o Geom. Rtrvl.
26.7
9.36
when the target task is beyond the scope of its memory. We evaluate this by blocking the given task from the affordance memory. As shown the 4th row Table 3, RAM maintains decent success rate of 25. 5..0% demonstrating an impressive cross-task transfer capability when facing unseen tasks.
Cross-Task
25.0
9.33
SD - CL.I [56]
15.0
57.4
SD DINOv2 [66]
23.3
5.09
SD SD DINOv2 (63]
36.7
3.04
Full Pipeline
38.3
3.39
Table 3: Ablation studies on differ-
Effects of Different VFMs. We further replaced Stable Diffu- sion (SD) used in our pipeline with other visual foundation mod- els to investigate the effects of different VFMs for affordance
components of method.
transfer. As depicted in Table 3, we found that SD [57] performs best on SR, and SD-DINOv2 [63] performs best on DTM, both surpassing CLIP [56] and DINOv2 [66] by large margin. Although SD- DINOv2 [63] excels in contact point matching. we choose SD for our pipeline due to its robustness in post-contact correspondence, higher success rate, and less inference time.
7

---

Effects of Data Amount. Ablation on the data amount in our retrieval memory M conducted as shown in Fig. 6. We reduced the retrieval memory for each task from minimum of 10% to full data. The results indicate that as the data amount increases, the overall performance also improves. Notably, using 50% of original data yields comparable results to using full data, demonstrating the data efficiency of RAM.
Success Rate (36)
Distamcc to Mark
33
20
35
60
Data Amount (%)
Figure 6: Performances of different data amounts for retrieval.
4.5 Downstream Applications and Discussions In this section, we aim show that method has the potential to enable broad spectrum of downstream applications, tackling general robotic problems.
Policy Distillation. As zero-shot robotic manipula- tion method, conditioned on language instructions, our RAM can perform fully autonomous exploration of the surrounding environment by effective trial-and-error without any human priors reward shaping. Therefore, the proposed RAM for efficient annotated data collection and learn an end-to-end policy from the affordance knowledge. To that end, we leverage our zero-shot pipeline to automatically collect successful demonstrations of opening drawer, a cabinet, or a microwave. Then, we learn an ACT policy [39] from these demonstrations. As shown in Table 4, using only 50 demonstrations for each task, our distilled policy achieve +-35.0% performance boost over the zero-shot pipeline, indicating the huge potential for automatic and scalable high-quality data collection for policy learning.
Task
AVG
VRB [16] 1/20
1/20
2/20
6.67
Zero-Shot 8/20
4/20
30.0
Distilled
12/20
14/20
13/20
65.0
Table 4: Policy distillation results on suc- cess rate. RAM performs fully autonomous exploration to collect high-quality demon- strations efficiently for policy learning.
One-Shot Visual Imitation with Human Preference. Apart from utilizing out-of-doman demon- stration retrieval for manipulation, our method is naturally adaptable for one-shot visual imitation for better controllability, given specific in-domain or out-of-domain demonstration. For example, as shown in Fig. 1c, given a tissue box with human preferences of picking up the tissue paper or tissue box, our method Tom and Jerry shows that our method is able to bridge the great domain gap between the real world and cartoon images, thanks to the generalizability of visual foundation models.
correspondingly to perform different visual imitations. Another example of
LLM/VLM Integration. Our method can also be easily integrated with LLMs/VLMs for open- set instructions [75] and long-horizon tasks, by decomposing them into smaller ones suitable for affordance transfer and other action primitives. As shown in Fig. 1d, given an instruction ("Clear the table."), "), we can leverage a VLM [76] to interpret and decompose it into several actions and primitives and leverage affordance transfer for certain actions to behave in a human-orientee way. This enables more flexible tasks with higher complexity. More details can be found in the appendix.
Conclusions and Limitations
Conclusions. We propose RAM, a novel pipeline for generalizable zero-shot robotic manipulation. At the core of RAM is a retrieval- based affordance transfer and lifting mechanism that effectively distills actionable knowledge from large out-of-domain data to an unseen target domain. This framework can potentially generalize other robotic tasks beyond manipulation, highlighting its versatility. In addition the major experiments where RAM outperforms its counterparts by large margins, also demonstrate the flexibility of RAM through its applications in several key areas.
Limitations. Despite compelling results, RAM shares certain limitations with prior works. For long-horizon tasks, takes multiple steps for our method to chain the actions, which might generate less natural long-horizon behaviors. Also, our method struggles with complex actions, such as screwing. Future works address these issues include integrating better foundation models for task planning and developing more advanced affordance transfer methods that directly lift 2D trajectories into 3D space to enable more complex tasks.

---

Acknowledgments
The authors express their sincere gratitude to Dieter Fox, Ruihai Wu, Yuanchen Ju, Jiazhao Zhang, Xiaomeng Fang for fruitful discussions and valuable feedback. This work is partly supported by gift from Google.
References
[1] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models, arXiv preprint arX0v:2310.08884, 2023.
[2] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, Ellis, al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprini (rXiv:2403.12945 2024.
[3] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. JEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[4] O. Mees, L. Hermann, E. Rosete-Beas and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3)77327-7334, 2022.
[5] H. Geng, H. Xu, C. Zhao, C. Xu, L. Yi, S. Huang, and H. Wang. Gapartnet: Cross-category domain eeneriiggnerallizalle object perception and manipulation via generalizable and actionable parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7081-7091, 2023.
[6] H. Geng, Z. Li, Y. Geng, Chen, H. Dong, and H. Wang. Partmanip: Learning cross- category generalizable part manipulation policy from point cloud observations. arXiv preprint arXiv:2303.16958. 2023.
[7] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. arXiv preprint arXiv:2303.008 000338 2023.
[8] W. Wan, H. Geng, Y. Liu, Z. Shan, Y. Yang. L. Yi, and H. Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-awant curriculum and iterative generalist- specialist learning. arXiv preprint arXiv:2304.0044 2023.
[9] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720-736, 2018.
[10] H. Luo, W. Zhai, Zhang, Cao, and D. Tao. Learning affordance grounding from exocentric images. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 2252-2261, 2022.
[11] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[12] Y. Liu, Y. Liu, C. Jiang, K. Lyu, Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21013-21022, June 2022.

---

[13] Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao, L. He, and I.. Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024.
[14] P. Sundaresan, Q. Vuong, J. Gu, P. Xu, T. Xiao, S. Kirmani, T. Yu, M. Stark, A. Jain, K. Hausman, D. Sadigh, J. Bohg, and S. Schaal. Rt-sketch: Goal-conditioned imitation learning from hand- drawn sketches, 2024.
[15] S. Liu, Tripathi, Majumdar, and X. Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282-3292, 2022.
[16] S. Bahl, R. Mendonca, L.. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile representation for robotics. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13778-13790, 2023.
[17] C. Yuan, C. Wen, T. Zhang, and Y. Gao. General flow as foundation affordance for scalable robot learning, 2024.
[18] K. Mo, L. J. Guibas, M. Mukadam, Gupta, and S. Tulsiani. Where2act: From pixels to actions for artieulated 3d objects. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 68 13-6823, 2021.
[19] R. Wu, Y. Zhao, K. Mo, Z. Guo, Y. Wang, T. Wu, Q. Fan, Chen, L. Guibas, and H. Dong. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference on Learning Representations, 2022. URL cestps://operreeee net/forum? www.ornmiiixiiiooooooooo
[20] C. Wang, H.-S. Fang, M. Gou, H. Fang. J. Gao, and C. Lu. Graspness discovery in clutters for fast and accurate grasp detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15964-15973, 2021.
[21] H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023.
[22] D. Coleman, I. Sucan, S. Chitta, and N. Correll. Reducing the barrier entry of complex robotic software: a moveit! case study. arXiv preprint arXiv:1404.32. 2014.
[23] B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, et Curobo: Parallelized collision- robot motion generation, In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8112-8 8112-119, 19...1111911111199111 IEEE, 2023.
[24] Y. Wang, R. Wu, K. Mo, J. Ke, Q. Fan, L. J. Guibas, and H. Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In European conference on computer vision, pages 90-107. Springer, 2022.
[25] Y. Geng, B. An, H. Geng, Y. Chen, Y. Yang, and H. Dong. End-to-end affordance learning for robotic manipulation. International Conference on Robotics and Automation (ICRA), 2023.
[26] W. Huang, Wang, Zhang, Y. Li, Wu, and Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[27] F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark- based visual prompting. arXiv preprint arXiv:2403.03174, 2024.
[28] Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision- language planning. arXiv preprint arX1v:2311.17842, 2023. 2023.
10

---

[29] H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models, 2024.
[30] H. Geng, S. Wei, C. Deng, B. Shen, H. Wang. and L. Guibas. Sage: actionable parts for generalizable articulated-object manipulation under language instructions, 2023.
Bridging semantic and
[31] Y. Ding, H. Geng, Xu, X. Fang, J. Zhang. S. Wei, Q. Dai, Z. Zhang, and H. Wang. Open6DOR: Benchmarking open-instruction 6-dof object rearrangement and a VLM-based approach. In First Vision and Language for Autonomous Driving and Robotics Workshop, 2024. URL
attps://one /ttp:////opeeeeeeeeeeeeee
[32] Y. You, B. Shen, C. Deng, H. Geng, H. Wang, and L.. Guibas. Make donut: Language-guider hierarchical emd-space planning for zero-shot deformable object manipulation, 2023.
[33] R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao, Q.
Wu, W. Ai, Z. Zhou, D. Terzopoulos, S.-C. Zhu, B. Jia, and Huang. Arnold: A benchmark for languag--rrundde task learning with continuous states in realistic 3d scenes. arXiv preprint arXtv:2504.0432.1 2023.
[34] Li, T. Liu, Y. Li, M. Han, H. Geng, S. Wang, Y. Zhu, S.-C. Zhu, and S. Huang. Ag2manip: Learning novel manipulation skills with agent-agnostic visual and action representations. arXiv preprint arXiv:2404. 17521, 2024.
[35] S. Nasiriany, F. Xia, W. Yu, T. Xiao, Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07877, 2024.
[36] H. Bharadhwaj. A. Gupta, V. Kumar, and S. Tulsiani. Towards generalizable zero-shot manipu- lation via translating human interaction plans. arXiv preprint arXiv:2312.0077: 2023.
[37] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639, 2023.
[38] C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning, 2023.
[39] T. 7. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 2301::2330..3555 2023.
[40] C. Chi, Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[41] Y. Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y.W.. Chao, and D. Fox. Anyteleop: general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577 2023.
[42] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body telcoperation. arXiv preprint arXiv:2401.022111 2024.
[43] S. Sontakke, J. Zhang, S. Arnold, K. Pertsch, E. Biyık, D. Sadigh, C. Finn, and I. Itti. Roboclip: One demonstration is enough to learn robot policies. Advances in Neural Information Processing Systems, 36, 2024.
[44] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint 1405 2024.
[45] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning, 2017.

---

[46] Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. In 2022 International Conference on Robotics and Automation (ICRA), pages 2434- 2444. IEEE, 2022.
[47] M. Du, S. Nair, D. Sadigh, and C. Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv preprint arXiv:2334408742, 2023.
[48] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and
J. Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. arXiv preprint arXiv:2308. 15975, 2023.
[49] N. Di Palo and E. Johns. Dinobot: Robot manipulation via retrieval and alignment with vision foundation models. arXiv preprini arXiv:2402.1318/ 2024.
[50] S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild. arXiv preprint DxXiv:207.094500 2022.
[51] J. Gao, Z. Tao, N. Jaquier, and T. Asfour. K-vil: Keypoints-based visual imitation learning. IEEE Transactions on Robotics, 2023.
[52] X. Zhang and A. Boularias. One-shot imitation learning with invariance matching for robotic manipulation. arXiv preprint arXiv:2405. 1311/7 2024.
[53] F. Malato, F. Leopold, A. Melnik, and V. Hautamäki. Zero-shot imitation policy via search in demonstration dataset. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7590-7594. IEEE, 2024.
[54] J. Sheikh, A. Melnik, G. C. Nandi, and R. Haschke. Language-conditionee semantic search- based policy for robotic manipulation tasks. arXiv preprint arXiv:23 2023.
[55] Y. Ju, K. Hu, G. Zhang, G. Zhang, M. Jiang, and H. Xu. Robo-abe: Affordance generaliza- tion beyond categories via semantic correspondence for robot manipulation. arXiv preprint ar arXiv:2401.0748/ 2024.
[56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PMLR, 2021.
[57] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[58] F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. arXiv preprint arXiv:21 10.08568, 2021.
[59] L. Zhang, S. Zhou, S. Stent, and J. Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In European Conference on Computer Vision, pages 127-145. Springer, 2022.
[60] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal. H. Küttler, M. Lewis, W.-t. Yih. T. Rocktäschel, et al. Retrieval-augmentee augmented generation for knowledge- intensive Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
inowledgeintensivee nlp tasks.
[61] OpenAl. Gpt-4 technical report, 2023.
[62] L. Tang, M. Jia, Q. Wang, C. P. Phoo, and B. Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:
2023.
[63] Zhang, C. Herrmann, Hur, L. Polania Cabrera, V. Jampani, D. Sun, and M.-H. Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[64] M. E. Banani, A. Raj. K.-K. Maninis, A. Kar, Y. Li, M. Rubinstein, D. Sun, L. Guibas, J. Johnson, and V. Jampani. Probing the 3d awareness of visual foundation models, 2024.
[65] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660 9660, 2021.
[66] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fermandez, D. Haziza, Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07195 2023.
[67] Zhang, C. Herrmann, J. Hur, Chen, Jampani, D. Sun, and M.-H. Yang. Telling left from right: Identifying geometry-awane semantic correspondence. arXiv preprint arXiv:2311.17034, 2023.
[68] M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6): 381-395, 1981.
[69] S. Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
[70] X. Li, M. Zhang, Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16211, 2023.
[71] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
[72] B. Calli, A. Singh, Walsman, Srinivasa, Abbeel, and A. M. Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[73] Y. Kuang, Q. Han, D. Li, Q. Dai, L. Ding, D. Sun, H. Zhao, and H. Wang. Stopnet: Multiview- based 6-dof suction detection for transparent objects on production lines. arXiv preprint arXiv:2310.05717.22 2023.
[74] J. Zhang, N. Gireesh, J. Wang, X. Fang, C. Xu, W. Chen, L. Dai, and H. Wang. Gamma: Graspabiltt-awware mobile manipulation policy learning based on online grasping pose fusion. arXiv preprint arXiv:2309. 15459, 2023.
[75] Y. Kuang, H. Lin, and M. Jiang. Openfinaa: Towards open-set zero-shot object navigation via vision-language foundation models. arXiv preprint 2rXiv:2402.10677 2024.
[76] Z. Yang. L. Li, K. Lin, J. Wang, C.-C. Lin, Liu, and L. Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision). 2023.
[77] T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, Cao, J. Chen, X. Huang, Chen, Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, Li, Q. Jiang, and I.. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024.
13

---

A Real-Robot Rollouts
Dynamic videos of real-robot rollouts can be found in the supplementary video and at our website: https://yuxuanks https///yuuuunkkcom/////////
B Data Collection and Affordance Extraction
B.1 Robotic Data
We adopt DROID [2] as source of robotic data, which includes 76,000 expert trajectories of robots conducting daily tasks, along with corresponding task instructions. To extract affordance information from DROID, we first query instructions for tasks of interest. For example, for the task "open the drawer," we perform hard search to filter out instructions that do not include the action "open" and the object "drawer."
Next, the filtered instructions sorted based on the 1.2 distances between their language embeddings and the query embedding. We generate these embeddings using the ftext-embedding- 3-mmm11" model of OpenAl. Based on the sorted instructions, we then select the Top-k episodes for further affordance extraction.
To extract affordance information from these selected episodes, we identify the cartesian position when the gripper is closed as the 3D contact point. We then track the gripper's position for the following 10 time steps or until the gripper stops moving for consecutive steps. This provides with the 3D post-contact trajectory. Using the provided camera parameters, we project the 3D contact point and the post-contact trajectory onto the first frame of each episode where the object is unobscured. Note that some affordance demonstrations are further refined manually by adding an offset or being removed due to inaccurate camera parameters. Should accurate camera calibration be available, the manual correction not necessary.
This method ensures precise and reliable extraction of affordance information from the DROID [2] dataset. Note that this method can also be used to process other robotic datasets (real-world or synthetic), such as [1, 3, 4].
B.2 HOI Data
In addition to the details in the main text, we further note that only average the hand keypoints within the object mask to determine the contact point and shift the post-contact trajectory accordingly. This ensures that the contact point is within the object for robust affordance transfer.
Note that this affordance extraction procedure can also be applied to other data sources where hand- object interactions are involved, such as more HOI datasets [9, 10, 11], vast amounts of unannotated human egocentric videos on the Internet, and user-provy eee-ipro demonstrations.
B.3
Custom Data
The annotation process for custom data affordance has been discussed in the main text. Additionally, the sources of custom data are highly diverse and configurable. For our experiments, we annotated objects of interest images obtained from the Internet (Google, YouTube, etc.).
Notably, custom data also come from variety of sources, ranging from user-captured images, cartoon images, Al-generated content, and even sketches, etc., demonstrating the flexibility and diversity of our data sources. This flexibility allows for a wide range of potential applications and extends our affordance memory's scalability to a greater extent. Examples of our custom data are shown in Fig. 7, which are merely ordinary pictures of various objects.
B.4 Affordance Memory Statistics
The statistics of our affordance memory can be found in Table 5.
14

---

Figure 7: Examples of custom data affordance.
Task Name
Icon
Data Source
Size
Open the drawer
DROID
Close the drawer
DROID
20
Open the cabinet
DROID
12
Close the cabinet
DROID
Open the microwave
DROID
42
Close the microwave
DROID
Open the dishwasher
Custom
10
Open the refrigerator Open the trashcan Pickup the pot Pickup the mug Pickup the bowl
Custom
Custom
20
DROID
11
HOI4D
149
HOI4D
252
Pickup the bottle
HOI4D
78
Total
660
Table 5: Affordance memory statisties.
C Implementation Details
C.1 Feature Extraction Using Foundation Models
We use different foundation models as visual feature extractors, including:
Stable Diffusion (SD) [57]. As illustrated in [62], given an original image zo. we first add noise of time step
move it to distribution 2,, and then feed the stable diffusion network fo along with for denoising to extract the intermediate layer activations as the diffusion features (DIFT). We use the same configuration in [62].
DINOv2 [66]. Extracting DINOv2 features is implemented by feeding the original image to the DINOv2 model and extracting the intermediate layer activations of DINOv2 ViT during the feed-forward process.
SD-DINOy2 [63]. As in [63], we first extract SD features and DINOv2 features and then do L2 normalization on them to align their scales and distributions. After that, we concatenate these two features together get the SD-DINOv2 feature.
CLIP [56]. Similar to DINOv2, We extract dense CLIP features by utilizing the intermediate layer activations of CLIP VIT.
15

---

C.2 IMD Metric Calculation
As in [67], Instance Matching Distance (IMD) is originally proposed to examine pose prediction accuracy. Given a source image 1 and target image /", their normalized and masked feature maps Fs and FT, and a source instance mask M the IMD metric is defined as:
IMD(I 11 II .MM
(p), E (1) - (p), (D), 17)) DEMS
where p denotes pixel within the source instance mask, (p)()) is the source feature vector pixel p. FT)
and NN(FF()) F7) denotes the nearest neighbor vector in the target feature map F7 with respect to the source feature vector. IMD measures the similarity of two images via the average feature distance of corresponding pixels [67]. Using IMD in the geometrical retrieval stage, we can accurately retrieve the demonstration where the object oriented in the most similar way as the observation.
C.3 Baseline Methods
Where2Act [18] is designed for articulated object manipulation only, which takes an object point cloud input and predicts point-wise actionability scores, action proposals, and action scores with three separate models. Another drawback of this method is that it processes the point cloud in task-agnostic way, leading to ambiguity of the generated affordance. We adopt it to the evaluation tasks by 1) randomly sampling the contact point from the predicted top-5 actionable points, 2) proposing 100 actions using the action proposal model, and 3) selecting the action with the highest action score.
VRB [16] predicts the contact point and direction only on 2D images. To make it applicable real manipulation tasks, lift the estimated 2D affordance to 3D using our proposed sampling-based affordance lifting module.
Robo-ABC [55] is initially designed for object grasping only, where only the contact point of source demonstration retrieved by CLIP [56] feature similarity is transferred on the 2D image, followed by AnyGrasp [21] for grasp pose selection. For a fair comparison, we feed with our collected affordance memory. To extend it for articulated objects, we use the proposed 2D affordance transfer module to transfer both the contact point and post direction. Subsequently, we follow the same procedure in method to lift 2D affordance to 3D.
D Experiment Details
D.1 Experimental Setup Details
In the simulation, we utilize a flying Franka Panda gripper for simplicity. We utilize cuRobo [23] motion planner for position control of the gripper.
In the real world, we adopt two different robotic systems. the Franka Emika robotic arm setting, we leverage on-hand RealSense D415 camera for RGBD perception and utilize Movelt! [22] motion planner for the transformation from the target end-effector pose to joint position trajectories. In the Unitree robot dog setting, we leverage a Unitree B 1 dog with Z1 arm, along with a Robotiq 2F-85 parallel gripper. The RealSense D415 camera is also hn-hand mounted, and we control the arm using the Z1 SDK for delta cartesian-space control.
For grasp generation, utilize AnyGrasp [21] produce grasp proposals, along with GSNet [20] with relatively low graspness score threshold and collision threshold for more dense grasp proposals in case there is no grasp pose close enough.
16

---

D.2 Additional Experiments
In this work, we primarily focus on object-centric manipulation and the transfer of primitive skills in a zero-shot manner. However, we also recognize the importance of evaluating our approach in more complex, cluttered environments.
To evaluate method more in-the-wild scenarios, have conducted additional experiments in cluttered environments, such the scene shown in Fig 8. As shown in Table 6, our method still outperforms VRB [16] and Robo-ABC [55], with performance dropping by only 6.7% compared to cleaner environments, demonstrating robustness and applicability in complex scenes.
Object
AVG
VRB* [16]
26
30
50
35.3
Robo-ABC* [55]
40
52
44.0
Ours
50
66
56
57.3
Ours (Clean)
56
72
64
64.0
Figure 8: An example of more cluttered table- top scene.
Table 6: Experiment results cluttered scenes.
E Failure Analysis
As holistic robotic system that enables zero-shot manipulation in novel scenes, failure can happen in various stages of our pipeline. In our experiments, most failure cases stem from poor retrieval (e.g., extreme viewpoints or rare poses) or from inaccuracies in per-point transfer using diffusion features (e.g., slight deviations in the nearest neighbor points in the target image). We did not observe any specific object geometries consistently leading failures. Additionally, some common failure modes found in previous works, such errors in grasp generation and motion planning.
system encounters
To provide further context, we present visualizations failure cases in Fig. 9. the first scenario ("close the microwave"), the door of the target microwave open at such an extreme angle that becomes almost parallel the line of sight. From this perspective, the door appears making it difficult to detect and resulting in suboptimal retrieval. In the second scenario ("open the cabinet")), the small, dark handles on the cabinet doors blend into the wood in low-light conditions. reducing their visibility. Consequently, the transferred contact point slightly misses the handle's 's exact position.
thin line,
Downstream Application Details
F.1 Training ACT Policy
For policy distillation, utilize ACT policy [39] perform imitation learning from our self- collected demonstrations. ACT is based on CVAE Transformer architecture and adopts the idea of action chunking to mitigate compounding errors that are common in behavior cloning (BC). More details can be found in their original paper [39].
We use RGB views (5 640 x 480 x 3) and the robot's proprioception as observation. We set the chunk size 60, and the latent space dimension to 512. We use LI loss plus KL divergence regularization for supervision. The number of training iterations is set to 200K, and set the learning rate to 10 and batch size to 8.
17

---

Source
Target
Close the microwave
Open the cabinet
Figure 9: Visualization of failure cases.
F.2 One-Shot Visual Imitation Details
For one-shot visual imitation conditioned on human preference, we pick out demonstrations either from our own in-domain demonstrations or from out-of-domain cartoon images (Tom and Jerry in this case). We ground and choose the first frame of interaction for I and use the custom data annotation method to extract affordance manually. We then skip the hierarchical retrieval step and directly use these chosen demonstrations for affordance transfer and lifting, followed by 3D affordance execution.
F.3 LLM/VLM Integration Details
For LLM/VLM integration, we utilize GPT-4V (gpt-4- sgpt--vvisionppeee -reviiin))eeew) [76] for task decomposi- tion and scene understanding. We also use Grounded-SANe [77] for object detection and segmentation produce 3D bounding boxes of objects the scene.
Specifically, we define 3 basic primitives: grasp (), move_to(). and release () for VI.M output. Note that these three primitives do not involve heuristics on specific object manipulation. Other than these primitives, when the VLM finds out there are relevant demonstrations in the affordance memory, will schedule the proposed RAM system as a retricval- -otrieval-augmented module to perform the action as a whole, followed by other defined primitives.
An example of our prompt and the VLM output is shown in Fig. 10.
18

---

Prompt
You tre an intelligent roses fog that has an am with paraliel aráspe sbject monipulation.
You given huran instruction to correctly nacipslate the abjects safely conditiored on the iestruction.
asservation, task
series of prinitives decostrratonn you can Laverage to pertors the instruction, you can freely decompose it into
First, you are entoedd with primitives, which are: 1. possp)), witeh takes in object naw, detects the otject, noes graspable pose, and closes the gripper. relesss(),
pose
plaminng
The calling format should like relesse)), stc.
powerful you called vrich utillie those devnttrations affordeere directly corressanting un-tacts, althout calling the prielsives, The tasts in your affordance wecry see:
plesty
Cpen: drower, aicrowave, distwasher, refrigerater, treshicnn towl, bettis
cailing format lite Open, sraver(), etc.
Eistruction and observation, please sholld convert the seguecee into fython 1ist for corvesience.
of prisitives foreat of function celling and Finally,
bastr
instructio: open the trawer and picc up the utenet). Aiservtioo:: drawer.
Your ostput:
cagent>
sirss opes
then utensil.
Sub-tasks:
1. open the
for cact subtask, will see whather directly parformed using affordance
For there are suecrstrations regarding opering dravers, as will directly use it, there dranestractons uteantis, but piekiig in stetler. Although contee peinttives rosure safety. drawer()
stencllt
3. sove pww_Ea((tarstl_aeoo Lbove, posiciio)
www__surellsstspsiieeo position retease()
Finally, sill convert the sequence isto python List: coutputs
(rower)) 'grasp(uteestt)", peve_tofutensil_ne (ttrppsttttt www..tareelae..nsstttnn)) 'release()]] c/ourpits
</4gent)
Ne. you need to clear this table, pleate find feactale sey of sning that. Fallmw the acsaple atowe:
instructio::
clese the table.
VLM Output
ragent>
decompose instruction into sub tasks,
clear
pice
Sub- tas-task::
1.Paex up the bettle tran the table.
z.Pove the bottle to the crashean.
for eacs sustask, will see whather directly performed using affordance
for i, there are cuncnstration regarding piccing up bottle, so i will directly use it. use grinitives stsure the correct safe eovement the trishcan relessing the aottis.
1,stexup sottlej)
Puroow inctrashcan posttiion
s.relasee()
will
list!
coutputs
('Paccup 2ostiel)". "
"aoee www.beocrrieennnpttttee.. "resease()"]
s/cutpsts
Figure 10: Example prompt and VLM outputs of our LLM/VLM integration system.
19


Reviews:
[{"review_id": "Lz2qoY8mEb", "reviewer": "Reviewer_Xxnf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The works proposes a unified representation for transfering action knowledge from robotic, human data, low-cost custom data.  The work proposes to us object affordance to transfer the knowledge of where and how to act. To enable the transfer of knowledge, it uses retrieval based method from a pre-processed data from a mix of robotic, human, custom data.", "summary_of_recommendation": "I recommend weak accept based on the concerns expressed in questions. Edit: I have updated my score to strong accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Why not fine-tune the vision models used for retrieval for affordance prediction instead of retrieval? \n\nAll the evaluation scenarios have one or two objects in the scene. Does the method also work in “in-the-wild” scenarios?\n\nI wonder what the contribution of each individual dataset (robotic, HOI, custom data) is to the evaluations.\n\nHow would the method work with increasing dataset sizes? Wouldn’t retrieval become a bottleneck compared to learning to predict affordance?\n\nWhile pre-processing human datasets, how does the method handle the occlusion of contact points?\n\nWould the method's reproject from 2D to 3D work for articulated objects like sliding doors?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfc2rrt1LH", "reviewer": "Reviewer_38ey", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes RAM, a method to retrieve, from a dataset of robot data, human videos, and more, affordances and actions that can be applied to new scenarios.", "summary_of_recommendation": "I overall believe this is a good paper, which proposes both an interesting method, demonstrating the benefits with respect to similar techniques from the literature, and that also provides a series of interesting insights to the reader regarding how to effectively perform visual retrieval, specifically by leveraging Vision Foundation Models.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1) How would you scale the method to trajectories more complex than the ones proposed?\n2) While matching pre-contact/contact/post-contact points in 2D, some of these may end up in the background of the scene. When using VFM features to match it on a new image, how do you avoid this sticking to a random background position?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOKHW3jiXp", "reviewer": "Reviewer_2wfH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper aims to build a robot agent that generalizes training-free to unseen tasks, environments and embodiments, using foundation models. The main argument is that collecting high-quality robot data for every task, environment and embodiment is infeasible. To this end, it proposes the collection of a bank of demonstrations from both robot videos as well as HOI videos. The bank contains image frames, paired with task descriptions and contact point trajectories. Given a new instruction, a relevant such triplet is retrieved from the bank and its point trajectories are mapped through correspondences in the current scene. The waypoints are then converted to actions. The method exhibits good results on articulated objects in sim and real, as well as several interesting ablations and applications.", "summary_of_recommendation": "I believe the paper is in a good state and has more merits than drawbacks. If the authors answer to the reviewers' questions it could be an even stronger submission.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See weakness above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Before Rebuttal**

Strengths. The paper is well written, well structured, and easy to follow. The research and proposed method are well motivated, with a clear novel technical contribution. The ability to extract affordances from multiple modes of data (both human interactions and robot interactions) is helpful for scalability. There is a broad range of experiments, showing the strong performance of the proposed method with respect to sensible baselines, and exploring the effect of design choices. There are real-world experiments studying performance on realistic, everyday objects.

Weaknesses. The method requires the dataset to be manually segmented, which potentially limits the scalability. The skills the robot is able to perform have relatively simple trajectories (although this is acknowledged in the paper's limitations section). It is unclear how strong the generalisation is to novel viewpoints not contained in the dataset. There is only a limited amount of clutter and distractor objects in the scenes, so although experiments are done on realistic objects, it is not clear how well the method would work in more cluttered, everyday scenes. For real-world experiments, only 5 test episodes per task is quite low for drawing meaningful conclusions with statistical significance. The paper does not compare to training-free methods, so we are missing an understanding of how retrieval performs relative to explicit policy learning. The paper lacks an analysis of failure modes. The discussion of the limitations could be deeper; introducing the limitations by stating "RAM shares certain limitations with prior works" seems to avoid an honest exposition of limitations unique to RAM.

---

**After Rebuttal**

After the reviews, authors responded to the questions and concerns raised by the reviewers, and it was good to see some back-and-forth between authors and reviewers. After the rebuttal, the AC and reviewers had a discussion, where there was unanimous agreement the the paper should be accepted. Reviewers still have some minor concerns regarding the assumptions required for the method to work well, but overall this is an interesting paper with some technical novelty, and good experiments and ablations.
Decision: Accept