Automated Creation of Digital Cousins for Robust Policy Learning
Authors: Tianyuan Dai
Josiah Wong
Yunfan Jiang
Chen Wang
Cem Gokmen
Ruohan Zhang
Jiajun Wu
Li Fei-Fei
Venue: CoRL 2024 (2024)
Keywords: Real-to-Sim; Digital Twin; Sim-to-Real Transfer
TL;DR: An end-to-end framework for automatically generating fully interactive simulated scenes ("digital cousins") from a single real-world RGB image enabling training of robot policies that can be deployed zero-shot in the real world.

Abstract: Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in *digital twins*, which serve as virtual replicas of a real scene but are expensive to generate and cannot produce cross-domain generalization. To address these limitations, we propose the concept of ***digital cousins***, a virtual asset or scene that, unlike a *digital twin*, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. As a result, *digital cousins* simultaneously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing a distribution of similar training scenes. Leveraging digital cousins, we introduce a novel method for their automated creation, and propose a fully automated real-to-sim-to-real pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and semantic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90\% vs. 25\% success rates under zero-shot sim-to-real transfer. Additional details are available at https://digital-cousins.github.io/.

Automated Creation of Digital Cousins for Robust Policy Learning
Tianyuan Dai Josiah Wong"', Yunfan Jiang'. Chen Wang', Cem Gokmen'
Ruohan Zhang! Jiajun Wu! Li Fei-Fei!.
Deppartment Computer Science, Stanford University
Department Mechanical Engineering, Stanford University Institute for Hlmann-Cettere AI (HAI), Stanford University
Abstract: Training robot policies in the real world can be unsafe, costly, and diffi- cult to scale. Simulation serves an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in digital twins, which serve as virtual replicas of real scene but are expensive to generate and cannot produce cross-domain generalization. To ad- dress these limitations, scene that, unlike digital twin, does not explicitly model real-world coun- terpart but still exhibits similar geometric and semantic affordances, As result, digital cousins simultancously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing distribution similar training scenes. Leveraging digi- cousins, we introduce novel method for their automated creation, and pro- pose fully automated real-to-sim-to-ree pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and seman- tic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90% vs. 25% suc- cess rates under zero-shot sim-to-real transfer. Additional details are available at https: / 6 g ta cous i ns.. g thub. io/.
propose the concept of digital cousins, virtual asset
Keywords: Real-to-Sim; Digital Twin; Sim-to-Real Transfer
AMM ACDC
Coais Orertir
Peler
3 Zaro-Siot Transher
Figure I: Overview. Fully interactive digital cousin scenes can be generated completely automatically from a single RGB image. Unlike digital twin, digital cousins relax the assumption of completely reconstructing the minute details of given scene and instead focus preserving higher-level details, such spatial relation- ships and semantic affordances. By leveraging motion planning and ground-truth simulation information, can automatically collect demonstrations our digital cousin scenes, augmented with physically plausible ran- domizations. A policy trained on these synthetic demonstrations can then be deployed zero-shot in the original scene, without requiring any additional finetuning.
*Denotes equal contribution. Correspondence to Tianyuan Dai
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Developing and training policy models for robotics in the real world can be unsafe, costly, and diffi- cult to scale with sufficient environment diversity. Learning in simulation is an attractive alternative, as it provides both an inexpensive and potentially limitless source of synthetic data that can be gener- ated at super real-time speed. Unfortunately, policies trained exclusively on simulated data require sim-to-real transfer, and often suffer from the semantics and physics disparity between the simu- lated and real-world environment. One broad approach to mitigate this issue is improve policy robustness by augmenting the distribution of synthetic data. Some efforts have sought to randomize over object-centric parameters such as visual semantics [1. 2] or physical parameters [3], whereas other methods have proposed scene-level distributions that are either curated [4-6] or procedurally- generated [7]. These methods, however, can lack the quality of synthetic interaction data at the scale necessary for real-world deployment.
In contrast generating distribution of environments, explicitly modeling fully interactive replica of specific real-world environment (a digital twin) can capture nuanced details within the original environment, but are labor-intensive to generate. While multiple recent efforts have explored reducing this cost by synthesizing real-world scans with either procedural [8, 9] or human- assisted [10] interactive object generation, these approaches can fail to capture necessary affordances needed for downstream tasks and still require human input. Ultimately, digital twins themselves are limited in their scope, as robot policies trained in these environments are optimized for a single real-world instance and cannot generalize to variations in the original scene.
To address the limitations of both extremes of sim-to-real approaches, we first propose the concept of digital cousins. We define a digital cousin as a virtual asset or scene that, unlike a digital twin, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. For example, we would expect an appropriate digital cousin of a real-world cabinet to share similar layout of handles and drawers, even if the material detailing differs between the two. A digital cousin of real-world kitchen might include similar arrangement of furniture objects, even if individual models slightly differ.
Unlike procedurally generated scenes, digital cousins are fundamentally grounded with respect to a real-world scene, similar to digital twins. However, unlike digital twins, digital cousins relax the requirement of reconstructing an exact replica, and scenes containing digital cousins instead focus on preserving high-level scene properties, such as spatial object layouts and key semantic and physical affordances. And, the name suggests, multiple distinct cousins can be generated for single
scene, whereas only a single digital twin can exist for that same scene. Thus, this relaxation serves two purposes: (a) it reduces the need for manual finetuning to guarantee a certain level of fidelity and thereby enables fully automated creation of digital cousins, and (b) it facilitates better robustness to variations in the exact original scene by providing an augmented set of scenes from which train robot policies.
Leveraging digital cousins, we then introduce novel method for the Automated Creation of Digital Cousins (ACDC) that can be used fully-automated end-to-end in a cal-to-sim-tooreal setup, in which digital cousins generated from a real-world image can be used to train policies deployed zero-shot in the original scene. ACDC leverages DINOv2 [11] as proxy for measuring similarities between given real-world asset and candidate digital assets, as it has been shown to visually encode relevant geometric and spatial information from diverse sets of images, and we consider assets with low feature embedding distances as being digital cousins of given real-world object.
Our contributions are threefold. First, we propose the concept of digital cousins and novel method ACDC for their automated creation from a single image requiring zero human input. Second, we provide an automated recipe to train simulation policies in digital cousins. Third, we show that robot manipulation policies trained within digital cousins can match the performance of those trained on digital twins, and can outperform digital twin policies when tested on unseen objects, both in simulation and in the original rorl-world scene. Code and videos can be found on the project website https: / /dggitalllaa legggiaalcouuinsss
cousins. itcubbioo..
2

---

Masked Otject
[' Toafae "bench" sex)
A ACDC
Matched rotation
Object tabel
DINO GPT
DID GPT
categories
datoset
Output
Digital
Model amapahots
Maiched model
2 Digital Cousin Matching
Real-Wortd Extraction
Sim-Scene Generation
"sofa b cab inst.
Q
Object masks
Repost for
Ploor
Groanded
("sofa" Object labols
Objoct Depth
Object point eloud
Caption
GPT
Matched
Depch Anything
model
Rotated
Depth map
Matched soene
Output
Imput
Figure 2: ACDC Pipeline. ACDC is composed of three sequential steps. (1) First, relevant per-object informa- tion is extracted the input RGB image. (2) Next, this information with asset datasct to match digital cousins each detected input object. (3) Finally, we post-process chosen digital cousins and generate fully-interactive simulated seene.
Methodology
In this section, we describe our fully automated end-to-end pipeline to generate and leverage digital cousins for sim-to-real policy transfer. Section 2.1, we describe ACDC, our automated system for generated digital cousins. In Section 2.2, we describe our method for automatically training simulation policies leveraging fully programmatic demonstrations.
2.1 Automated Creation of Digital Cousins (ACDC)
ACDC is our automated pipeline for generating fully interactive simulated scenes from a single RGB image, and broken down into three steps: (1) an extraction step, which relevant object masks extracted from the raw input image. (2) matching step, in which we select digital cousins for individual objects extracted from the original scene, and (3) generation step, in which the selected digital cousins are post-processed and compiled together to form fully-interactive, physically- plausible digital cousin scene. An overview of our method can be seen in Fig. 2. Further technical details can be found in Appendix A.
Real-world extraction. ACDC only requires single RGB image X taken by calibrated camera with intrinsic matrix K as the input. To extract individual object masks from the input image, we first prompt GPT-4 [12] to generate captions Cj,j E {1, M) for all objects observed in X. The captions are then passed to GroundedSAM [13] with to generate a set of detected object masks m,, i E {1...., N). re-synchronize the captioning between GroundedSAMM-v and GPT-4, we re- re-prompt GPT-4 to select the accurate label 1, {c, ) for each object mask m, from the previously generated caption list.
We additionally require a depth map in order to properly position and rescale matched digital cousins when generating our scene. Depth cameras are widely used but cannot accurately capture reflective surfaces and prevent usage on an-the-wild images. To mitigate these limitations, we leverage Depth- Anything-v2 [14], state-of-the-art monocular depth estimation model, estimate the correspond- ing depth map D from X. We then extract point cloud P D K-1, and leverage individual object masks m, to generate the subset of points p; from P and pixels x; from X corresponding to that object, resulting in a set of object representations fo, = (I,, m, Pa, xx) N

---

Digital cousin matching. Given our extracted object representations o, we perform a hierarchical search through our virtual asset dataset to match digital cousins. We assume that each asset in our dataset assigned a semantically meaningful category t, and that each asset model has multiple snapshots {iss )m of itself taken under different orientations, including a representative snapshot I, forming asset tuples {a, (t,, I, {it [1:s)= = )==1 Narnets where Nassets is the total number of assets included in the dataset. In this work, use the BEHAVIOR-IK [4] assets, though in practice, method can use any asset dataset that satisfies the above properties.
For given input object representation o;, we first select the matching candidate categories by com- puting the CLIP [15] similarity score between label 1, and all asset category names [t,),=n selecting the top keatt closest categories. Given the selected categories, we then select potential digi- tal cousin candidates amongst all the models within those categories by computing DINOv2 feature embedding distances [11] between the masked object RGB x, and representative model snapshots re-compute the DINOv2 distances over each candidate's individual snapshots {i,,)-- - and ultimately select the closest Accuse cousins, where each selected cousin consists of a specific virtual asset A. and corresponding orientation qe based on the selected snapshot.
I,. After selecting keand candidates,
Simulated scene generation. The final step is to compile our matched cousins into physically plausible digital cousin scene. For given input object information o, and corresponding matched cousin information (Ac, 9c), place the asset's bounding box center at the centroid of the cor- responding imput object point cloud pæ, and then rescale to align with p,'s extents. We additionally fit floor and wall planes from their obtained point clouds from the extraction step, and query GPT-4 to determine whether any objects should be mounted on either the floor or wall. Finally, we de-penetrate all objects so that the scene is physically stable. For additional scene post-processing details, please see Appendix A.3.
2.2 Policy Learning
Once we have a set of digital cousins, we train robot policies within these environments that can transfer to additional unseen setups. While our digital cousins are amenable to multiple training paradigms, such as reinforcement learning or imitation learning from humans, we choose to focus on imitation learning from scripted demonstrations, as this paradigm requires no human demonstrations and can instead be coupled end-to-end with our similarly fully autonomous ACDC pipeline.
To facilitate automated demonstration collection in simulation, we implement set of sample-base based skills that leverage both motion planning and ground ground-truth truth simulation data. Concretely, our skills in- clude Open, Close, Pick, and Place. For specific implementation details, please see Appendix A.4. While currently limited, these skills already enable demonstration collection across a wide range of everyday tasks, such as object rearrangement and furniture articulation.
Moreover, because our generated digital cousin scenes are both modular and configurable, we can casily apply broad domain randomization to these scenes without losing their underlying scene-level semantics through a combination of augmentations, including visual, physics, kinematic (pose and scaling), and instance-level randomization. Using our skills and domain randomization techniques, we can autonomously collect demonstrations across all of our generated digital cousin scenes and train behavior cloning policy from this offline data. For additional details, Appendix B.5.
3 Experiments
We answer the following research questions through experiments:
Q1 Can ACDC produce high-quality digital cousin scenes? Given a single RGB image, can the recovered digital cousins capture the high-level semantic and spatial details inherent in the original scene?

---

Input Scene
Causin Scene
Scale
Cat.
Mod.
Dist. (em)
Ori. Diff. (md))
Bbox IoU't
Cen. IoU
6/6
4.15 1
0.64 0.23
0.73 0.22
4.17
8/8
7.65 # 5.62
0.05 0.00
0.66 + 0.21
0.74
6.89
10/10
10/10
4.77 a: 3,38
0.03 0.01
0.74 = 0.20
0.77 = 0.19
10.23
15/15
15/15
15.67
8.86
0.12
0.59 1:9::..
0.72 0.14
Table 1: Quantitative and qualitative evaluation of nearest digital cousin scene reconstruction in sim-to-sim scenario. 'Scale' the largest distance between objects in the input scene. "Cat.' indicates the ratio of correctly categorized ohjects to the total number ohjects in scene. 'Mod.' shows the ratio of correctly modeled objects to the total numher of objects. 'L2 Dist.' provides the mean and standard deviation of the Euclidean distance between the centers of the bounding boxes in the input and reconstructed scenes. 'Ori. Diff.' represents the object. "Bbox IoU" presents the Intersection over Union (IoU) for assets' 3D bounding boxes. 'Cen. IoU" shows the loU for assets' 3D bounding boxes after aligning their center position. Please refer to Appendix B.1 for more results.
and standard deviation of the orientation magnitude difference of each centrosymmetric
Q2. Can policies trained on digital cousins match the performance of policies trained digital twin when evaluated on the original setup?
Q3. Do policies trained on digital cousins exhibit better robustness compared to policies trained digital twin when evaluated on out-of-distributtin setups?
Q4. Do policies trained on digital cousins enable zero-shot sim-to-real policy transfer?
3.1 Digital Cousin Scene Generation via ACDC
Experiment setup. We will show quantitative evaluation and qualitative results of recovered digital cousins answer Q1. To quantify the quality of generated digital cousins, we first test method on variety of simulated and real scenes to perform both sim-to-sim and real-to-sim digital cousin scene generation, where we input a single RGB image of a simulated scene and generate the closest digital cousins using ACDC. In the sim-to-sim setup, we have guaranteed access to its "digital twin" (i.e., the ground truth category and model), as well as ground truth information about all scene objects' poses and scales, and can quantitatively measure the reconstructive fidelity. this setting, we measure the proportion of scene objects whose category and model were successfully preserved in the nearest digital cousin capture the digital cousin's semantic fidelity, and measure the averaged per-object pose error (via L2 distance and orientation difference) and scale error (via bounding box loU) to capture its geometric fidelity. However, we do not have access to digital twins for the real-world objects nor ground truth spatial information; instead, we provide qualitative side-by-side comparisons between the real-world scene and its corresponding digital cousin scenes. Sim-to-sim results in Table real-to-sim results in Fig. 3, and additional results Appendix
Digital cousin generation: Semantic and spatial details preserved (Q1). In the sim-to-sim setup, we find that the original per-object category and model are correctly reproduced in most cases. Spatially, we also find that scales and positions of reconstructed digital cousins can similarly match their original counterparts in the input scene. Qualitatively, the side-by-side comparison of our input- and ACDC-generated scenes showcase the immediate visual similarity between the two, and suggest that our quantitative results imply a digital cousin scene quality that can successfully preserve the original scene's 's object layout. In the real-to-sim setup, we find that ACDC produces reasonable scenes that are both physically plausible and able to preserve scene-level semantic and spatial details.
5

---

Input IGB Image
Recunstructed Scenes
Figure 3: Qualitative real-to-sim digital cousin scene generation results. Multiple cousins are shown with robot collecting demonstrations. Please refer to Appendix B.2 for more results.
Twin
Cousins
Consins
8 Cousins
All Asseta
Task 1: Door Opening
Task Drawer Opening
Task Putting Away Bowl
Siccess Rate (%)
00
15
80
(3
10
70
40
5
7.25
18.03
9.42
14.97
17.00
14.17
14.44
17.33
DINO Distance
Figure 4: Sim-to-sim policy results. Aggregated success rates of policies trained on the exact twin, different numbers of cousins, and all assets in the three nearest categories. Policies are tested on four setups: the exact digital twin, and three increasingly dissimilar setups measured DINOv2 embedding distance probe zero-shot generalization. Note for Task 3, there are much fewer cabinet models that enable the task be feasible, so we only compare the digital-twin and 8-cousin policies. Note that during digital cousin training data does not include any of the evaluation instances. Additional information at Appendix B.6.
Summary. Based on these results, we can safely answer Q1: digital cousins can indeed preserve semantic and spatial details of input scenes, reconstructed from single RGB image that can be accurately positioned and scaled to match the original scene.
3.2 Sim-to-Sim Policy Learning with Digital Cousins
Experiment setup. To answer Q2 and Q3, we then analyze our ability to train robust robot policies using ACDC-generated digital cousins on three tasks: Door Opening and Drawer Opening, in which the robot must open furniture equipped with either a revolute- or prismatic-joint, respectively, and Putting Away Bowl, in which the robot must open a cabinet's drawer, pick up a bowl on the cabinet and place it the drawer, and finally close the drawer. We compare policies trained on digital cousins against those trained either exclusively the digital twin on all feasible object setups. In each case, training data consists of 10000 sampled programmatic demonstrations leveraging analytical skills and divided equally amongst the number of training cabinet instances. However, as the Putting Away Bowl task has a much longer horizon compared to the other tasks, we constrain that task's total demonstration count to 2000 to maintain roughly the same training dataset size.
For each policy, evaluate 100 rollouts over six runs on both the original digital twin setup as well multiple unseen setups with increasing DINOv2 embedding distance. Our aggregated results shown in Fig. 4. Additional training details and ablations can be found in Appendix B.6.

---

Digital cousin policies can match digital twin policy performance (Q2). As digital twins perfectly model the target object, policies trained on digital twins serve as oracles for our within- distribution test, and we find that when evaluated on this setup, digital cousin-trained policies can often perform similarly to its equivalent digital twin policy despite not being trained on that specific setup, We hypothesize that because our digital cousin policies are trained on data collected across different setups, can cover a broad state space that generalizes well to the original digital twin setup, However, on the other extreme, we also find that policies trained on all feasible assets perform much worse compared to the digital twin policy, suggesting that naive domain randomization is not always unequivocally useful and that digital cousins serve as a more beneficial, conditional form of randomization.
Digital cousins improve policy robustness (Q3). In held-out setups unseen by both the digital twin and digital cousin policies, we find that the performance disparity sharply increases. While policies trained on digital cousins exhibit more robust performance across these setups, the digital twin policy exhibits significant degradation. This suggests that digital cousins robustness to setups that are unseen but still within the distribution of cousins that the policy was trained on. Moreover, policies trained on all assets exhibit consistent but low performance, again highlighting the improvement of guided domain randomization via digital cousins.
improve policy
Digital cousins provide a proxy for out-of-distributtin performance (Q3). We additionally observe that digital twin policy performance generally degrades proportionally as the DINOv2 embedding distance increases across evaluation setups. This suggests that digital cousins may serve as proxy for out-of-distribution performance, with "further away" setups capturing setups that are proportionally further away from the data distribution seen in the original setup.
3.3 Sim-to-Real Policy Learning with Digital Cousins
Ultimately, we want our pipeline to accelerate sim-to-real policy transfer, where digital cousins may cover a conditioned but wider distribution to mitigate the sim-to-real gap. To evaluate our approach, use real-world IKEA cabinet and its corresponding digital twin model, train both digital cousin policy using cousins matched from ACDC and multiple digital twin policy baselines using the virtual asset, and then evaluate zero-shot on the real cabinet. Our results are shown in Fig. 5.
Digital cousins can enable zero-shot sim-to-real policy transfer (Q4). We find that while both the digital twin and digital cousin policies perform well in simulation, only the digital cousin policy able to transfer to the real world. We hypothesize that because digital cousins provide a wider dis- tribution of training data, the resulting sim policy is better able overcome the tim-to-real domain gap resulting from asset modeling and sensor perception errors. Moreover, find that naive domain randomization alone is insufficient to overcome the sim-to-real domain gap, and that leveraging digital cousins can better overcome this gap and reduce the need for exact twin reconstruction.
3.4 Real-to-Sim-to-Rell Scene Generation and Policy Learning
Finally, we test our full pipeline and automated policy learning framework end-to-end with a fully in-the-wild kitchen scene, We find that our policy can successfully open the kitchen cabinet after being trained exclusively in simulation on digital cousins, as seen in Fig. 1, Experiment videos and additional results can be found at https::///// /ttp//diggaaallaaa cousins gi githuub... io/.
safely answer Q2, Q3, and Q4: Policies trained using digital cousins exhibit comparable in-distribution and more robust out-of-distributis performance compared to policies trained on digital twins, and can enable zero-shot sim-to-real policy transfer.
Summary. Based on these results,
7

---

Related Work
Real-to-Sim Scene Creation for Robotics Creating realistic and diverse digital assets and scenes from real-world inputs is a prevalent and long-standing problem [16-19]. Within robot learning, real-to-sim scene creation has been achieved through manual curation [4, 10, 20- 25], procedural generation [26-28], few-shot interactions [8, 29, 30], inverse graphics [31], and more recently foundation model-assisted generation [32, 33). However, these methods either cannot handle scene-level generation, require human labor, or cannot retain physical plausibility. In contrast, ACDC is fully auto- mated and the recovered digital cousins are faithful to the input physical scenes.
Policy
Sim Success
Real Success
Twin Twin + DR
100%
25%
70%
55%
Twin + Cousin
92%
95%
Cousin
94%
90%
Figure Zero-shot real-world evaluation digi- tal cousin policy vs. digital twin baselines. Task is Door Opening an IKEA cabinet. Mctric is success rate: sim/real results averaged over 50/20 trials. Twin DR is trained using increased domain (pose, scale) randomization, and Twin + Cousin trained on both twin and cousin data.
Policy Learning with Synthetic Data Data synthesis for robot learning can alleviate the burden of collecting data in the real world with physical robots [34-6]]. To synthesize complete robotic trajectories (sequences of observation- action pairs), researchers develop action primitives operating privileged information available in simulation [37-40], leverage task and motion planning (TAMP) [41] to generate robot mo- tions [31, 42, 43], train and distill RL policies [44-46], and automate data generation given an initial set of human demonstrations [33, 47, 48). In this vein, our work also leverages primitive skills for efficient and robust data collection. However, unlike previous methods, which use generative models to synthesize data [49, 50], our reconstructed scenes are physically plausible, which eases policy learning and better facilitates transfer real hardware.
Sim-to-Re Real Policy Transfer Seamlessly deploying robot policies learned in the simulation to the real world is critical. Successful sim-to-real transfer has been demonstrated on dexterous in-hand manipulation [45, 46, 51-53), robotic-arm manipulation [54-64), quadruped locomotion [65-68], biped locomotion [69-74], and quadrotor flight [75, 76]. Methods to bridge sim-to-real gaps mainly include domain randomization [51, 77-791, system identification [60, 65, 80, 81] and simulator augmentation [82-84]. Notably, recent work demonstrates robust real-world deployment of manipulation policies by training on diverse simulated scenes [10, 31]. Our work expands the simulation training coverage and hence further robustifies policies by training "digital
cousins" wider distribution than the nearest-asset training scenario.
5 Conclusion
Digital cousins can be quickly generated by a fully automated pipeline, called ACDC, from a single real-world RGB image. We find that policies trained on these digital cousins are more robust than those trained on digital twins, with comparable in in-domain performance and superior out-of-domain generalization, and enable zero-shot sim-to-real policy transfer.
Limitations. Our system has few limitations. First, ACDC is bounded by the diversity of its underlying asset dataset. While BEHAVIOR- IK contains thousands of unique assets, we find that it is still insufficient to densely capture the real-world distribution of objects. Second, because ACDC is built upon multiple large pretrained models, our pipeline inherits the limitations of these models, including adversarial and out-of-distributiio scenes. Third, policy learning with digital cousins can still be significantly improved and can benefit from recent advancements in robot learning, such as diffusion policies [85].

---

Acknowledgments
We are grateful the SVL PAIR group for helpful feedback and insightful discussions. This work is in part supported by the Stanford Institute for Human-Centered AI (HAI), ONR MURI N00014- 22-1-2740, ONR YIP 100014-24-1-2117, ONR MURI N00014-21-1-280) and Schmidt Sciences. Ruohan Zhang is partially supported by the Wu Tsai Human Performance Alliance Fellowship.
References
[1] K. Bousmalis, A. Irpan. P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine, and V. Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. arXiv preprint arXiv: Arxiv-1709.07880 2017.
[2] D. Ho, K. Rao, 7. Xu, E. Jang, M. Khansari, and Bai. Retinagan: An object-aware approach to sim-to-real transfer. arXiv preprint arXiv: Arxiv-201 1.00 2020.
[3] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots, 2021.
[4] C. Li, R. Zhang. J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, Y. Kim, A. Lou, C. R. Matthews, Villa-Renteria, J. H. Tang, C. Tang. F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and Fei-Fei. Behavior- 1k: benchmark for embodied with 1,000 everyday activities and realistic simulation. K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 80-93. PMLR, 14-18 Dec 2023. URL https:/ proceedings mwm...r...... wwwpppresssssseee v205//225//////// 1123a.1...... wt..ii2..ttm...
[5] X. Puig. E. Undersander, A. Szot, M. D. Cote, T.-Y. Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y. Min, V. Vondruš, T. Gervet, V.-P. Berges, J. M. Turner, O. Maksymets, 7. Kira, M. Kalakrishnan, J. Malik, D. S. Chaplot, U. Jain, D. Batra, A. Rai, and R. Mottaghi. Habitat
3.0: co-habitat for humans, avatars and robots, 2023.
[6] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
[7] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi. ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. In NeurIPS, 2022. Outstanding Paper Award.
[8] C.-C. Hsu, Z. Jiang, and Y. Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. arXiv preprint arXiv: 232.0-230..01195, 2023.
[9] Z. Zhang, L. Zhang, Z. Wang, Z. Jiao, M. Han, Zhu, S.-C. Zhu, and Liu. Part-level scene reconstruction affords robot interaction, 2023.
[10] M. Torne, A. Simeonov, 7. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal. Reconciling reality through simulation: A real-to-sim-to-rees approach for robust manipulation. arXiv preprint 2024.
arXiv:
[11] M. Oquab, Т. Darcet, Moutakanni, H. V. Vo, M. Szafraniec. Khalidov, P. Fernandez, D. Haziza, F. Massa, El-Nouby, Howes, P.-Y. Huang, H. Xu, Sharma, S.W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.

---

[12] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Bal- tescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Fcoffet, A. Fleti, T. Hloun- dou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lpps,, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. Gu, Y. Guo, C. Hallacy, Han, Harris, He, M. Heaton, Hei- decke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, Jomoto, B. Jonn, H. Jun, T. Kaftan, Lukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kil- patrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, Lue, A. Makanju, K. Malfacini, Manning, T. Markov, Y. Markovski, B. Mar- tin, K. Mayer, Mayne, McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. Mc- Neil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O'Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, H. Pong. T. Powell, A. Power, B. Power, E. Prochl, R. Puri, A. Rad- ford, Rae, Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli. T. Sanders. S. Santurkar, G. Sastry, Schmidt, D. Schnurt, J. Schul- man, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, F. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, Welinder, Weng,
K. Button, T. Cai, Campbell, A. Cann, B. Carey,
Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. Gpt-4 technical report, 2024.
[13] T. Ren, S. Liu, A. Zeng. J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F, Yan, Z. Zeng, H. Zhang, F. Li, J. Yang. H. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024.
[14] L. Yang, B. Kang, Z. Huang, 7. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2, 2024. 244660000 09414.
URL https:://aatt http::/wwwwwwaaaaa.aa/aoae
[15] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[16] Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox. Rgb-d mapping: Using kinect-style depth cameras for dense 3d modeling of indoor environments. The International Journal of Robotics Research, 31(5):647-663, 2012. 10::10011770007733449143333 1444444911444488 //doi.org////
URL https:
[17] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes neural radiance fields for view synthesis. arXiv preprint arXiv: Arxiv- 0893,, 2020.
10

---

[18] M. Tancik, E. Weber, E. Ng, Li, B. Yi, KeTr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv: Arxiv- 2xxv-23302.04264, 2023.
[19] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. arXiv preprint arXiv: Arxiv-2308.04400 2023.
[20] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: Arxiv-1709.06158 2017.
[21] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: Arxiv-1712.05474, 2017.
[22] Xia, A. Zamir, Z.-Y. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real-world percep- tion for embodied agents. arXiv preprint arXiv: Arxiv-1808. 1065-4440 2018.
[23] Xia, W. B. Shen, C. Li, P. Kasimbeg, M. Tchapmi, A. Toshev, L. Fei-Fei, R. Martín- and Savarese. Interactive gibson benehmark (igibson 0.5): A benchmark for interactive navigation in cluttered environments. arXiv preprint arXiv: Arxiv-1910 14442, 2019.
[24] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, Malik, Parikh, and Batra. Habitat: platform for embodied ai research. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[25] A. Szot, A. Clegg, H. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondruš, S. Dharur, Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and Ba- Habitat 2.0: Training home assistants to rearrange their habitat. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 251-266. Curran Associates, Inc., 02 htttps://proceeddingg http://oocceciiin....ieeeeeeeeee ce cc/paper.p.p.......peee ppaper_filesss_ wwwef/ii//pppe patls///pppr//21// pdde pdf.
2021. URL https:
[26] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi. Procthor: Large-scale embodied ai using procedu- ral generation. arXiv preprini arXiv: Arxiv-2206.006 06994, 2022.
[27] A. Raistrick, L. Lipson, Z. Ma, L. Mei, M. Wang, Y. Zuo, K. Kayan, H. Wen, B. Han, Y. Wang, A. Newell, H. Law, A. Goyal, K. Yang, and J. Deng. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12630-12641, June 2023.
[28] Y. Yang. F.-Y. Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, J. Wu, N. Haber, R. Krishna, L. Liu, C. Callison-Burch M. Yatskar, A. Kembhavi, and C. Clark. Holodeck: Language guided generation of 3d embodied ai environments. arXiv preprint arXiv: Arxiv-2312.09067, 2023.
[29] 7. Jiang, C.-C. Hsu, and Y. Zhu. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5616-5626, June 2022.
[30] N. Nie, S. Y. Gadre, K. Ehsani, and S. Song. Structure from action: Learning interactions for articulated object 3d structure discovery, 2022.
[31] Z. Chen, A. Walsman, M. Memmel, Mo. A. Fang, K. Vemuri, A. Wu, D. Fox, and Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint arXiv: 165. 2024.

---

[32] Y. Wang, Z. Xian, F. Chen, T.-H. Wang. Y. Wang, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv: Arxiv-2311.01455 2023.
[33] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024.
[34] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, Mordatch, O. Nachum, C. Parada, J. Peralta, Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv: Arxiv-2212.0663 2022.
[35] O. X.-E. Collaboration. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv: Arxiv-2310.00 08864, 2023.
[36] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mer- cat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, Xiao, H. Yang, A. Yavary, Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, Hu, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O' D'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Ja- yaraman, J. J. Lim, J. Malik, R. Martin, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv: Arxiv-2403. 12945, 2024.
[37] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, A. Wahid, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv: Arxiv-2010. 2020.
[38] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation, arXiv preprint arXiv: Arxiv-2109, /2098, 2021.
[39] Y. Jiang, A. Gupta, Z. Zhang, G. Wang. Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210.03094 2022.
[40] M. Heo, Y. Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world bench- mark for long-horizon complex manipulation. In K. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi:10. 1r..10..5607RSS.200000 XIX.041. URL
RSS 2023 XIX. 041.
[41] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. arXiv preprint arXiv: Arxiv-2010.01083 2020.
[42] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint Xiv: Arxiv-2305.16339, 2023.
12

---

[43] H. Ha, P. Florence, and S. Song. Scaling up and distilling down: acquisition. arXiv preprint arXiv: Arxiv-2307. 2.055.357..45 2023.
robot skill
[44] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv: Arxiv-2111.11744, 2022.
[45] Y. Chen, C. Wang. L. Fei-Fei, and C. K. Liu. Sequential dexterity: Chaining dexterous policies for oin-horizoon manipulation. arXiv preprint arXiv: Arxiv-2309.09987, 2023.
[46] H. Qi, Yi, Suresh, M. Lambeta, Ma, R. Calandra, and Malik. General in in-hand object rotation with vision and touch. arXiv preprint arXiv: Arxiv-2309 09979, 2023.
[47] A. Mandlekar, S. Nasiriany, Wen, Akinola. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv: Arxiv-2310.17500 2023.
[48] R. Hoque, A. Mandlekar, C. Garrett, K. Goldberg, and D. Fox. Intervengen: Interventional data generation for robust and data-efficient robot imitation learning. arXiv preprinl arXiv: Soxiv-2405.01472 2024.
[49] Z. Chen, S. Kiami, A. Gupta, and V. Kumar. Genaug: Retargeting behaviors to unseen situa- tions via generative augmentation. arXiv preprint arXiv: Arxiv- Sxxi--2302.0661,, 2023.
[50] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, Singh, C. Tan, D. M, J. Per- alta, B. Ichter, K. Hausman, and Xia. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv: Arxiv-2302. 11550, 2023.
[51] OpenAl, I. Akkaya, M. Andrychowicz. M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with robot hand. arXiv preprint arXiv: 110.0119 2019.
[52] T. Chen, M. Tippur; S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In- hand reorientation of novel and complex object shapes. Science Robotics, 8(84)caad9244, 2023. doi: 1tg.10..116/scinootiiss...e 10. 10.1125/scirobticcsss adc9244
URL https:tt/////cc.t. wwt.://.eene.org.o.oooooooooooo. /
[53] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. arXiv preprint arXiv: Arxiv-2210.04887 2022.
[54] Y. Chebotar, Handa, V. Makoviychuk, M. Macklin, Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv: 11x10-1810.05687, 2018.
[55] S. Kozlovsky, E. Newman, and M. Zacksenhouse. Reinforcement learning of impedance poli- cies for peg-in-hole tasks: Role of asymmetric matrices. IEEE Robotics and Automation Let- ters. (14):10898-10900) 2022. doi: 100.1019/AAA.222..3000
[56] D. Son, H. Yang. and D. Lee. Sim-to-real transfer of bolting tasks with tight tolerance. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9056-9063, 2020. doi: 10::10.11 100.10000000444442200000.4
[57] B. Tang, M. A. Lin, I. Akinola, A. Handa, G. S. Sukhatme, F. Ramos, D. Fox, and Y. S. Narang. Industreal: Transferring contact-rich assembly tasks from simulation reality. In K. E. Bekris, Hauser, S. L. Herbert, and Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic Korea, July 10-14, 2023, 2023.
URL
https: / doi wrg.100.g...... 15607 RSS. 2023 XIX. 039.
13

---

[58] X. Zhang, C. Wang, L. Sun, Z. Wu, X. Zhu, and M. Tomizuka. Efficient sim-to-real transfer of contact-rich manipulation skills with online admittance residual learning. In 7th Annual Conference on Robot Learning, 2023. URL https: https:://ppnrevveww net forum? r gFXVysXh48K.
[59] X. Zhang, M. Tomizuka, and H. Li. Bridging the sim-to-real gap with dynamic compliance tuning for industrial insertion. arXiv preprint arXiv: Arxiv-2311.07499, 2023.
[60] V. Lim, H. Huang, L. Y. Chen, J. Wang. J. Ichnowski, D. Seita, M. Laskey, and K. Goldberg. Planar robot casting with real2sim2real self suff-uueeeeieed learning. arXiv preprint arXiv: Arxiv- 2/11.04814 2021.
[61] W. Zhou and D. Held. Learning to grasp the ungraspable with emergent extrinsic dexterity. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 150-160. PMLR, 2022. URL https: htt.tt://p/oceeeeeeeeeeee /ww.eeddoooeeeeoeooooooooooocccc ngs .ml www.mmlr.pre.... p.rsss//// v205 www.wh....a...oooooooo html.
[62] M. Kim, J. Han, J. Kim, and B. Kim. Pre- and post-contact policy decomposition for non- prehensile manipulation with zero-shot sim-to-real transfer. arXiv preprint arXiv: Arxiv- 2309.02754, 2023.
[63] Y. Jiang, C. Wang, R. Zhang. J. Wu, and L.. Fei-Fei. Transic: Sim-to-real policy transfer by learning from online correction. arXiv preprint arXiv: Arxiv-2405. 10315, 2024.
[64] X. Zhang, S. Jain, B. Huang, M. Tomizuka, and D. Romeres. Learning generalizable pivoting skills. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 June 2, 2023, pages 5865-5871. IEEE, 2023. 10161271. URL https: /woi.doiio//// org/ / 10. 1101/11999/1 RA..00/ICRAAB8888 2023 10161271.
[65] Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: Arxiv- 1804. 10332, 2018.
[66] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: rapid motor adaptation for legged robots. In D. A. Shell, M. Toussaint, and M. A. Hsieh, editors. Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021.
202.:SSS...22..I...11111 URL https:
//doi. org/ 10. 15607 /RSS.RSS// 2021 XVII. 011.
[67] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprint arXiv: Arxiv-2309.05665,, 2023.
[68] R. Yang, Yang, and X. Wang. Neural volumetric memory for visual locomotion control. arXiv preprint arXiv: Arxiv-2304.01201, 2023.
[69] H. Benbrahim and J. A. Franklin. Biped dynamic walking using reinforcement learning. Robotics and Autonomous Systems, 22(3):283-302 1997. ISSN 0921-8890. doi:htips://
URL https::t//// /www. www.ncedireeeeeeeeeeeeeeee com/ sccianne/arrtcaea/aa///222288819
Robot Learning: The New Wave.
[70] G. A. Castillo, B. Weng, W. Zhang, and A. Hereid. Reinforcement Boriing-based cascade motion policy design for robust 3d bipedal locomotion. IEEE Access, 10:20135-20144 2022. doi: 000::001100//ACCSSS0222.3333331
[71] L. Krishna, G. A. Castillo, U. A. Mishra, A. Hereid, and Kolathaya. Linear policies are suf- ficient realize robust bipedal walking challenging terrains. arXiv preprint arXiv: Arxiv- 12665, 2021.
14

---

[72] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. arXiv preprint arKiv: Saxiv-2105.08322 2021.
[73] I. Radosavovic, T. Xiao, B. Zhang. T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. arXiv preprint arXiv: Arxiv-2303.0333 030300331,, 2023.
[74] Z. Li, X. B. Peng, Abbeel, S. Levine, G. Berseth, and K. Sreenath, Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. arXiv preprint arXiv: Arxiv- 16889, 2024.
[75] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 2023. 1oi:10.1038/ 023-06413 s4 941586-023 0et---233----44 URL https::tp///.. / /oii www.orr.//.............o 10w.//108///11 00044155660022011-910 9-4.
[76] Y. Song, A. Romero, M. Müller, V. Koltun, and D. Scaramuzza. Reaching the limit in au- tonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8(82): eadg1462, 2023. doi:10. 112//ccrobboticssagg1144 doi/abs
URL https: //wps://wwww../ www.nce.iiccc..ccoooo........ org/ 121/12//ii roboti cs adg1 462.
[77] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. arXiv preprint arXiv: Arxiv-/710.0653/ 2017.
[78] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. V. Wyk, A. Zhurkevich, B. Sundaralingam, and Y. S. Narang. Dextreme: Transfer of agile in- hand manipulation from simulation to reality. In IEEE International Conference on Roboties and Automation, ICRA 2023, London, UK, May 29 -June 2, 2023, pages 5977-5984. IEEE, 2023. doi: 109//C 10910ICAAA8999.222200000000 ICRA4 8 91 2023 101 60216.
URL https: / /ttp//ii.oogo.1111///1
[79] J. Wang, Y. Qin, K. Kuang, Y. Korkmaz, A. Gurumoorthy, H. Su, and X. Wang. Cyberdemo: Augmenting simulated human demonstration for real-world dexterous manipulation. arXiv preprint arXiv: Arxiv-2402. 14795, 2024.
[80] L. Ljung. System Identification, pages 163- 173. Birkhäuser Boston, Boston, MA, 1998. ISBN 1-111/997788-1-1--766-- URL https:://dee doii org/10.
978-1-4612-116688.
1007/978-1-461-
[81] Chang and Padir. Sim2real2sim: Bridging the gap between simulation and real-world in flexible object manipulation. arXiv preprint arXiv: Arxiv-2002.02533.. 2020.
[82] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, Issac, N. D. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pages
IEEE, 2019. 111:10..119/IRRA...... 111.:///AAA..219899999838. URL https: doi org/ 10 1109 ICRA 2019. 87937897777.
[83] P. Hanna and Stone. Grounded action transformation for robot learning simulation. In Proceedings of the Thirty-Firss AAAI Conference on Artificial Intelligence, AAAI' 17, page 4931 19322 AAAI Press, 2017.
[84] E. Heiden, D. Millard, E. Coumans, and G. S. Sukhatme. Augmenting differentiable simulators with neural networks to close the sim2real gap. arXiv preprint arXiv: saziv22007.060444 2020.
[85] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2023.
[86] W. Guan, W. Li, and Y. Ren. Point cloud registration based improved iep algorithm. In 2018 Chinese Control And Decision Conference (CCDC), pages 1461-1465, 2018. d0i:10.
1109/CCDC.208..880 840735..
15

---

[87] P. Li. R. Wang, Y. Wang, and W. Tao. Evaluation of the icp algorithm in 3d point cloud registration. IEEE Access, 8:68030-68048, 2020. 10: 2020
[88] B. Wen, W. Yang, J. Kautz, and S. Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects, 2024.
[89] B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. V. Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, Ramos, N. Ratiff, and Fox. curobo: Parallelized collision-free minimum-jerk robot motion generation, 2023.
[90] M. Gualtieri, A. ten Pas, K. Saenko, and R. Platt. High precision grasp pose detection in dense clutter, 2016.
[91] H. Liang. Python binding for grasp pose generator (pygpg), Aug. 2021. URL https:// // doi. coi.org/10 wtt.oorg......zeennne...
5247189.
[92] Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Trans- actions on Big Data, 7(3):535-547, 2019.
[93] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. kdd, volume 96, pages 226-231, 1996.
[94] L. Jin, J. Zhang, Y. Hold-Geoffroy, O. Wang. K. Matzen, M. Sticha, and D. F. Fouhey, Per- spective fields for single image camera calibration, 2023.
[95] L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. arXiv:2406.09411 2024.
[96] H. Cheng and A. G. Schwing. XMem: Long-term video object segmentation with an atkinson-shiffe memory model. ECCV, 2022.
[97] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv: Arxiv-1612.00533 2016.
[98] I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019.
16

---

Appendix
A Additional Cousin Creation Details
A.1 Offline Dataset Generation
Cousins creation requires a large-scale asset set. We adopt BEHAVIOR-IK [4], which includes over 10,000 object assets. The goal of this stage is to preprocess the whole asset set for later usage. Since objects may have occlusion in the input image, common approaches that can estimate the scale and orientation of real objects such as point cloud registration [86, 87] and monocular pose estimation methods [88] are not feasible because these generally require two complete, unobstructed point clouds for given object. Instead, we choose to represent each asset set of visual 2D images, under the expectation that we will use visual encoder (such DINOv2) downstream to match geometric correspondences between objects. For our given dataset, we rotate each asset a; in the whole asset set and take snapshots from a fixed camera pose Psim: resulting in a set of images {i:. ) and representative snapshot I, Each asset is pre-annotated with its own semantically meannnngl category t,. This results asset tuples (a, = (t,, I, { N=r")) ) where N Nassets is the total number of assets included in the BEHAVIOR- 1K dataset, Note that this stage occurs once offiin,, and can be cached when running ACDC.
A.2 Mounting Type
We observe that scene objects often serve different semantic roles and fall under difference pose dis- tributions depending on whether an object is fixed with respect to the room. Therefore, as mentioned in Section 2.1, we leverage this inductive bias and prompt GPT to determine if an object is mounted on a wall or not. This distinction helps address a key limitation with our one-shot approach: because of heavy occlusion resulting from a single camera view, objects such as televisions or cabinets that are mounted to walls may only have its frontal face observed from a single camera view, resulting in a insufficient extracted point cloud that does not fully capture its underlying volumetric depth. television cabinet fixed on wall, frontal view image may only cover the frontal face of the mounted object.
We mitigate this limitation by prompting GPT to classify each object into one of three semantic categories: (1) Wall Mounted: An object is fixed on a wall with nothing closely beneath it; (2) On Floor or On Another Object: An object is placed on the floor, or on another object, but the object does not touch wall; (3) Mixture: An object is not mounted
wall, but one of its face touches wall, like bookshelf putting the floor but touches the wall behind it, or microwave oven putting on a cabinet but its back face touches the wall behind it. In cases (1) and (3), we also require GPT to specify the specific wall on which the object is mounted by feeding all masked walls in the input image generated by Grounded-S -AM.e-2...-22.. In practice, we first prompt GPT to identify whether an object is installed or fixed on one or more walls and to specify which wall(s) is attached to. the object is mounted on wall, is classified as (1) Wall Mounted. For objects not installed on any wall, we prompt GPT to determine if the object is aligned with and in contact with one more walls. This step further classifies the object into either mounting type (2) (3). Users have the option to disable the second prompt, thereby distinguishing only whether an object is wall-mounted or not. Please see Appendix A.3 for how objects with different mounting types are processed.
A.3 Generated Scene Post- Post-Processing
After putting all assets in the correct position in the Simulated scene generation stage described in Section 2.1, we post-process each asset for a physically plausible scene. For each asset i, we should have its bounding center position peen bounding box's top-right vertex position p!'R, and bounding box's bottom-left vertex position L. First, we sort all assets from low to high by sorting p. pen in ascending order, and project each asset's 3D bounding box to the x-y plane, resulting in 2D polygon poly; for each asset i. We then infer "on top" relationships from sorted asset list.
17

---

For each asset i, we search over all assets with lower P; to determine another asset j right beneath it. Whenever the overlapped area between the lower asset j's projected 21D polygon poly, and the current asset's 's projected 2D polygon poly; exceeds 70% of the area of either one of the 2D polygons, i.e., area (iee((inersecc(ppl (polye, poly;)) > 0.7 minl (iin(are((pop))) area(poly;) )), we determine that the higher asset i on top of the lower asset j. and the lower asset j is beneath the higher asset Intuitively, this checks for vertical spatial alignment between two objects. If matching asset is found, the asset is regarded as being on top of the floor. After all assets have been evaluated in this way, each asset should have another asset or floor beneath it after performing the above searching.
Next, we post-process all assets based on their mounting type: For an asset with mounting type (1) (Wall Mounted), we first adjust its scale and orientation, and then adjust its position. Since asset i is mounted on wall, we determine the face of asset i that should be adjusted such that it becomes parallel to the wall. First, fit plane the wall from its corresponding extracted point cloud. We then compute the minimum rotation that aligns either the object's local x or y axis with the normal vector of the wall plane. Finally, we compute the distance between p and the wall, and rescale and translate asset in the x-y plane such that the object's rear face is co-planar with the wall plane and object's front face maintains its same position. Finally, we de- penetrate this object from others by adjusting pf z value: We increase p.e by =(p) =(p, if =pp > =(pf")) where z()) if the z coordinate of 3D vector, and j is the index of the asset beneath asset i, and then fix asset i on the wall that GPT selected for asset i. When Rp) I (.., we directly fix asset i on the wall without adjusting its position.
For an asset with mounting type (2) (On Floor or On Another Object), we similarly de-penetrate by placing asset i on top of asset j by adjusting p by (=(p"") =p; )1 For an asset i with mounting type (3) (Mixture), we adjust the orientation and scale the same way as assets with mounting type (1); and then adjust p. in the same way assets with mounting type (2).
Finally, we check for collisions between the collision meshes of each pair of placed assets and adjust their positions in the x-y plane to avoid any overlap.
A.4 Skill Definition
In order to bootstrap automated demonstration collection, we define a library of analytical and sampling-basee skills that can be chained together to solve long-horizon tasks, such as the Putting Away Bowl task. For collision-free motion planning, we leverage CuRobo [89]. For sampling- grasp generation, we leverage Grasp Pose Generator (GPG) [90] [91] based on given object's sam- pled point cloud from its analytical mesh. Below, we briefly describe the high-level implementation of each skill:
Open. This skill consists of five steps: Approach, which computes collision-free trajectory towards point offset in front of the desired handle to articulate, Converge, which computes an open-loop straight-line trajectory to the actual grasping point on the handle, Grasp, which closes the gripper to grasp the handle, Articulate, which computes an open-loop analytical trajectory to articulate the link, and Ungrasp, which opens the gripper to release the handle.
For given articulated object, we leverage ground-truth knowledge of its geometric affordances to compute a corresponding trajectory. Given specific articulated asset and desired link to articulate 1, we first infer the link's 's corresponding handle location by shooting rays towards the link and define the mean handle location as mean location over the rays with the shortest distance. This assumes that the most protruding geometric feature corresponds to the handle. Given handle location, we inspect I's parent link j's properties, determining its type (prismatic or revolute) and pose with respect to the handle. Given this information, we can compute a desired analytical trajectory for the handle to open link 1. This can easily be transformed into the robot frame, and offset according to the robot's end-effectoo size.
18

---

Close. This implementation is nearly identical to Open, though for computing the desired articulation trajectory, the start end points are reversed.
This skill consists of three steps: Move, which computes collision- free trajectory towards a sampled grasping point, Grasp, which closes the gripper to grasp the object, and Lift, which computes an open-loop trajectory to lift the object slightly.
Pick.
Note that during the Move phase, sample grasping points that are both feasible, collision-free, and minimize robot gripper orientation changes to avoid bad robot configurations.
Place. This skill consists of three steps: Move, which computes collision- free trajectory towards sampled placement pose, Ungrasp, which opens the gripper to release the object, and Lift, which computes an open-loop trajectory to lift the gripper slightly.
This skill assumes that an object is already grasped prior to its execution. We assume the desired placement pose is kinematic predicate relative to another scene object, e-g.: insidd (cabinet). rejection sampling sample collision-free poses for the robot's end-
Given this predicate, effector and grasped object that satisfy the given predicate, prioritizing poses that minimize end- effector rotation.
A.5 Demonstration Collection
We use fully automated demonstrations using our programmatic skills defined above. For the Door Opening and Drawer Opening tasks, this simply consists of executing the Open skill. For the Putting Away Bowl task, this consists of Open, Pick, Place, Close sequence. We use rejection sampling so that our resulting dataset only includes successes, that is, if any skill execution fails midway, we do not save that episode. This allows us to significantly increase the randomization range between episodes without being limited by poor edge cases.
Across all tasks, we randomize the agent's pose as well as scene objects' poses and scales between episodes.
A.6 Using DINOv2 for Digital Cousin Matching
For a given input image x and set of candidate matching images {i,} we define the top-1 matched candidate through a DINOv2-based voting system. First, we pass both input image x and all candidate images {i, ( through DINOv2, retrieving their feature patches e and {f, 1 re- spectively. Next, we compute the nearest neighbor (defined as the L.2-norm) the DINOv2 feature embedding space for each pixel in over all pixels across all candidate feature embeddings { ) and record the running count of nearest neighbors across all candidates j E {1.....N], I matched candidate is then the candidate with the highest count of per- per-pixel nearest neighbors ie.: the candidate image i, that has the highest number of closest visual feature correspondences to input image x. For top-k matched candidates, we repeat the process iteratively, selecting the top- each time and subsequently removing the selected i, during proceeding iterations. We leverage GPU-acceleratee nearest neighbor computations using the open-source faiss [92] package.
N}. The top-
Given a matched pair of images x, iz, define the DINOy2 embedding distance as the average nearest neighbor L.2-distance between each pixel in corresponding input feature map and all pixels in corresponding matched feature map fy. Note that we exclude the largest 10% of nearest neighbor distances in this calculation, as we find empirically that the sorted results across matched candidates are more salient with these outliers removed.
A.7 Additional Real-to-Sim Details
In this subsection, provide additional implementation details of ACDC real-to-sim pipeline:
19

---

Depth image and point cloud processing. One key design decision is to use synthetic depth via Depth-Anything-v2 [14), instead dedicated depth camera. This decision is guided by our observation that it performs more consistently on reflective surfaces. However, this synthetic depth approach still generates artifacts occurring near object boundaries, the image periphery, and under lighting changes. To further remove noise in object point clouds, we apply DBSCAN clustering [93] on each object point cloud p; filter out noisy points.
Orientation Refinement. DINO performs rough estimation of asset orientations, which for most objects the orientation is sufficiently accurate. However, we additionally provide an option to further refine the orientation refinement based on an object's extracted point cloud. By computing the z-aligned minimum bounding box of the given point cloud, we can apply an additional z-rotation to DINO's outputted estimated orientation so that the matched asset's canonical xy-axes aligns with the computed minimum bounding box frame. We find this is especially useful for object's that have sharp geometric boundaries, such furniture objects.
Heuristics for articulated objects. In this project, articulated objects refer to those with doors (revolute) and drawers (prismatic). To ensure the selected digital cousins of an articulated object are also articulated, so that door opening or drawer opening demos can be collected on all digital cousins, we propose to search digital cousins for articulated objects only among articulated assets. Because we have ground -tuut information for all of our dataset assets, we know apriori which assets are articulated. During the Real-world extraction stage, additionally prompt GPT to determine whether objects are articulated.
An optional heuristics is to apply a door/drawer count threshold on digital cousin creation of artic- ulated objects. During the Offline Dataset Generation stage, we can count the number of doors (revolute joints) and drawers (prismatic joints). When creating cousins, we only search among as- sets with "similar" number of drawers and doors. This threshold is open to users set. In all of our real-to-sim results, set the threshold to 2 in the nearest cousin selection too guarantee affordance preservation, but do not apply this heuristic to the rest of the scenes.
GPT API Usage.
We use GPT-4o for the real-to-sim pipeline.
Inference Time. While ACDC 's overall wall-clock time varies as a function scene complexity, in general, we empirically observe the following:
Step 1. World Extraction] takes around 7 seconds per object.
Step 2. [Digital Cousin Matching] takes around 20 seconds to select one digital cousin for object.
Step 3. [Simulated Scene Generation] takes less than 30 seconds for whole scene.
B Additional Experimental Details
B.1 Visual Encoder Ablation Study
In this subsection, we extend Section 3.1 of our main paper by conducting an ablation study on the real-to-sim pipeline sim-to-sim setting. We seek evaluate whether DINO is sufficient for digital cousin matching, if applying GPT finetune DINO's selections can result in improved performance. Our quantitative and qualitative results cover the following comparisons: (a) DINO Model Selection & GPT Orientation Selection; (b) DINO Model Selection & DINO Orientation Selection; (c) GPT Model Selection & GPT Orientation Selection; (d) GPT Model Selection & DINO Orientation Selection.
DINO Model Selection involves selecting an asset A the best digital cousin of an object based solely on the DINOv2 embedding distances between the masked object RGB x, and all assets' representative model snapshots I, within the nearest heat categories. While DINO Model Selection
20

---

Imput Scene
Cousin Scene
Scale (m)
Cat.
Mod.
L2 Dist (cm)
Ori. DUE.
Bbox IoU
Oni. Bbox IoU 1
5.27 2.85
0.07
0.75
0.07
(b)
515
5/5
5.27 + 2.85
0.07 + 0.07
0.75 4 0.14
0.64 0.07
3,68
5.27 2.85
0.07 0.07
0.75 + 0.14
0.64
5/5
5/5
5,27 = 2.85
0.07 = 0.07
0.75 = 0.14
0.64 + 0.07
4.79 1.52
0.07 0,03
0.54 a 0.32
0.52 a: 0.28
6/6
4/6
0.54 0.32
3.42
(6)
4.15 + 2.04
0.10 0.14
0.64 + 0.23
0.73 + 0.22
=
0.54
(8)
5,51 +: 2.71
0.03 + 0.00
0.64 a: 0.16
0.60 + 0.17
5,97
0.03 0.00
0.64
0.60
2.91
(c)
7.13 4.77
0.19 0.16 0.19
0.54 a: 0.24
0.51 + 0.21
(d)
5.60 a 2.92
0.10 0.10
0.65 0.18
0.60 0.17
(a)
6.47 =: 2.79
0.94 = 0.01
0.64 0.23
0.65 + 0.28
(b)
2/5
6.83 a 3.20
0,03 0.01
0.63 0.24
0.64 0.30
3.54
(c)
5/5
3/5
6.51 = 2.77
0.03 +: 0.01
0.64 0.22
0.60 +: 0.20
(d)
6,90 : 3.21
0.03 + 0.01
0.62 + 0.24
0.58 + 0.22
(8)
516
3/6
6.64 + 3.34
0.24 0.20
0.57 0.21
0.58
(b)
5.92 : 3.32
0.06 = 0.03
0.69 + 0.14
0.66 + 0.14
3.24
6.00 3.79
0.06 0:05
0.68 0.14
0.65
(d)
516
5/6
5.33 # 3.77
0.04 + 0.03
0.69 # 0.15
0.67 + 0.15
7,81 4.87
0.05 0.00
0.65 0.18
0.64 0.17
(h)
8/8
3/8
7.91 + 5.04
0.05 + 0.00
0.65 0.19
0.64 + 0. 16
4.17
(c)
7.65 + 5.62
0.05 1: 0.00
0.66 : 0.21
0.74 0.16
(d)
8/8
8/8
7.94 + 5.01
0.05 0.00
0.66 0.17
0.63 0.17
(e)
7.27 + 5.51
0.46 0.50
0.54 + 0.30
0.55 + 0.25
10/10
6/10
4.78
6.89
(E)
10/10
10/10
5.96 4.335
4.77 + 3.38
0.05 0.04
0.03 0.01
0,70 0.18
0.74 0.20
0.63 0.18
0.77 + 0.19
(e)
15/15
12/15
17.40 9.005
17.01 9.3.4
0.15 0.14
0.17 =: 0.17
0.49 a: 0.20
0.51 + 0.20
0.50 0.19
0.50 0.20
(c) 15/15
15/15
15.67 + 8.86
0.12 0.11
0.59 + 0.14
0.72 0.14
(d)
17.09 9.26
0.11 0.10
0.55 a 0.21
0.53 0.21
Table 2: Quantitative evaluation of nearest digital cousin scene reconstruction in sim-to-sim scenario. This table is an extension of Table I main paper. 'Cat. indicates ratio correctly categorized objects to the total number of objects in the scene. 'Mod.' shows the ratio of correctly modeled objects to the total number of objects in the scene. 'L2 Dist' provides the mean and standard deviation of the Euclidean distance between the centers of the bounding boxes in the input and reconstructed scenes. "Ori. Diff." represents the mean and standard deviation of the orientation magnitude difference of each non-uniformly symmetric object. "Bbox IoU' presents the Intersection over Union (loU) for axis-aligned 3D bounding boxes. 'Ori. Bbox loU' displays the loU for oriented 3D bounding boxes.
Cousin Rank
Training Models
Method
Twin
2 Cousins
Cousins
8 Cousins
0
97 94 93 93
92 87
93 93 93 92 88 85
92.91 90 90 88 85
90 89 88 88 87 87
2
85
94
88
81
94 89
95
90
91
94
89
96
88
92 89 90 89 93
DINO
6
37
42
60
38
45
74
87
91
87 92
93 93
90
92
91
96
95
90 86 94 88 95 90
OOD
30
62
58
51
38
48
65 64
63
44
46
44
61 52 61
74
68
72 54 49
61 50 55
0
97 94 93 93
92 87
98 94 93 91 89 86
92 92 92 90
88 87
94 92 91 88 86 79
DINO
2
85 94 88
81
95 94 95 92 93 95
95 92 88
97
96
95
90
97 88 92
6
88 87 90
91
91 87
87 95 92 94 93
87 92 86
85
91
88 88
88
86
87 89
GPT
OOD
30 62 58 51 38 48
44 43 57 42
46 55
56 43 56
63
47
54
51 37
51 43 64 52
0
97 94 93 93
92 87
85 82 71 61 58
65 63 62 54
52
84 82 81 76 72 70
2
80 80 87 80
74
89
97 97
92 90 90 89
87 76 87 79
85
81
88 87 86 90 83 81
CLII
6
61
67 77
68
64
82
89 94
91 99
96 90
84 89 90 83
90 90
90 90
90 85 78 82
OOD
62 58
48
44 46
52 74
65 63 85
67
64 62 64 57 56
0
94 93 93
87
90 87 84 82
93 91 90
85
84 84
CLIPP
2
0 10
021100
000010
2 0
6
88 85 94
88
90 81
93 91 86 91
92 90
96 95 95 88 95
91 95 90 82 96 93
GPT
OOD
30
62 58 51 38 48
48 69
63 53 70 62
61 62 62 49 34 49
55 50 62 71 46 68
Table 3: Success rates (%) of all policies used Fig. "Cousin Rank" shows rank of test cousins selected by each method. Notice that all test assets are not seen during policy training. "OOD" stands for an asset that is not selected as top- 12 digital cousin by all four methods.
21

---

Input Image
Output Pipeline (a) Output of Pipeline (b) Output of Pipeline Output of Pipeline (d)
Figure 6:
Qualitative sim-to-sim digital cousin scene reconstruction results. Overall, pipeline (d) gives the best scene reconstruction results, while pipeline (c) balances inference time and reconstruction quality.
generally yields reasonable results, the default scale when capturing representative model snapshots affect the selection of the best digital cousin. To refine this process, we propose GPT Model Selection, which first uses DINOv2 embedding distances to select kmodel candidate models and then prompts GPT to choose the best one, with kmnode 10 in practice.
To select the best orientation q. of Ac, we first identify kori candidate orientations based on DI- NOv2 embedding distances between and all snapshots (i,s ) Manoy of the selected digital cousin A.. DINO Orientation Selection involves reorienting the asset Ac, rescaling it, placing it in the scene as described in Section 2.1, normalizing its bounding box, and retaking a snapshot with the same relative position to the viewer camera as detailed Appendix A. The best orientation qe then selected based on DINOv2 embedding distances with the retaken snapshots and x,. How- ever, orientation can be defined for objects within the same category based on key features, under different scales. For example, taller cabinet can be considered to have the same orientation as a shorter cabinet if their frontal faces align. Motivated by this, we propose GPT Orientation Selection, where GPT is prompted to directly select the best orientation among the kort candidate orientations, with k = 4 in practice.
Table presents quantitative evaluation of our digital cousin creation in the sim-to-sim setting, while Fig. 6 provides qualitative visualizations of the output scenes for each pipeline. To ensure diversity the object level, no model is present in more than one test scene.
22

---

Train Set (10k Demoe]
Twin
Cousins
Cousins
(1) DINO
(2) DINO+GPT
(3) CLIP
(4) CLIP+GPT
Shooess Rate (%)
0 [[Twin)
(.(wiin)
(Twin)
(Twin)
Cousin Ranking of the Unseen Test Asset the Digital Twin
Figure 7: Ablation study of how choose digital cousins. Average success rates of door opening policies
trained demonstrations collected from the exact twin, and different numbers cousins. Policies are tested on four assets (from left to right in cach line plot): the exact digital twin, the second unseen cousin selected by the corresponding method. the sixth unseen cousin selected by the corresponding method, and more dissimilar asset (OOD), quantify out-of-domain generalization ability.
Digital Twin
2nd Coussin
6th Cousin
An Out Of Distribation Asset
Digital Cousins Selected
Digital Cousims Selected By DINO-GPT
Digital Cousins Selected CLIP
Dightal Cousinn Seleeted-eeeeeeeeeeeeeee CLIP-GPT
Figure 8: Visualization of digital cousins selected by different methods. Within each row, digital cousins are arranged in descending order based on their ranking. Assets enclosed in dashed boxes represent unseen test assets. DINO based methods better than CLIP based methods for selecting geometrically similar digital cousins.
Based on the category and model matching accuracy, we observe that prompting GPT to select the nearest neighbor from list of candidates outperforms pure DINOv2 embedding distance selection. This advantage likely stems from DINO being influenced by factors such lighting conditions, occlusions, and changes in object scale and orientation. In contrast, GPT focuses better on geometry matching given proper prompting, which is crucial in our real-to-sim setting where an exact digital twin an object is not always available in the simulator. Although GPT occasionally selects an in- correct model, such the bookshelf in the sixth row Fig. still chooses a reasonable substitute that can be appropriately scaled, oriented, and positioned to represent the target object.
Comparing (d) with (c), and (b) with (a) in terms of orientation difference and loU-related met- rics, we find that the performance of GPT Orientation Selection and DINO Orientation Selection is generally comparable. This represents trade-off between time and robustness. Prompting GPT to select the best orientation takes less than 10 seconds per object, whereas the DINO-based method, which involves rescaling, reorienting assets, taking snapshots, and computing DINO scores, takes about 60 seconds per object but is more robust and accurate. Given that orientation will be ran- domized during policy training, recommend GPT Orientation Selection for practical use. For all real-to-sim results, adopt GPT Orientation Selection.
23

---

When comparing (b) with (d), the differences in orientation difference and IoU metrics are minimal, indicating that high-quality scenes can be reconstructed even when the assets in the simulated scene are close approximations (cousins) rather than exact replicas (twins) of the target objects.
Finally, examining the L2 Dist column in Table 2, we that each asset is placed very close the ground truth position. The average L2 distance errors are less than 10 cm for the first seven test scenes, and is only 17 cm for the eighth scene whose scale is 10.23 m.
We further compare DINOv2 against CLIP, another off-the-shelf visual encoder that may be used to match digital cousins. We use Door Open task to verify the best approach to match digital cousins for better policy performance. For each training set, we train policies with different hyperparameters and select the best two combinations based the rollout success rate the original digital twin asset. We then train policies using these best two combinations with three different seeds, resulting in six policies. The results reported in Fig. 7 are based on these six policies.
In Fig. 7, we compare four methods for selecting digital cousins:: (I) DINO: Selecting the asset with the smallest DINOv2 embedding distance the exact digital twin; (2) DINO+GPT: First using DINOv2 embeddings to generate candidate list, then using GPT refine and select digital cousins from these candidates; (3) CLIP: Selecting the asset with the smallest CLIP embedding distance to the exact digital twin; (4) CLIP+GPT: First using CLIP embeddings to generate candidate list, then using GPT to refine and select digital cousins from these candidates. The success rates of all runs used to produce Fig. 7 are shown in Table 3.
Comparing the (1)(2) with (3)() in Fig. 7, we can infer that DINO a better encoder than CLIP to select digital cousins. Policies trained on demonstrations from digital cousins selected by DINO and DINO+GPT achieved approximately 90% success rates on the exact digital twin and demonstrated strong generalization to the second unseen cousin. In contrast, policies trained on cousins selected by CLIP failed to exceed 80% success rates on the digital twin. Interestingly. DINO+GPT appears to act as a more 'dense sampler', focusing more effectively on assets with geometric similarity to the digital twin. The observation that twin policies achieve much higher success rates on the sixth unseen digital cousin selected by DINO+GPT than the sixth unseen digital cousin select by DINO conform this hypothesis.
Fig. 8 presents digital cousins chosen by each method. Digital cousins selected by DINO and DINO+GPT exhibit more consistent overall geometry and handle design with the digital twin than those selected by CLIP. Notably, the cousins chosen by DINO+GPT show the least geometric vari- ance, all featuring two or four symmetrically arranged doors with similar handles to the digital twin. This observation further supports our hypothesis that DINO+GPT may serve as a more 'dense sampler' compared to DINO alone.
B.2 Real-to-Sim Scene Generation: Additional Results
Additional qualitative results of our real-to-sin digital cousin creation and scene generation pipeline are presented in Fig. 9. For multi-view visualizations, please refer to our accompanying video and website.
Our real-to-sim digital cousin creation pipeline has the potential to create cousins and reconstruct scenes from single RGB image without requiring ground truth camera intrinsics. We employ the
model from PerceptiveFields [94] to estimate camera in- trinsic matrix K from the input RGB image. Fig. 10 and Fig. 11 present the ACDC real-to-sim digital cousin scene generation results using the estimated K. This capability may enable large- scale demonstration collection in the future by leveraging in in-the-wild web images that lack ground truth camera intrinsics.
B.3
Failure Cases
We observe that ACDC often struggles under the following conditions:
24

---

Figure 9: Qualitative real-to-sim digital cousin scene generation results. Multiple cousins shown with collecting demonstrations. Images cropped by dashed squares are input RGB images.
(a). High frequency depth information
(b). Occlusion
(c). Semantic category discrepancies
(d). Lack assets within the corresponding category
(e). Object relationships other than "on top"
The first three limitations are directly tied to how ACDC is parameterized, For (a), because ACDC relies on relatively accurate depth estimations for computing predicted object 3D-bounding boxes, poorly estimated depth maps can result in correspondingly poor object model estimations. Native depth sensors can struggle to produce accurate readings near object boundaries where discontinuities in the depth map may occur, and compounded when an object has many fine boundaries, such as plants and fences. Moreover, because rely off-the the-shehf foundation model (DepthAnything- v2) to predict synthetic depth maps, inherit its own set of limitations, such poor predictions
25

---

Qualitative real-to-sim digital cousin seene generation results without ground truth camera trinsics K. Images cropped by dashed squares are input RGB images.
Figure 10:
esoteric objects or under adversarial visual conditions. Similar to (a), occlusion (b) becomes signif- icant when it results in an inaccurate estimation of a given object's overall bounding box. For some objects, such as cabinets and other furnitures, observing two faces is usually sufficient, for other smooth objects, such as balls or plushes, occlusion can have nontrivial impacts on the corresponding generation of digital cousins. Lastly, ACDC struggle when there is mismatch between object category labels from the input RGB image and the available object asset categories from our dataset. Because we do not enforce any naming or category-abstracte matching method (CLIP) may fail to associate categories due to esoteric naming schemes (e.g.: bottom cabinet no top) or abstraction level mismatches (e.g.: cup vs. coffee cup vs. drinking cup vs. water_cup), resulting in suboptimal object asset candidates when selecting digital cousins.
level from our dataset, our category-
However, believe that increasingly powerful foundation models can help address some of the current limitations. For instance, have replaced DepthAnything with DepthAnything-v2 [95].
26

---

which offers improved depth estimation, even capturing fine- ene-ggained details more effectively. As shown in the last two rows of Fig. 11, the plant is reconstructed with greater accuracy, benefiting from the enhanced depth estimation provided by DepthAnything--v.. Using SAM-v2 instead of SAM offers better object masks. Replacing GPT-4v with GPT-4o also results in smaller orientation differences and higher bounding box IoU.
For (d), our method relies sufficient number of candidate assets to select digital cousins for real- world objects. This limitation can negatively impact feature matching and orientation estimation. When the number of available assets is limited within certain categories, the reconstruction quality can be sub-optimal. For instance, in BEHAVIOR- IK, there is only one pot asset, one toaster asset, and two coffee maker assets. When the input scene contains these objects, most digital cousins do not fit the corresponding category, leading to inaccurate orientation estimations due dissimilar assets.
For (e), our method only models the "on top" relationship between objects. For other relationships, such as a kettle inside coffee machine or books on a bookshelf, one object is placed on top of the other. However, when an object is "inside" another without a top, like a cushion in a sofa, we can still achieve reasonable reconstruction. We do this by initially placing the cushion on top of the sofa's bounding box, then moving it downward until it makes contact with the sofa.
B.4 Comparison with URDFormer
URDFormer [31] is recent state-of-the-ar method for scene-level generation from single RGB image, with a focus on object articulation reconstruction. As this method is quite relevant to our setup, we run a qualitative experiment to compare ACDC against URDFormer. We evaluate both ACDC and URDFormer on five real-world kitchen scenes: our kitchen scene, URDFormer's high- lighted kitchen scene, and three additional kitchen scenes. We showease the original RGB image as well URDFormer's and ACDC's outputs side-by-side in Fig. 12. We highlight some key differ- ences between URDFormer and ACDC below:
URDFormer is optimized for trained of object categories, while ACDC is object- agnostic and can be applied any arbitrary set of objects.
URDFormer can generate realistic synthetic textures from the given input image, while
ACDC does not modify matched object asset textures.
URDFormer relies on accurate bounding box information which often requires manual human annotation, whereas ACDC is fully automated and uses no human input.
In general, we find that while URDFormer can produce synthetic scene textures that visually match the real-world scene's textures, ACDC can match or even outperform URDFormer's ability to spa- tially reconstruct given scene accurately, while additionally being object-agnostic (and thus able to detect and generate a much more diverse set of object categories) and fully automated with no manual human annotation.
B.5 Policy Training Details
We train robot policies using the demonstrations collected (see Appendix A.5. Our action space is delta end-effector actions, expressed as 6-dimensional (dr, dy. dz) delta position and (dax, day, daz) delta axis-angle orientation command. The commands are then executed via Inverse Kinematics (IK). Our observation space consists of (end-effector position, end-effecctor orientation, end-effector gripper joint state) propriaception, and unified point cloud.
The point cloud is computed by first converting all depth images into a single point cloud with a unified frame (in our case, the robot frame), with all non-task relevant objects such as the robot and background masked out. For the real-world setting, we efficiently mask and track all non- task relevant objects using XMem [96], allowing align the sim- and real-worke point clouds. We then additionally add pre-computed point cloud representation of the robot's 's gripper fingers,
27

---

placed at the known ground-trune -ruth location using the robot's onboard proprioception and forward kinematics. In addition to the (I, y; z) per-point values, we additionally add a fourth binary value e E {0, 1), classifying whether that point belongs to either the scene or the robot's gripper fingers. Finally, we downsample the point cloud to a fixed size using farthest point sampling (FPS). Note that with the exception of the Putting Away Bowl task, the point cloud is generated from single, over-the-shoulde camera. In the Putting Away Bowl task, we additionally add another over-the- shoulder camera the other side of the robot, as well as wrist camera, since this task exhibits much heavier occlusion during different stages compared to the other tasks.
All of our policies are trained using Behavioral Cloning with an RNN to capture the prior history of actions and GMM to capture the distribution over demonstrations. We use 2-layer, 512- dimension PointNet [97] encoder to encode our raw point cloud observations, which undergo further random /downsampling, translation, noise jitter) before being passed to the actor network. We also convert the binary value into 128-dimensional learned embedding, to better enable the network to differentiate useful features between the robot fingers and the scene. Our policies use an RNN horizon of 10, RNN hidden dimension 512, are optimized using Adam [98].
During evaluation, we take the best performing checkpoint for given run and evaluate 100 times. These results are then aggregated across multiple runs to give our finalized results.
B.6 Sim-to-Sim Policy Learning with Digital Cousins
DINO Dist.
Training Models Cousins
Task
Twin
Cousins
8 Cousins
All Assets
0
96 92 91 91 87 83
100 96 95 92 90 74
95 94 94 92 87 84
97 95 95 94 90 88
94 93 86 67 66 60
Door
7.25
83
82
95 91
93
93 91
88 99 95 92
75 68
Opening
7.59
69
60
66
65
73
72
98 85 93 87 95
96 77
95
96
87
91 96 98
91
87 97
86 83 88
65
73 74
18.93
58
63
72
64
66
57
74 80
85
57
71 65
47 68
62
78
77 72
80 72
73 75
75 76
72 68
68
53
60 59
0
87 86 85 73
718
85 80 72 69 53 7
81 67
63
61
84 73 71 65
63 62
73 68
58
51
6
Drawer Opening
9.42
85
10
85 86
86
82
78 72
79
12 12
14.97
81
56
84
14
60
88
84
65
76
71
16 14
86
69
81 66
76 67
82
81
16
10
17.6
38
36
45
30 32
17
37
35
41
42
13 11
79 32
20
23 15
97 88
94 74 90 62
81
75
84
80
8 15
0
33 14 14
11 10 9 8.5 10 1 31 24
Putting
14.17
000000
Away
14.44
30000
Bowl
17.73
00000
0.0.000
Table 4: Success rates (%) of all policies used in Fig. 4 and Fig. 13, "DINO Dist." shows the DINOv2 embedding distances between test assets and the original digital twin.
Training Models
Cousin
Rock
Twin
(1Rund.)
Twin
Asseist (Rand.)
Twin All
Cousins
Cousins
S Cousins
Twis +
Conoin
Twin Cousins
7 Cousins
Twin
Twin+3Cuusins ("Rund.)
TousinCousins (1Rand)
0
94-91 92
88 58 85
92 86 81
13.935 89
92. 90
94 91 86
97 94
94 91 86
95 94
97 96
97 93.93.0
2
85
53 90
90 77 76
91 36 89
94 95 93
92 S3 95
91 90
91 81 89
96 9199996 96
96 91 92
93 96 9:1
92 92.5599 99
6
87
84 81 86
88 X1
95 92.93
92 06 94
88 88
94 92
90 90
89 13 91
96 98 33
94 94 91
11
2.42
36 36 26
41 1..00-3
39 03 47
25 34
58 45
77.24
36 42:47
40 33 29
40 46 422
35 43 45
12
35 49449.4 48
76 60 76
90 88
87 96 91
83.85 92
94 92.05
51 52.52.9 52
87 82 02
87 88
89.93
87 97 90
OOD
62
50 54
76.6177 76
43.57 46
43 56.66.3
31 51.61
69.59:54
53 43.55.11
69.53
70
55.65 65
Table 5: Success rates (%) of all policies used in Fig. 14. "Cousin Rank" shows the rank of test cousins selected by each method. Notice that all test assets are not seen during policy training. "OOD" stands for an asset that is not selected as top-12 12 digital cousin by all four methods.
As an extension of Fig. 4, Fig. 13 presents the average and standard deviations of success rates of policy rollouts on the original digital twin and multiple unseen assets. The success rates of all runs used to generate Fig. 4 and Fig. 13 are detailed in Table 4. For each training set, we train policies with different hyperparameters and select the best two combinations based on the rollout success rate on the original digital twin asset. We then train policies using these best two combinations with three different seeds, resulting in six policies. The results reported in Fig. 4, Fig. 13, and Table 4 based on these six policies. We note that for the third Putting Away Bowl task, we only evaluate on five runs due to resource constraints.
28

---

An unexpected behavior is observed in the Drawer Opening task, where the 4-cousin policies per- form sub-optimally. We believe this is due to the limited number of cabinets with drawers available for cousin selection. Among the four cousins, the first two are geometrically similar, as are the last two, but there is a significant similarity gap between the second and third cousins. This is partially illustrated by their DINO embedding distances to the digital twin: 7.78, 9.32, 14.10, and 14.90. The demonstrations collected these four assets may not form high-quality distribution for train- ing. In contrast, the 4-cousin policy in the Door Opening task yield decent results, likely because there are more than 40 assets available for cousin selection, allowing reconstructed digital cousins to form a relatively narrower distribution. The geometric similarities between the four cousins in the Door Opening task are more continuous in terms of DINO similarity to the digital twin, with DINO distances being 6.49, 7,51, 8.13, and 9.66. However, 8-cousin policies still performed well this relatively limited category, much better than all-assets policies and twin policies. A key takeaway is that: (1) when there are a sufficient number of assets choose cousins from, all cousin policies can outperform twin policies held-out cousins, and (2) more cousins should be found when the number of available assets is relatively small for the target category.
Digital Cousins Improve Policy Training Stableness. Comparing the standard deviation of poli- cies trained on the digital twin, digital cousins, and all assets in Fig. 13, we find that all-assets policies are the most unstable, followed by twin policies, while 8-cousin policies are the most sta- ble. This highlights another advantage of training digital cousin policies: the policy training process on demonstrations collected from a set of high-quality cousins can be more stable, i.e., more robust against different random seeds and requiring less tuning.
Digital Cousins Improve Policy Robustness. To further examine the relative impacts of digital cousins against naive domain randomization, we re-run our sim-to-sim experiment on the Door Opening task against additional baselines: (a) policies trained digital twins with increased do- main randomization (greater scaling randomization: =75)), (b) policies trained both the digital twin and digital cousins, where half of the dataset (5k demonstrations) are collected from the exact digital twin, and another half of the dataset (5k demonstrations) are collected from digital cousins, (c) policies trained on both the digital twin and digital cousins with increased domain randomiza- tion (greater scaling randomization: =75%), and (d) policies trained on both the digital twin and all assets from the nearest three categories with increased domain randomization (greater scaling randomization: =75%). Our results can be seen Fig. 14. The success rates of all runs used to generate Fig. 14 presented in Table 5. We use DINO+GPT to select digital cousins. For each training set, we train policies with different hyperparameters and select the best combination based on the rollout success rate on the original digital twin asset. We then train policies using the best combination with three different seeds, resulting in three policies. We also report policy rollout success rates on two more unseen digital cousins. Test assets are seen during training of Twin + All Assets (More Rand.) policies, but are not seen during training of other policies. Other experiment settings are the same as how Fig. 7 and Fig. 13 are produced. We find that naive domain randomiza- tion, even when increased, is insufficient to overcome the increasing domain gap when the digital twin policy is deployed on unseen cabinets. On the other hand, find that the policies trained on the digital twin and digital cousins/all assets together exhibit similar performance compared to the policies trained exclusively on digital cousins, suggesting that perfect reconstruction via digital twins may not be necessary for sufficiently transferring a trained digital cousin policy to the original target scene.
29

---

Qualitative real-to-sim digital cousin seene generation results without ground fruth camera in- trinsics K. Images cropped by dashed squares are input RGB images.
Figure 11:
30

---

URDFormer Result
Input Image
ACDC Result
Figure 12: Qualitative comparison between ACDC and URDFormer.
31

---

Twin
Consins
Cousins
8 Consins
Assots
Task 1: Door Opening
Task 2: Drawer Opening
Task 3: Putting Away Bowl
100
80
Steccess Rate (%)
600
0 (Twin)
7.25
7.50
18.05
(TTaa))
9:42
16.07
17.00
0 (Twm)))
7.25
7.59
18.00
DINO Distance of the Test Asset the Digital Twin
Figure 13: Average success rates (with standard deviations) of policies trained on demonstrations collected from the exact twin. different numbers of cousins, and all assets in the three nearest categories. Success rates reported for three tasks: Door Opening, Drawer Opening, and the composite task of Putting Away Bowl. Policies are tested on four assets (from left to right in each line plot): the exact digital twin, the second unseen cousin, the sixth unseen cousin, and a more dissimilar asset, to quantify out-of-domain generalization ability. The DINO embedding distance to the digital twin is used as the quantitative metric to rank asscts and select cousins. Error bars indicate the standard deviation, reflecting the stability of policy training.
Train Set
Twin
Cousine
Twin + Cousin
Twin + 3 Cousin [More Read.)
Twin (More Rand.)
1 Cousine
Twin 3 Cousin
Twin 7 Cousin (More Rend.)
Twin Asarts (More Panui.)
8 Cousins
Twin 7 Cousin
100
Success Rate (%)
80
60
60
60
20
(1TWm)) 2nd
6th
11s. 12th OOD
0 (Twm)) 2nd
6th
11s. 12th COD
01 (TWta)) 2nd
6th
11:t
12tb
OOD
Cousin Ranking the Unseen Test Asset the Digital Twin
Figure 14: Average success rates of door opening policies trained on demonstrations collected from the exact twin, the exact twin with more aggressive randomization, different numbers of cousins, the exact twin with asset-level randomization, and the exact twin with asset-level randomization and more aggressive shape ran- domization. For Twin Cousins and Twin All Assets training datasets, half of the dataset is demonstrations collected from the exact twin, and another half of the dataset demonstrations collected from different num- bers cousins or all assets from the nearest three categories. Policies are tested on six assets (from left to right in each line plot): the exact digital twin, the second unseen cousin, the sixth unseen cousin, the eleventh unseen cousin, the twelves unseen cousin, and more dissimilar asset (OOD).
32


Reviews:
[{"review_id": "ehQ5gOzHO9", "reviewer": "Reviewer_wrjg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a pipeline named ACDC that can automatically create digital cousins of a real-world scene from a single RGB image. Compared to digital twins that try to exactly replicate the real-world scene, digital cousins allow more variations in the generated scenes. The proposed pipeline first extract objects and their masks using a combination of GPT4v and grounded-SAM. It then retrieves the relevant objects and its orientation from an existing object database, by matching DINO-v2 features. Finally it places the retrieved objects into the scene. Qualitative results and some quantitative results are presented to access the quality of the generated digital cousins in both sim2sim and real2sim settings. Additionally, policy learning can be conducted in the digital cousins, and the policy learned in digital cousin is shown to perform better than the policy learned in digital twins when generalizing to more variations. The learned policy also shows zero-shot sim2real transfer to a real-world door opening task.", "summary_of_recommendation": "Please see the detailed review section.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- For the real2sim setting, how are you getting the articulation type/axis of the real-world object? E.g., a cabinet can have its hinge joint on the left edge or the right edge. However, this seems to be not considered in the current asset retrieving process. \n- Line 118 - 119: \"we place the asset’s bounding box center at the centroid of the corresponding input object pointcloud\" -- Isn't the input point cloud in the camera frame? But I imagine you need to put objects in the simulator in the simulation's world frame? How is the conversion done here?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n677UTR22Z", "reviewer": "Reviewer_oRdT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method for creating digital cousins: simulated environments which match the scene in front of the robot at a high level (e.g. similar spatial layout and object categories), but not exactly, allowing for some variation in the specific object instances. This allows the robot to train on a variety of object instances in simulation, aiming to improve robustness and address the sim2real gap.", "summary_of_recommendation": "Digital cousins is an elegant method idea with potentially large impact potential, as it takes scalable simulation training and grounds it in the scene that the robot actually has in front of it, while relaxing the need for perfect digital twin reconstruction. However, there are still a few issues regarding motivation and experiment setup described above which I would like to see ironed out to make this a stronger paper. Update post-rebuttal: my concerns have largely been addressed, so I have upgraded my score to Strong Accept.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How long does the whole process take, from start to end (and stage by stage), for the digital cousins method vs digital twins? Specifically for the policies used to produce the results in the paper. It may be worth discussing this, because likely digital cousin training takes longer than digital twin training.\n2. Figure 4: it is difficult to understand why the digital cousins method can match or even exceed the performance of the digital twin method in the zero-distance setting, where the digital twin is a perfect replica of the test-time object and there is no sim2real gap. For example, for Task 1, the method trained on just 2 cousins already matches the performance of the digital twin method. How can this be? Some more insights on this would be much appreciated. Is it because the closest cousin is a very similar object instance to the twin anyway? It may be good to see images side by side to check this. Or is it because of how the demonstrations are collected, and the distribution of robot states seen during training being wider for the digital cousin than for the digital twin?\n3. Figure 4: “all assets in the three nearest categories”. Why was this baseline chosen instead of simply “all assets in the same category”? Does it make sense to define the task on an object of a different category? Comparing against all assets of the same category would help answer the question of: is there a \"sweet spot\" of choosing the number of cousins which are quite related to the test object but not so many that training is wasted on completely irrelevant instances.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eoe5GM5STb", "reviewer": "Reviewer_hB4w", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a new method for building \"Digital Cousins\", environmental models that are similar to, but not identical copies of real world scenes from limited human data collection.", "summary_of_recommendation": "I have recommended weak-reject for now, but will easily improve my score if the issues discussed above are adequately addressed.  There is clearly a lot of good work here, and this is a solid contribution, but I think there are important and hopefully not too difficult adjustment that should be made before it is accepted.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Please discuss/address the issues around disentangling the questions of accurate reconstruction and intentional variation (the first bullet point under cons above).\n- What are the relative benefits of ACDC compared to URDFormer?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers thought that the method here was good and practical. 

The main complaint was in the motivation of digital cousins vs digital twins + domain randomization. Reviewer hB4w states this most clearly. Instead of generating a bunch of digital cousins, why not generate a digital twin and then do domain randomization? An effective way to answer this objection would be to provide baselines using SOA digital twin methods (reviewers provided appropriate citations) and to add domain randomization. If this comparison is not appropriate, the authors will need a good argument to that effect.

It would also help to improve the experimental section. Currently there are only 3 real2sim exps and it would help to have more. It would help to have quantitative evaluations in the real2sim setting. And, the results of Table 1 are hard to interpret; the authors should try to think of different ways to evaluate or provide benchmarks that could provide context.

Post rebuttal: great work
Decision: Accept