DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control
Authors: Guiliang Liu
Yueci Deng
Runyi Zhao
Huayi Zhou
Jian Chen
Jietao Chen
Ruiyan Xu
Yunxin Tai
Kui Jia
Venue: ICML 2025 poster (2025)
Keywords: Data Engine
Embodied AI
Robot Control
Manipulation
Policy Learning
Primary area: applications->robotics
TL;DR: We introduce DexScale, a data engine designed to perform automatic skills simulation and scaling for learning deployable robot manipulation policies.

Abstract: A critical prerequisite for achieving generalizable robot control is the availability of a large-scale robot training dataset. Due to the expense of collecting realistic robotic data, recent studies explored simulating and recording robot skills in virtual environments. While simulated data can be generated at higher speeds, lower costs, and larger scales, the applicability of such simulated data remains questionable due to the gap between simulated and realistic environments. To advance the Sim2Real generalization, in this study, we present DexScale, a data engine designed to perform automatic skills simulation and scaling for learning deployable robot manipulation policies. Specifically, DexScale ensures the usability of simulated skills by integrating diverse forms of realistic data into the simulated environment, preserving semantic alignment with the target tasks. For each simulated skill in the environment, DexScale facilitates effective Sim2Real data scaling by automating the process of domain randomization and adaptation. Tuned by the scaled dataset, the control policy achieves zero-shot Sim2Real generalization across diverse tasks, multiple robot embodiments, and widely studied policy model architectures, highlighting its importance in advancing Sim2Real embodied intelligence.

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control
Guiliang Liu
Yueci Deng"
Runyi Zhao'
Huayi Zhou' Jian Chen Jietao Chen 2
Ruiyan Xu'
Yunxin Tai
Kui Jia" 12
Abstract
A critical prerequisite for achieving generalizable robot control is the availability of large-scale robot training dataset. Due to the expense of collecting realistic robotic data, recent studies explored simulating and recording robot skills in virtual environments. While simulated data can be generated at higher speeds, lower costs, and larger scales, the applicability of such simu- lated data remains questionable due the gap between simulated and realistic environments. To advance the Sim2Real generalization, in this study, we present DexScale, a data engine de- signed to perform automatic skills simulation and scaling for learning deployable robot manipula- tion policies. Specifically, DexScale ensures the usability of simulated skills by integrating diverse forms of realistic data into the simulated envi- ronment, preserving semantic alignment with the target tasks. For each simulated skill in the envi- ronment, DexScale facilitates effective Sim2Real data scaling by automating the process of do- main randomization and adaptation. Tuned by the scaled dataset, the control policy achieves zero- shot Sim2Real generalization across diverse tasks, multiple robot embodiments, and widely studied policy model architectures, highlighting its im- portance in advancing Sim2Real embodied intel- ligence. The project webpage at: https://edem- aigiihub.io/dexsealee github..o/.
Project
Simulator
Generate
Data
Finetune
Model
DexScale
Deploy
Figure 1: An example of using DexScale. Given a human video, DexScale simulates robot skill dataset for training deployable control policy without requiring realistic data.
making processes (Paolo et al., 2024). In recent years, robots have emerged widely studied form of embodi- ment (Duan al., 2022). In principle, robot control skills can be highly persatile-genrralle mains and incorporating multiple forms of sensory data. Prioritizing the availability of sensors and the impact of real-world application, we primarily focus on advancing visuomotor robotic learning for dexterous manipulation.
across various do-
Many recent studies have proposed learning- based systems for manipulating various object based on image sensor sig- nals. For example, modular approaches first predict affor- dances, such as grasp poses (Mousavian et al., 2019; Sunder- meyer et al., 2021; Wu et al., 2020; Jiang et al., 2021; Fang et al., 2023) or keypoints (Manuelli et al., 2019; Qin et al., 2020; Mo al., 2021; Yuan al., 2024), for different tasks and train low-level policies complete these task. On the other hand, end-to-end approaches aim at learning pol- icy model that directly maps visual inputs robot actions methods (Levine et al., 2016; Zhang et al., 2018; Florence et al., 2020; Chi et al., 2023). While these methods have promising performance in specific scenarios, the learned skills lack generalizability to diverse environments, making them unsuitable for deploying the wild.
1. Introduction
key milestone in advancing modern AI systems is the development of embodied intelligence, which aims to seam- lessly integrate agent's 's physical body, sensory perception, and environment into its learning, reasoning, and decision-
Equal contribution 'School of Data Science. The Chinese University Hong Kong. Shenzhen DDexForee, Shenzhen. Corre- spondence to: Kui Jia <kuijia@ cuhk.edu.en>.
With advancements in learning algorithms, large-scale datasets, and hardware systems, large foundation models have achieved human-level performance across wide vari- ety of environments and downstream tasks. Their success inspired the construction of foundation models for robot control such as RDT (Liu et al., 2024b), Open-VL.A (Kim
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
al., 2024), ACT (Zhao al., 2023), and Octo (Team et al., 2024). Training these models depends on massive amounts of data. Apart from the text and image data readily available on the internet, learning robotic control policies requires multi-modal sensory signals, robot self-states (i.e., propri- oception), action trajectories for end-effectors, and body joints, all situated a calibrated 3D space with real-world physics. It often requires structured pipelines or specially designed equipment generate such datasets.
configurations. This enables DexScale to actively discover effective robot control trajectories across wide range of scenarios. Additionally, to remove task-irrelevant features from the dataset. DexScale adapts visual sensory data from both real and virtual environments into object-oriented and pose-affordance representations. The resulting trajectory dataset facilitates the development of imitation policies that can be seamlessly deployed in real-world applications.
We demonstrate that the DexScale pipeline can be seam- lessly integrated into the training of widely studied em- bodied robot policies, including Acting Transformer (Zhao et al., 2023), Diffusion Policy (Chi et al., 2023), and Robot Diffusion Transformer (Liu et al., 2024b). By leveraging DexScale, these methods achieve zero-shot deployment for desired tasks. Such deployments can be effectively scaled across various robot configurations (e.g., single-arm and dual-arm setups with different robot models) and diverse manipulation tasks, including object grasping, articulated object (e.g., box) manipulation, and table rearrangement. With extensive performance evaluation and case studies, we demonstrate that DexScale can significantly advance the Sim2Real learning of deployable policies across multiple scenarios.
Although previous works have developed real-world data collection systems for robotic action trajectories (Zhao et al., 2023), these systems often rely on costly equipment and sub- stantial manpower. Furthermore, in the absence of standard- ized data collection protocols, significant effort is required to align datasets collected from different systems. To over- come these limitations, an alternative method is data synthe- sis based on the simulated robot learning environment. In principle, simulated data generation can produce an infinite amount of robot skills (Wang et al., 2024), significantly ad- dressing the "data hunger" challenge in training large-scale robotic control policy. However, skills trained on simulation data face significant challenges when deployed to real-world environments due to: 1) Semantic Mismatch: There is no guarantee that the semantics of the simulated environment align with the target environment. A policy trained in mis- aligned simulation is unlikely to be successfully deployed in practice. 2) Simulation Discrepancy: Despite substantial advancements in simulators with realicc- and photo- realistic rendering (Todorov et al., 2012; Xiang al.. 2020; Makoviychuk et al., 2021), discrepancies between simulated and weal-world environments are inevitable, continuing to hinder deployment performance.
2. Related works
Large-Scale Dataset for Robotic Learning. Building large-scale datasets is essential to training generalizable robotic policies. For instance, to support the development of adaptable robotic controllers, RoboNet (Dasari et al., 2019) and D4RL datasets (Fu et al., 2020) capture rich robot actions across different manipulation tasks. In re- cent years, to enhance the training of end-to-end foundation models, extensive data collection systems have been built across different scenarios, including object manipulation (e.g., Bridgedata (Ebert et al., 2022; Walke et al., 2023)) and language conditioned and vision-based tasks (e.g., RT- (Brohan et al., 2023) and RT-2 (Zitkovich et al., 2023)). To address the scarcity of robot training data, MimicGen (Man- dlekar et al., 2023) and DexMimicGen (Jiang et al., 2025) leverage scaled human teleoperation motions to generate robotic manipulation data. To further improve generaliza- tion performance, RT-X (O'Neil et al., 2024) introduced the Open X-Embodiment (OXE) datasets, which are cross- domain and were collected from multiple types of robots across numerous institutions. A fundamental challenge in constructing real-world datasets lies in the high costs asso- ciated with equipment and manpower.
In this paper, we introduce a data engine, DexScale, to ad- dress these challenges and facilitate the training of Rim2Real transferable dexterous manipulation policies based on visual observations. Figure illustrates an example application of DexScale. Given task-descriptive observations (e.g., human demonstration), DexScale generates simulated dataset, based on which the robot control policy can generalize to realistic tasks in a zero-shot manner.
To ensure semantic alignment, DexScale introduces a data projection mechanism that enables practitioners to incor- porate heterogeneous formats of environmental and action priors from the target tasks into the simulator. These priors capture both static features (e.g., object types and layouts) and dynamic features (e.g., human demonstrations and in- teractions with objects), enabling the simulator to generate desired environments for learning preferable skills in an effective and reliable manner.
Environment Simulation for Robotic Learning, Instead of collecting data from real-world, an alternative method is generating robot skills (e.g., action trajectories) from a simulated environment based on physics engines, such as MuloCo (Todorov et al., 2012), PyBullet (Coumans & Bai,
To address the Sim2Real discrepancy, DexScale employs strategic data scaling approach, incorporating an automatic domain randomization process to generate diverse learning
2

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
2016-2021), Isaac Gym (Makoviychuk et al., 2021) and Sapien (Xiang al., 2020). In this context, numerous vir- tual environments have been proposed for a variety of tasks, including dexterous manipulation (e.g,, RLBench (James et al., 2020), ManiSkill (Tao et al., 2024), and Robo- Casa (Nasiriany et al., 2024)), robot skill imitation (e.g., Robo.Mimic (Mandlekar et al., 2021)), physical interactions (e.g.. ThreeDWorld (Gan et al., 2022)), rearrangement (Ba- tra et al., 2020), high-level reasoning (e.g., Alfred (Shridhar et al., 2020) and CabiNet (Murali et al., 2023)), and genera- tive skill learning (e.g. GenAug (Chen et al., 2023b), Robo- Gen (Wang et al., 2024) and GenSim (Hua et al., 2024)). To better reflect the realistic control environment, Habitat 3.0 (Puig et al., 2024) and A12-THOR (Kolve et al., 2017) enable embodied agents to interact with photo-realistic en- vironments for navigation and manipulation tasks. The con- tinuously works ProcTHOR (Deitke et al., 2022), expand A12-THOR with procedurally generated environments to enhance generalization. These simulators aim to benchmark robotic skill learning by offering diverse environments and tasks. However, transferring the learned control skills to weal-world scenarios often requires additional fine-tuning with realistic data. The development of simulators for zero- shot Sim2Real deployment remains challenge.
lated environments.
These environments can be modeled as episodic Markov Decision Process (MDP), represented as M = (S, A, PT, R. 7, Po), where:
a state 8; E S captures the semantic information of a scene, encompassing the configuration (e.g., layouts, ap- pearance, and physical characteristics) of various types of objects and the robots subject to control.
an action a E A define control signals for robot. The common representations of action include the end effec- tor pose of the robot, the angle of each joint, the torque applied to each joint, and the velocity of each joint.
Transition function Py characterizes the impact of robot action 0r to the configuration of current state st. thereby projecting the St to a new scene represented by St+1-
Reward function R(s, a) captures various degrees of opti- mality of the robotic agent in completing targeted tasks after acting a in state 8. For instance, R(s, a) can assess "hard" optimality (i.c., sparse rewards), which determines whether a task is completed (e.g.. whether the target object is successfully grasped) or "soft" optimality (i.e., dense rewards), which measures the degree of task completion (e.g., how close the gripper is to the target object).
Po denotes the initial state distribution. Our DexScale support projecting the initial state from realistic scene or automatically generating a new scene.
Sim2Real Generalization. In the application of robot con-
trol, Sim2Real generalization techniques often involve: 1)
Domain randomization tackles Out-offDDstrrbuut scenarios in practical applications augmenting the training dataset with randomized visual and physical features (Chen
(OoD)
7 (0, 1] denotes the discounting factor which weights the importance of future rewards relative to immediate rewards in decision- processes such that a lower y emphasizes immediate rewards more heavily, and a higher gives greater significance to future rewards.
et al., 2022a). To determine the scale of randomization, recent studies consider updating the distribution of random- ization parameters by automatic learning (OpenAl et al., 2019), active exploration (Mehta et al., 2019). Bayesian update (Muratore et al., 202)b;a), offline inference (Ti- boni et al., 2023) and continual learning (Josifovski et al., 2024). 2) Domain adaption addresses the gap between simulated and real-world domains by mapping them into feature space. Commonly studied spaces include depth images (Agarwal et al., 2022; Cheng et al., 2024), point clouds Taubok-a & Harada, 2020; Qin et al., 2022: Chen et al., 2023a; Hua et al., 2024), and environmental dynamics (Memmel et al., 2024). 3) Feature alignment is essential when the learned policy depends on privileged information that is inaccessible in real-world applications. A commonly studied approach is knowledge distillation, where skills are transferred from the learned policy to a de- ployable policy using teacher-student framework (Kumar et al., 2021; Qi et al., 2022; Agarwal et al., 2022; Cheng et al., 2024).
Note that we consider an episodic MDP with stationary policy r(a)s), which depends only on the current state and is invariant to the decision time step t. Additionally, in many environments, robot sensors only observe partial in- formation about the underlying states. In this case, a com- mon method is to embed the historical observations into the state, such that 8; = [o,, ot-1,.... J. where o; represents the partial observation at time step t. As a data engine, DexScale supports the automatic discovery of skills, denot- ing the sequence of action trajectory for finishing task T = [So. ao..... ST, ar). To acquire these skills, DexScale automatically decomposes long-term tasks (c.g., cooking a meal) into atomic tasks (e.g., chopping potatoes). For tasks with a "hard" optimality (e.g., whether the target object is successfully grasped), DexScale primarily relies on motion planning techniques (e.g., antipodal point detection (Chen & Burdick, 1993) combined with inverse kinematics (D' Souza et al., 2001)). For tasks with "soft" optimality (e.g., how close the gripper is to the target object), DexScale sup- ports the automatic design of dense reward functions (Ma et al., 2024) and employs Reinforcement Learning (RL) algorithms (Sutton, 2018) to acquire control skills.
3. Problem Formulation
Simulation Environments for Robotic Learning. The goal of Dexecale to learn robot control skills based on simu-
3

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Heterogeneous Data
Environment Simulation
Sim2Real Data Scaling
Sim2Real Deployment
Projection
Objects
Layouts
Deplovable model
Image
Video
Read Config
Pro-
LLM
ject
Features Ranking
Imitate
#1
#2
#N
Obs-Acttio Trajectories
Tele-
Action Trajectory
operation
Actions
Scene
Auto. DR
AIDR
SADR
DA
Feedback
Reinforcement Learning, Trajectory Optimization, and Motion Planning
Policy Fine-Tuning
The pipeline of DexScale operates data engine, taking task-descriptive data as input and generating skill dataset to support Sim2Real transfer. This enables the zero-shot deployment of robot policies in realistic environments, The details of these components are introduced in Section 4.
Figure 2:
Sim2Real Domain gap. The skills learned in a simulated environment often face significant domain gap when ap- plied to the deployment environment (Xie et al., 2024). In this section, we systematically analyze the factors that cause the gap in deploying models trained with simulated data onto real robots. As it is illustrated in Table 1, we analyze by inspecting the steps in simulation- cetrriiooppeepptioonnneee acting loop. Such analyses would give insights on how to better design the simulated robotic learning environment that can effectively close the Sim2Real gap.
the physical properties in the realistic environment might misalign with those in the simulated environment, inducing inconsistent outcomes of implementing the same action.
Table 1: Features that might induce Domain Gap. The determination of such parameters is a recent study of the generalization gap (Xie et al., 2024).
Lighting, Table Texture, Background, Distractors, Object Locations, Object Orientations, Object Texture and Object Shape
Simulation Stage
Simulation Stage. A major factor impacting Sim2Real per- formance is the inaccuracy in simulating real-world see- narios. Notably, the semantic information the simulated environment often does not align with that of the realistic environment. For instance, objects may be inaccurately scaled, either oversized undersized, and the location and shapes may not accurately reflect those found in the real world. More importantly, there is no guarantee that the sim- ulated environment can fully cover the realistic semantic distributions, and the disparity between simulated and real- (OoD)
Camera Position,
Perception
Camera Orientation,
Stage
Camera Field of View
Acting
Robot Configuration.
Stage
Physics Properties
4. Data Engine for Sim-to-Real Generalization
To acquire robot control skills that effectively bridge the aforementioned Sim2Real gap, DexScale can serve as an automated data engine. As shown Figure 2, DexScale takes descriptive data (e.g., scenes and demonstrations) from the target task input and generates scalable dataset that supports the efficient training of Sim2Real-deployabbe policies. We detail the design of DexScale in the following.
world environments can result in scenarios during deployment, making the planned motions less adaptable the deployment environment.
Perception Stage. A primary application of simulator is to replicate or reconstruct perception from realistic sensors. A critical factor affecting Sim2Real performance during this process is the inconsistency hardware setups between the simulated and real environments (e.g., differences in the position, orientation, and field of view of cameras).
4.1. Heterogeneous Data Projection
The goal of heterogeneous data projection is to map descrip- tive observations of the target task from real-world applica- tions to a simulated environment, ensuring the applicability of the generated skills. To achieve this, DexScale primarily supports two key types of projection: scene projection and action trajectory projection, as detailed below.
Acting Stage. Embodied agents control various types of embodiments (e.g., arm, and hand) by executing robot ac- tions. In this stage, mismatches between embodiments can impact deployment. For example, using different robots during simulation and deployment, or having URDF config- uration errors, can lead to discrepancies. More importantly,
Scene Projection. To overcome the Sim2Real gap, a crit- ical challenge to handle the semantics disparity between simulated and real- world environments, For example, the

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
scene constructed by generative simulation, there is no guar- antee the distribution of objects and their configurations can semantically align with these in realistic scenarios. To over- come this limitation, DexScale enables projecting the static seene information to the simulated environment. Specifi- cally, inspired by Digital Cousin (Dai et al., 2024), DexScale extracts relevant per-object information from the input RGB image. Using this information, DexScale matches each de- tected object to their "digital cousins" (i.e., similar objects) in the asset dataset. For the articulated objects (e.g., draw- ers and boxes), DexScale can post-proces, them to create a fully interactive simulated scene by matching them with CAD models and the generated objects (Liu et al., 2024a). Besides the automatic scene project, DexScale provides user-friendly interfaces to support manual adjustment of the retrieved objects and the generated scenes. This adjustment is essential if the single-vies view image can capture only partial information about the scene (e.g., details on sides not visible to the camera are unknown) or if the retrieved scene can not match the realistic semantics.
2) Robot-Object Interaction. A crucial prerequisite for ac- tion projection is reconstructing the objects and their rela- tionships with the operators (Liu et al., 2024c). For example, as shown in Figure 3, it is essential to determine the 3D pose at which the end effectors (e.g., hands) interact with the ob- jects (e.g,, the cup), thereby ensuring the projected action trajectory can accurately reflect the interaction under real- world physies. To achieve this goal, DexScale reconstructs the 3D meshs and the poses of objects from the sequences of images in the action trajectory video data. The sequence of object poses, along with the object models, is then jointly optimized with the poses of the target end effectors captured in the robot-action projections (as mentioned above), until they are properly aligned. This process fully automatic without any manual intervention. For each image, the goal of joint optimization is to refine the interaction between ob- jects and end effectors, accurately capturing realistic contact dynamics such as grasping forces and stability. DexScale further refines the trajectory by ensuring smoothness and plausibility for the sequence of actions. Figure 6 shows an example of the recovered 3D mesh.
End-Effector Pose
Action Videos
Action-Object Detection
Hand
Retarger
Hand
Gripper
4.2. Environment Simulation for Robot Learning
DexScale enables the conditional generation of a robot learn- ing environment based on the projected data. This environ- ment can simulate static and dynamic features by integrating scenes and action trajectories as described below.
Object
Refine
Object Mesh
Figure 3: The flowehart of action-trajectory projection.
Scene Simulation. DexScale supports the construction of different scenarios by leveraging scene projection, which focuses on providing static features such as shape, pose, appearance, object layout, and other background details. However, since our scene projection primarily relies on single-view or multi-view images, there is no guarantee that the scene is fully observable, nor that these visual observa- tions capture complete information about the scene. As a result, in addition to the projected information, DexScale en- ables the automatic construction of complete scene based the available data. To achieve this, we follow (Wang et al., 2024; Hua et al., 2024) and leverage large foundation models, such as GPT-4, to generate the scene configuration based on the projected information and task description.
Action- ceiio-Trrjeccory Projection. Given an egocentric video of humans manipulating objects (e.g., picking up a teapot and pouring tea into a paper cup), DexScale projects actions and objects from realistic human motion onto robot control signals within the simulated environment. We illustrate this process in Figure 3. This action trajectory captures dynamic information from human motion and can act as the seeding trajectories for the following skill scaling.
1) Robot-Action Projection. DexScale enables detecting movements of hand models from video data and retargeting these models to robotic end effectors, such as grippers and dexterous hands, thereby transforming human hand poses into those compatible with robotic devices. Within thin pro- cess, a significant challenge is the disparity in the Degrees of Freedom (DoF) between the hand model and the robotic end effectors. For instance, a robot hand might possess 6 DoFs, whereas the human hand model features over 20 DoF's To align the movements of the human and robotic hands effec- tively, it is essential to consolidate multiple DoFs from the human hand into single DoF of the robotic hand, ensur- ing minimal impact on model performance. In the case of retargeting to a gripper, that has only one DoF (either open or closed), we select two fingers to act as pivotal points for picking up and placing objects, and then retargeting these fingers' movements to the jaw movements of the gripper.
DexScale supports retrieving objects in the scene from the Objaverse-XI dataset (Deitke et al., 2023) or generating them based on language descriptions. Furthermore, if the generated scene does not align with practical requirements, DexScale provides a user-friendly interface that allows users to manually adjust and refine the generated scenarios.
A Action Trajectry Simulation. The objective of action simulation to generate continuous action trajectories for the robot, enabling it to execute the tasks effectively and achieve the desired results. To construct such action tra- jectories, DexScale primarily relies on the projected poses of end effector extracted from human video, or the robot
5

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
joint positions collected from tele-operation. For the pro- jected end-effector poses, DexScale utilizes generalized Inverse Kinematics (IK) algorithm to compute the corre- sponding joint configurations, ensuring precise achievement of the desired end-effector pose. To maintain a smooth trajectory, joint-space interpolation is applied across the mo- tion. In scenarios requiring obstacle avoidance, DexScale integrates widely used motion planning algorithms, such as RRTT-Connect (Kuffner& & Vallle 2000), to efficiently gen- erate collision-free trajectory. In applications where action projection is unavailable, DexScale supports the automatic design of reward and goal functions by consulting with large language models (Ma et al., 2024). This enables the RL and trajectory optimization algorithms to effectively learn robot control skills. To ensure that the simulated actions align with the projected action trajectories, DexSca le allows for the replay and automatic refinement of these trajectories via learning-based methods. Using these refined trajecto- ries, we can train an imitation policy goo (als), for robot control. However, directly deploying (als) in real-world applications is challenging due to the existing Sim2Real gap between the simulated and real-world environments. To address this issue, DexScale incorporates the following Sim2Real scaling techniques to bridge this gap.
The (als) Teo (als) V(s, a) 8 A where The denotes the optimal policy under the DR environment MS.
2) Semantic-Aware Domain Randomization (SA-DR). The goal of SA-DR is to generalize the embodied agents' skills from the source environment .M to different variations of DR environment M This approach allows the agent's skills to overcome the Sim2Real gap, provided that the gap falls within the range of simulated variations. To better handle these DRs (configured by {'), the agent must adapt the decisions and movements so that the optimal actions may vary before and after applying SA-DR to an environment M. The adapted policy π " can be represented as:
THss = aagg max J (ME. , (Tge)) Div(Tes [|
(I)
Jppt
Where J(ME",) :) denotes the optimality function under the environment MS and Div( (1=Te) indicate the di- vergence between policies Taet and ago. For example, in the continual learning setting, by setting Div(T l1=Te)
> >||05* - 11 the objective (1) can effectively represent the Elastic Weight Consolidation (EWC) objective. More im- portantly, as opposed to the action invariance in AI-DR, we must finetune the policy Too based on the updated objective J(M Agee) under the new environment MS after apply- ing SA-DR. In this setting, both RI. and motion planning algorithms can be utilized fine-tune the original policy and adapt it to the new environment.
4.3. Sim2Real Data Scaling
In a simulated environment, various factors can contribute to the Sim2Real gap (Section 3). undermining the effec- tiveness of generalizing the learned robot control policy to real-world applications. To address this gap, DexScale primarily focuses on establishing pipeline for automatic domain randomization and adaptation, as outlined below.
Automating DR. Each DR configuration & corresponds to control policy Tee. which captures skill for solving the task under the DR environment MS. By generating skills T using T , we can construct a Sim2Real dataset DDR. Based on this dataset, the imitation policy can effectively bridge the Sim2Real gap, provided that the experimented DR configurations and their combinations accurately cap- ture the underlying Sim2Real discrepancies. To achieve this goal, a critical prerequisite is being able to characterize the types, ranges, and styles of applied DRs. Within DexS- cale, our DR selection primarily focuses on the key factors for the Sim2Real gap (Section 3). For example, given an enviromment M, we must first identify which features are suitable for DR and determine the corresponding types of DRs, such as AI-DR and SA-DR. Additionally, for specific DR parameter &, we need to model the distribution of DR parameters, Pv(E)= E A (Chen et al., 2022a). While tradi- tional simulators often manually specify po((). DexScale seeks to automate the selection of DR features and mod- cling (c( conditioning on specific tasks. As illustrated in Figure 2, DexScale integrates large foundation model into the DR process by leveraging its ability to rank DR features based on the configuration of the simulated envi- ronment. For each DR configuration &, DexScale employs the ADR algorithm (OpenAl et al., 2019) to calculate and update (E) by incorporating feedback from the tne-ttned policies and their outcomes in the environment. Domain
Automatic Domain Randomization. Domain Randomiza- tion (DR) enables Sim2Real transfer by varying simulated environments (Chen et al., 2022b). We achieve DR by mod- ifying the original environment .M with configuration &, creating a new environment M In robotic learning en- vironments, better assess the effects of different DRs, we categorize these DRs based on their impact agents' actions in the following.
1) ActionnInvariant Domain Randomization (AI-DR). Within our DexScale, the goal of AI-DR is to prevent the em- bodied agent from overfitting to the simulated features that are irrelevant or ineffective in completing a task (c.g., the lighting condition manipulations). By removing the effects of these features and concentrating only on essential features for finishing a task, the agent experi- ences a less complex sim-to-real gap, and the learned skills have stronger transferabilityy For these DRs (configured by f), they have limited impact on the agents' decisions and movements. To better formulate this consistency, we assume the optimal actions remain uninfluenced before and after applying AI-DR to an environment M, so that
6

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Domain Cap: Background
Domain Gap: Distradtons
Human
Deno.
(Action Trajectory Generated DexSan (Size of Dalaset)
Sim2Real Data Scaling
Policy Deployment
Figure 4: Visualizing the input demonstration (top row). examples of the scaled data (middle rows), and the realistic deployment of the robot policy (bottom row). The Sim2Real domain gaps are background (left) and distractors (right).
randomization alters the size, shape, pose, and texture of objects during training, enabling the model to generalize its learned skills to wide variety of objects. As result, the model can effectively handle real-world objects, even if they differ significantly from those seen in simulation.
In the following experiment section, we demonstrate that various high-performing imitation prodels-includde ac- tion transformer policies (Zhao et al., 2023), diffusion poli- cies (Chi et al., 2023), and Lasion-Language-Aeee Actige--Actonnn (VLA) policies (Liu et al., 2024b)-can be trained and deployed to control real-world robots under different environments.
Domain Adaption (DA). To more effectively bridge the gap between simulation and reality, domain adaptation project the observations from the simulated and realistic environ- ment into a unified target space. DexScale supports various types of DAs, including 1) Object- Oriented Rep- resentations, where DexScale excludes background infor- mation from image observations from RGB cameras and focuses only on each object. To better capture the geometric and spatial properties of objects, DexScale can map objects into point clouds based on their masks within the simulated environment. Pose Affordance Representations, where DexScale selects key affordances (e.g., pose) from action trajectory, capturing the critical poses necessary complete the task. For example, when grasping an object, the affor- dance represents the gripper's pose when first contacts the object, enabling the robot to complete the task by reach- ing these poses using inverse kinematics. These prediction targets. derived from affordance representations, occur less frequently than full action trajectories, thereby reducing accumulated errors during deployment. With these meth- ods, DexScale can map DDR to the post- adaptatin dataset DDR-AR-
5. Experiments
The pipeline of DexScale is generic and agnostic to simula- tion platforms (Appendix A. shows our specifications). We evaluate the performance of Sim2Real deployment across various applications and assess the validity of Real2Sim projection in terms of action and object mapping by ad- dressing the following questions: 1) Generalizability: How effectively does DexScale bridge the Sim2Real gap between simulated environments and real-world applications? 2) Scalability: Can the control skills learned by DexScale be scaled across different models and embodiments?
5.1. Generalizability: Bridging the Sim2Real Gap
Experiment Setting. This experiment aims to quantify DexScale by how effectively it can overcome the Sim2Real gap. We first simulate baseline environment by projecting the real-world scene into a simulation (via our Real2Sim projection in Section 4.1) and then manually refining to ensure alignment with the realistic setting. Using this baseline environment, we can modify its parameters and intentionally introduce the Sim2Real gaps that frequently occur in practice based on analysis (see Table 1). We summarize such modifications in Appendix A.2. DexScale is then evaluated based on its effectiveness in overcoming these gaps and acquiring robust skills that can be seamlessly applied real-world applications. To gain deeper under- standing of DexScale, we conducted an ablation study by removing either the strategic Domain Adaptation (DA) Domain Randomization (DR) components from our DexS- cale dataset. This resulted in skill only dataset that records
4.4. Sim2Real Depolyment
DexScale adopts a data-driven approach achieve Sim2Real deployment. Specifically, the deployable policy trained by imitating the data trajectories in DDR+AR- Since DDR-AR captures diverse and rich action trajectories across various environments, the resulting control policy remains effective under the Sim2Real gap, provided the realistic envi- ronment lies within the support the dataset's distribution.

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Table 2: Success rates of imitation policies learned by different datasets under various Sim2Real gaps. For the first eight domain gaps, we employ the transformer-based policy (Zhao et al., 2023) to tackle grasping tasks, For the last two domain gaps, we use the diffusion-based policy (Chi et al., 2023) to address the open-box task.
Seit_
SKILIDA
0.20
0.E0
0.36
9-40
6.30
3.30
0:30
0.20
0.32
0.20
0.E2
D.ED
Success
Dataset
Skills Only
SRill>DDR
SkillsDDA
DexScale
sim
sine
Domain Cap
Sim.
Real.
Sim.
Real.
Sin.
Real.
Sam.
Real.
Light
64/100
0/10
75/100
4/10
74/100
1/10
73/10
4/10
0.32
0.33
0.50
Object Texture
49/100
1/10
81/100
5/10
75/100
3/10
83/100
6/10
0.4
0.42
TTable Texture
55/100
2/10
81/100
4/10
73/100
2/10
82/100
4/10
Background
67/100
1/10
83/100
3/10
71/100
1/10
82/100
3/10
Distractors
46/100
0/10
65/100
2/10
73/100
2/10
72/100
4/10
0.73
0:30
Camera Position
72/100
2/10
73/100
4/10
71/100
2/10
80/100
5/10
Object Grasping
Open Box
Po Orientation
Witer VeatrrPPouring
Camera Orientation
69/100
0/10
74/100
4/10
66/100
0/10
78/100
3/10
Table 3: Robot control performance for dif- ferent tasks under both realistic (upper) and simulated (lower) environments.
Camera Field of View
69/100
1/10
80/100
3/10
66/100
1/10
82/100
6/10
Object Fose
13/100
T/T0
33/100
2/TO"
427100
3/10
697100
3/10"
Object Shape
11/100
0/10
27/100
2/10
38/100
2/10
61/100
3/10
only the action trajectory for the target task in the simu- lated environment. Additionally, To prevent overfitting, we augment the skill dataset by adding random noise to skills.
that the success rates of pick-and-plase tasks in simulation and the real world are 0.62/0. 10, 0.63/0.30, 0.73/0.40, and 0. 79/0.566 respectively. This increasing trend highlights the critical role of automatic DR in improving Sim2Real transfer performance.
To evaluate the model's control performance in bridging the Sim2Real gap, we trained the control model using different methods and assessed its performance through 100 trials in the simulated environment and 10 trials in the real-world environment. The success rates are presented in Table 2. We found that the Sim2Real data scaling performed by DexS- cale significantly enhances the control model's ability to consistently bridge the domain gap across both simulated and real-world environments. Removing the designed DA and DR components leads to a notable decline in perfor- mance. To better understand the performance of DexScale, Figure 4 shows keyframes of scaled data and policy de- ployment in realistic scenarios (Appendix B.2 shows more examples). To address the domain gap caused by distracting items (e.g., umbrellas, scissors, or pens not being the target in the right columns of Figure 4), an interesting observation is that DexScale not only randomizes different objects but also backgrounds and textures in the scene. While textures are not directly related to distractors, they are effectively in bridging the Sim2Real gap, and DexScale discovers this relation. To better understand the difficulty of real-world performance, we include failure case study on the project page, highlighting the following common issues: 1) unob- servable grasping orientation, 2) incorrectly predicted grasp pose, 3) inaccurate grasp depth prediction, and 4) cross- chunk jitter.
5.2. Scalability across Diverse Tasks and Embodiments
Experiment Setting, A critical prerequisite for developing a Sim2Real simulator is its ability to scale across diverse robot control tasks and environments. Unlike most prior work, which primarily demonstrates realistic performance with single embodied robot, our approach emphasizes cross-embodiment evaluation to comprehensively assess Sim2Real performance. In this experiment, we evaluate our approach on four challenging tasks: 1) object grasping, which requires the robot to accurately detect objects and predict appropriate grasp poses; 2) paper box manipulation, involving precise control and planning to sequentially open all four flaps of box; 3) dual-arm table rearrangement, where the robot must reorient both fork and spoon to face the front of the plate and place them accurately around it; and 4) bottled water pouring, which involves grasping and reorienting a water bottle to pour water precisely into a paper cup. For these tasks, we train imitation models using various architectures, including the transformer- based pol- icy (Zhao et al., 2023) for grasping, the diffusion policy (Chi et al., 2023) for manipulation, and Vision-Language- Actiin (VLA) models (Liu et al., 2024b) for table rearrangement. Appendix A.3 reports the specific training configurations.
Figures 4, and 5 illustrate examples of action trajectorics for the tasks of object grasping, box manipulation, and table rearrangement. To demonstrate the scalability of DexScale, the control policies are deployed on different robots, includ- ing two single-arm robots and a dual-arm robot equipped with wrist-mounted cameras. Figure 3 presents the end- to-end performance across various tasks, showcasing both real-to-sim (from dask-deee data to the simulator) and sim-to-real (from simulation to real world deployment) tran-
Comparison with Handcrafted DR. In addition, we inves- tigate an alternative approach that relies on human expertise to manually select the type and scale of DR, instead of us- ing our Scale Following the empirical analysis in (Xie et al., 2024), which ranks the importance of various DR features, we apply the top-ranked features, 1) camera ori- entation, 2) table texture, and 3) distractors, to bridge the Sim2Real gap. We then evaluate model performance under each setting and report the results accordingly. We observe
8

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
sitions. By leveraging the scaled data generated by DexS- cale, models consistently outperform baseline meth- ods across different tasks, model architectures, and robot platforms, highlighting the scalability and effectiveness of DexScale.
adaptation for data scaling. These capabilities can be ef- ficiently expanded to accommodate wide range of tasks, learning models, and robotic platforms. In future work, aim to extend DexScale to more complex robots (e.g.. hu- manoid robots) and long-term tasks (e.g., cooking a meal).
Iuman Demo.
Acknowledgments
Trajecterics Gencrated DexSim)
This work is supported in part by Science and Technol- ogy Major Program under grant KJZDD204000310400000 Guangdong-SSenzzher Joint Research Fund under grant 2023A1515110617 Guangdong Basic and Applied Basic Research Foundation under grant 2024A 111211115012003 and Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B 12331122001 211200001).
Sim2 Data Scaling
......
Deployment
Impact Statement
The broader impact of this work lies in its potential to sig- nificantly advance the field of robotics by addressing one of its most pressing challenges: the gap between simulated and real-world environments.
Domo
(Action Trajectoriss Generated DeaSim))
Sien2oel Data Scating
From ethical perspective, DexScale promotes inclu- sive and equitable innovation by making the development of robotic systems more accessible, even those with limited resources. However, with any technology that advances automation, there are potential societal consequences to con- sider. The widespread adoption of robots in tasks such as manufacturing, service, or even domestic work could lead to significant shifts in labor markets, raising concerns about job displacement and economic inequality.
Daployment
In the long term, DexScale's ability to scale robotic intelli- gence across diverse applications could have transformative societal implications. Robots trained using DexScale could deployed in healthcare, disaster response, education, and other critical areas, amplifying their positive impact on soci- cty. However, ensuring the ethical use of such robots will require careful consideration of privacy, safety, and account- ability. Future research should also explore ways to mitigate potential biases in simulated datasets to ensure fairness and reliability in real-world deployments.
Deno.
(Action Tmieestoiis Generated by DexSim)
Real Data Seating
Policy Deployment
References
Figure 5: Visualizing the human demonstration, scaled tra- jectories and realistic robot action for the task of opening box, table rearrangement and pouring water.
Agarwal, A., Kumar, A., Malik, J., and Pathak, D. Legged locomotion in challenging terrains using egocentric vi- sion. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205, pp. 403-415. PMLR, 2022.
6. Conclusion
We introduce DexScale, a data engine designed to generate simulated robot action trajectories for learning deployable control policies by mapping realistic observations into sim- ulation and utilizing automated domain randomization and
Batra, D.,, Chang, A. X., Chernova, S., Davison, A. J., Deng, J., Koltun, V., Levine, S., Malik, J., Mordatch, I., Mot- taghi, R., et Rearrangement: A challenge for embodied

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
ai. arXiv preprini arXiv:2011.0197: 2020.
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L.,, Ehsani, K., Salvador, J., Han, W., Kolve, E.,, Kembhavi, A., and Mottaghi, R. Procthor: Large-scale embodied Al using procedural generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnam, K., Hausman, K., Herzog, A., Hsu, J., al. RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems, RSS, 2023.
Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S. Y., VanderBilt, E.,, Kembhavi, A., Vondrick, C., Gkioxari, G., Ehsani, K., Schmidt, L.,, and Farhadi, A. Objaverse-xl: A universe of 10m+ 3d objects. In Advances in Neural Information Processing Systems, NeurIPS, 2023.
Chen, I.-M and Burdick, J. W. Finding antipodal point grasps on irregularly shaped objects. IEEE transactions on Robotics and Automation, 9(4):507-512, 1993.
Chen, S., Xu, H., Li, R., Liu, G., Fu, C., and Liu, S. SIRA-PCR: sim-to-real adaptation for 3d point cloud registration. In JEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, Oc- tober 1-6, 2023, pp. 14348-14359, IEEE, 2023a. doi: 10.1 00.10000CCV5110002223..111
D' Souza, A., Vijayakumar, S., and Schaal, S. Learning in- verse kinematics. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, volume 1, pp. 298-303, 2001.
Duan, J., Yu, S., Tan, H. L.,, Zhu, H., and Tan, C. A sur- vey of embodied AI: from simulators research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-244 2022.
Chen, X., Hu, J., Jin, C., Li, L., and Wang, L. Understand- ing domain randomization for sim-to-real transfer. In International Conference on Learning Representations, ICLR. OpenReview.net, 2022a.
Ebert, F., Yang, Y., Schmeckpeper, K., Bucher, B., Geor- gakis, G., Danilidiis, K., Finn, C., and Levine, S. Bridge data: Boosting generalization of robotic skills with cross- domain datasets. In Robotics: Science and Systems, RSS, 2022.
Chen, X., Hu, J., Jin, C., Li, L., and Wang. L. Understand- ing domain randomization for sim-to-real transfer. In The Tenth International Conference on Learning Representa- fions, ICLR, 2022b.
Fang, H., Wang, C., Fang, H., Gou, M., Liu, J., Yan, H.,, Liu, W., Xie, Y., and Lu, C. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 39(5):3929-333)) -945,, 2023.
Chen, Z. Q., Kiami, C., Gupta, A., and Kumar, V. Genaug: Retargeting behaviors to unseen situations via generative augmentation. In Robotics: Science and Systems, RSS, 2023b.
Florence, P. R., Manuelli, L., and Tedrake, R. Self- supervised correspondence in visuomotor policy learning. IEEE Robotics Autom. Lett., 5(2):492 409, 2020.
Cheng, X., Shi, K., Agarwal, A., and Pathak, D. Extreme parkour with legged robots. In IEEE International Con- ference on Robotics and Automation, ICRA, pp. 11443- 11450. IEFE, 2024.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL: datasets for deep data-driven reinforcement learning. CoRR, 0tss/2004.02119, 2020.
Chi, C., Feng, S., Du, Y., Xu, Z.,, Cousineau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, RSS, 2023.
Gan, C., Zhou, S., Schwartz, J., Alter, S., Bhandwaldar, A., Gutfreund, D., Yamins, D. L. K., DiCarlo, J. J., McDer- mott, J. H., Torralba, A., and Tenenbaum, J. B. The three- dworld transport challenge: A visually guided task-and- motion planning benchmark towards physically realistic embodied AI. In International Conference on Robotics and Automation, ICRA, pp. 8847-8854, 2022.
Coumans, E. and Bai, Y. Pybullet, python module for physics simulation for games, robotics and machine leamn- ing. http:: http///yyblll/ let. org, 2016-2021.
Dai, T., Wong, J., Jiang, Y., Wang, C., Gokmen, C., Zhang, R., Wu, J., and Fei-Fei, L. Acde: Automated creation of digital cousins for robust policy learning, arXiv preprint arXiv:24 10.0:2410000408. 2024.
Hua, P., Liu, M., Macaluso, A., Lin, Y. Zhang, W., Xu, H., and Wang, I.. Gensim2: Scaling robot data generation with multi-modal and reasoning 1lms. arXiv preprint arXhv:2410.03645, 2024.
Dasari, S., Ebert, F., Tian, S., Nair, S., Bucher, B., Schmeck- peper, K., Singh, S., Levine, S., and Finn, C. Robonet: Large-scale multi-robot learning. In Anmual Conference on Robot Learning. CoRL, volume 100, pp. 885-897 PMLR, 2019.
James, S., Ma, Z., Arrojo, D. R., and Davison, A. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):30193300 3026, 2020.
10

---

DexScale:
Automating Data Scaling for Sim2Real Generalizable Robot Skills
Jiang, Z., Zhu, Y., Svetlik, M., Fang, K., and Zhu, Y. Syn- ergies between affordance and geometry: 6-dof grasp detection via implicit representations. In Robotics: Sci- ence and Systems. RSS, 2021.
Tobos-Tsunekawa, K. and Harada, T. Point cloud based reinforcement learning for sim-to-real and partial observ- ability in visual navigation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, pP. 5871 2020.
Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L.,, and 7hu, Y. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imi- tation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025.
Ma, Y. J., Liang, W., Wang. G., Huang, D., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-leve reward design via coding large lan- guage models. In The Twelfih International Conference on Learning Representatinss ICLR, 2024.
Josifovski, J., Auddy, S., Malmir, M., Piater, J. H., Knoll, A., and Navarro- N. Continual domain randomiza- tion. IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, pp. 4965-4972, 2024.
Makoviychuk, V., Wawrzyniak, L.,, Guo, Y., Lu, M., Storey, K., Macklin, M.. Hoeller, D., Rudin, N., Allshire, A.. Handa, A., and State, G. Isaac gym: High performance GPU based physics simulation for robot learning. In Vanschoren, J. and Yeung, S. (eds.), Neural Information Processing Systems Track on Datasets and Benchmarks, NeurIPS Datasets and Benchmarks, 2021.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Bal- akrishna, A., Nair, S., Rafailov, R., Foster, E. P., Lam, G., Sanketi, P., Vuong, Q.. Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. Openvla: An open-source vision- waaoo--lanagggatttiinin model. CoRR, 0ew22466092466 2024.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Martín Martin, R. What matters in learning from of- fline human demonstrations for robot manipulation. In Faust, A., Hsu, D., and Neumann, G. (eds.), Conference on Robot Learning, CoRL, volume 164, pp. 1678-1690, 2021.
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 171.05571 2017.
Kuffner, J. J. and Vallee S. M. Rrt-connect: An efficient approach single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Pro- ceedings ( (Cat. No. 00CH37065), volume 2. pp. 995-1001. IEFE, 2000.
Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y.. Fan, L., Zhu, Y., and Fox, D. Mimicgen: A data generation system for scalable robot learning using hu- man demonstrations. In Annual Conference on Robot Learning, CoRI., 2023.
Manuelli, L., Gao, W., Florence, P. R., and Tedrake, R. KPAM: keypoint affordances for category-level robotic manipulation. In The International Symposium of Robotics Research, volume 20, pp. 132-157. Springer, 2019.
Kumar, A., Fu, Z., Pathak, D., and Malik, RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, JMLR, 17:39:1-39:44 2016.
Mehta, B., Diaz, M., Golemo, F., Pal, C. J., and Paull, L. Active domain randomization. Conference on Robot Learning, CoRL, volume 100, pp- 1162-11766 2019.
Liu, J., Savva, M., and Mahdavi-Amiri, A. Survey modeling of articulated objects. arXiv preprint arXiv:2403. 14937, 2024a.
Memmel, M., Wagenmaker, A., Zhu, C., Fox, D., and Gupta, A. ASID: active exploration for system identification in robotic manipulation. In International Conference on Learning Representations, ICLR, 2024.
Liu, S., Wu, I.., I., B., Tan, H., Chen, H., Wang, 7., Xu, K., Su, H., and Zhu, J. RDT- IB: a diffusion foundation model for bimanual manipulation. CoRR, abs/2410.07864, 2024b.
Mo, K., Guibas, L. J., Mukadam, M., Gupta, A., and Tul- siani, S. Where2act: From pixels to actions for articulated 3d objects. In IEEE/CVF International Conference on Computer Vision, ICCV, pp. 6793667988880 6803. IEEE, 2021.
Liu, Y., Long, X., Yang. Z., Liu, Y., Habermann, M., Theobalt, C., Ma. Y., and Wang, W. Easyhoi: Unleashing the power of large models for reconstructing hand-object interactions in the wild. arXiv preprint arX20 2024c.
Mousavian, A., Eppner, C., and Fox, D. 6-dof graspnet: Variational grasp generation for object manipulation. In IEEE/CVF International Conference on Computer Vision, ICCV, pp. 2901-2910. IEFE, 2019.

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Murali, A., Mousavian, A., Eppner, C., Fishman, A., and
Qin, Z., Fang, K., Zhu, Y., Fei-Fei, L.., and Savarese, S. KETO: learning keypoint representations for tool manip- ulation. In IEEE International Conference Robotics and Automation, ICRA, pp. 7278-7285. IEEE, 2020.
Fox, D. Cabinet: Scaling neural collision detection for object rearrangement with procedural scene generation. In IEEE International Conference on Robotics and Au- tomation, ICRA, pp. 1874. IEFF, 2023.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T.,
Muratore, F., Eilers, C., Gienger, M., and Peters, J. Data- efficient domain randomization with bayesian optimiza- tion. IEEE Robotics and Automation Letters, 6(2):911- 918, 2021a.
Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007 00714, 2024.
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 JEEF/CVF Conference on Com- puter Vision and Pattern Recognition,Cvi pp. 10737- 10746, 2020.
Muratore, F., Gruner, T., Wiese, F., Belousov, B., Gienger, M., and Peters, J. Neural posterior domain randomization. In Conference on Robot Learning, CoRL, volume 164, pp- 1532-1542, 2021b.
Nasiriany, S., Maddukuri, A., Zhang, I.,, Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. Robocasa: Large- scale simulation of everyday tasks for generalist robots. arXiv preprint wwXiv:2446.025222 2024.
Sundermeyer, M., Mousavian, A., Triebel, R., and Fox, D. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. IEEE International Conference on Robotics and Automation, ICRA, pp. 13438-13444. IEEE, 2021.
O' Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al. Open x-embodiment: Robotic learning datasets and RT-X models: Open x-embodiment collab- oration. In IEEE International Conference on Robotics and Automation, ICRA, pp. 1803. IEFE, 2024.
Sutton, R. S. Reinforcement learning: An introduction. Bradford Book, 2018.
Tao, S., Xiang, F,, Shukla, A., Qin, Y., Hinrichsen, X., Yuan, X., Bao, C., Lin, X., Liu, Y., Chan, T.-k., et al. Maniskil13: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.00425, 2024.
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plap- pert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. Solving rubik'ss 's cube with robot hand. CoRR, abs/1910.07113 13, 2019.
Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C.,, Luo, J., Tan, Y. L.,, Chen, L. Y., Sanketi, P., Vuong, Q., Xiao, T., Sadigh, D., Finn, C., and Levine, S. Octo: An open- source generalist robot policy. CoRR, 122/23 2024.
Paolo, G., Gonzalez-Billandoo.. J., and Kégl, B. Position: A call for embodied AI. In International Conference Machine Learning, ICML, 2024.
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T., Partsey. R., Desai, R., Clegg, A., Hlavac, M., Min, S. Y., Vondrus, V., Gervet, T., Berges, V., Turner, J. M., Maksymets, O., Kira, Z., Kalakrishnan, M., Malik, J., Chaplot, D. S., Jain, U., Batra, D., Rai, A., and Mot- taghi, R. Habitat 3.0: A co-habitat for humans, avatars, and robots. In International Conference on Learning Representations (ICLR), 2024.
Tiboni, G., Arndt, K., and Kyrki, V. DROPO: sim-to-real transfer with offline domain randomization. Robotics and Autonomous Systems, 166:104432, 2023.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics en- gine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, pp. 5026-5033. IEEE, 2012.
Qi, H., Kumar, A., Calandra, R., Ma, Y., and Malik, J. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, CoRI, volume 205, pp. 1722-1732, 2022.
Walke, H. R., Black, K., Zhao, T. Z., Vuong, Q., Zheng, C., Hansen- -Estruc,,,, P., He, A. W. Myers, V., Kim, M. J., Du, M., Lee, A., Fang, K., Finn, C., and Levine, S. Bridgedata V2: A dataset for robot learning at scale. In Conference on Robot Learning, CoRL, volume 229, pp. 1723-1736. 2023.
Qin, Y., Huang, B., Yin, Z., Su, H., and Wang, X. Dexpoint: Generalizable point cloud reinforcement learning for sim- to-real dexterous manipulation. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205, pp. 594
Wang. Y., Xian, Z., Chen, F., Wang, T., Wang, Y., Fragki-
adaki, K., Erickson, Z., Held, D., and Gan, C. Robo- gen: Towards unleashing infinite data for automated robot
594-605. 605. PMLR, 2022.
12

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
learning via generative simulation. In International Con- ference Machine Learning, ICMI., 2024.
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, E, Wu, Wohlhart, P., Welker, S., Wahid, A., al. RT-2: vision-language-actionn models transfer web knowledge to robotic control. In Tan, J., Toussaint, M., and Darvish, K. (eds.). Conference on Robot Learning, CoRl, volume 229, pp. 2163-2183, 2023.
Wu, C., Chen, J., Cao, Q., Zhang, J., Tai, Y., Sun, L.,, and Jia, K. Grasp proposal networks: An end-to-end solution for visual learning of robotic grasps. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
Xiang. F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., Yi, L.., Chang, A. X., Guibas, L. J., and Su, H. SAPIEN: A simulated part- based interactive environment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. pp. 11094-11104, 2020.
Xiang, J., Lv, Z., Xu, S., Deng, Y., Wang, R., Zhang, B., Chen, D., Tong, X., and Yang, J. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01500 2024.
Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M.. Liu, C., and Yuan, L. Florence-2: Advancing unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4818-4829, 2024.
Xie, A., Lee, L., Xiao, T., and Finn, C. Decomposing the generalization gap in imitation learning for visual robotic manipulation. In IEEE International Confer- Robotics and Automation, ICRA, pp. 3153-3160. IEEE, 2024.
Xu, G., Wang. X., Ding, X., and Yang, X. Iterative geometry encoding volume for stereo matching. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 21919-21922 IEEE, 2023.
Yang, J., Cao, Z.aa., Deng, C., Antonova, R., Song, S., and diffusion policy for generalizable and data efficient learning. In 8th Annual Conference on Robot Learning, 2024.
Bohg. J. Equibot:
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., and Fox, D. Robopoint: A vision-language model for spatial affordance prediction for robotics. CoRR, abs/2406.1070 2024.
Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Gold- berg, K., and Abbeel, P. Deep imitation learning for complex manipulation tasks from virtual reality teleoper- ation. In IEEE International Conference on Robotics and Automation, ICRA, pp. 1-8. IEBE, 2018
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-g ine-grained bimanual manipulation with low-cost hardware, In Robotics: Science and Systems, RSS, 2023.
13

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
A. Implementation Details
A.I. Specification of DexScale
DexScale is scalable, automated data engine designed to bridge the Sim2Real gap in robot skill learning by combining physically based rendering (PBR), physies simulation and high performance murtii-h system. Its architecture optimized for generating large-scale, diverse, and realistic robot learning environments and datasets, while ensuring computational efficiency. Below, detail its specifications:
System Architecture.
DexScale is built on a modular framework with three interconnected subsystems:
Coysically-Based Rendering Engine: Combines ray tracing and rasterization techniques to synthesize photorealistic visuals with dynamic lighting effects and physically accurate material properties, such surface roughness, metallic reflectance, and albedo maps.
Physics Simulation Engine: Employs a rigid-body dynamics solver with support for frictional contacts and articulation modeling (e.g.. grippers, robotic arms)
Scene Composition: Compose complex scenes with customizable objects, lighting, and backgrounds. Users define object dynamics properties, such mass, friction, and elasticity, material properties to match real-world conditions.
DexScale's 's modular architecture is engine-agnostic, enabling seamless integration with arbitrary rendering physics engines as backends. This design ensures adaptability to diverse use cases while decoupling core functionality from third-party dependencies.
Robotic Simulation. Our robot simulation module provides a unified framework for modeling, controlling, and training robotic systems through three core components:
Robot Abstract Models: Predefines the parameterized templates for common robotic platforms, such as Manipulators ((e.g.. 6-DoF industrial arm), Dexterous Hands and Humanoids with unified URDF description and configurable kinematic chains, inertial properties, Joint limits, etc. It also supports the customization of robot morphologies through a modular assembly of actuators and sensors.
Control System: Provides forward/inverse kinematics solvers with singularity handling and motion generation capacities with trajectory interpolation and collision-aware planners. It has user-friendly interface to access robot proprioception and execute the control signal in both joint space and task space (eg, euclidean space)
Robot Learning Environment: Adopts OpenAl gym-compatible API with standardized method (e.g., reset, step), and support domain randomization for both dynamics and visuals properties of the objects and scene. It includes task suites (c.g.. manipulation, locomotion) equipped with configurable callback and reward functions, enabling the implementation of customized features tailored for both imitation learning and reinforcement learning.
Data Generation. Our data generation module enables large-scale synthesis of labeled simulation datasets through pipeline combining assets generation, geometric processing, and automated annotation. The architecture comprises four core subsystems:
Scene Construction: Combines procedural generation, leveraging parameterized templates for randomized object place- ment, lighting configurations, and camera viewpoints, with rule-based scene assembly (c.g., clutter or ordered ar- rangements) and collision-free guarantees. It integrates 3D AIGC for asset synthesis via state-of-the-art models like TRELLIS (Xiang al., 2024), while also supporting retrieval of 3D assets from large-scale, multi-label database.
Mesh Processing:
Provides an automated pipeline to ensure 3D geometry assets are simulation-ready. This pipeline includes UV mapping, geometric processing (e.g., remeshing, hole filling, and simplification), and convex decomposition for efficient collision detection.
Domain Randomization: Provides interface to adjust or change the parameters of the domain randoization factors described section 4.3.
14

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Annotation Computation: Supports diverse annotation types, including instance/semantic segmentation masks, 6D object poses, keypoints, and physics-based ground truth such as contact vectors interaction points. It also provides interfaces to export datasets in standard formats like COCO and HDF5.
A.2. The Details of Sim2Real Gap
Light. We adjust the lighting conditions by varying the distance between the light source and the scene of the given tasks. Specifically, 3-meter difference is introduced between the simulated and real-world environments, resulting in darker lighting condition in the real-world setting.
Object Texture. We primarily focus on the material properties and appearance, including the color, of each object. Specifically, in the real-world environment, the objects covered with tissues, while in the simulated environment, the color and texture are tailored to match the object type (e.g., a banana has smooth yellow surface).
Table Texture. We primarily focus the differences in table surface patterns between the simulated and real-world environments. Specifically, in the real-world environment, the table is covered with dark paper, while in the simulated environment, table surface in light grey table surface is used.
Background Texture. Since the camera's view primarily captures the floor where the table is placed, we focus on the differences in floor surface patterns between the simulated and real-world environments. Specifically, in the real-world environment, the floor is covered with light grey surface, while in the simulated environment, wooden surface is used.
Distractors. The distractors consist of objects that are not intended to be grasped by the gripper. In the real-world environment, these objects include paper cup, folding umbrella, a pull-tab can, scissors, and spray bottle. However, in the simulated environment, these objects are intentionally removed.
Camera Position. We adjust the camera position in the simulated environment. Specifically, the camera is placed cm higher and em to the right compared the position of the camera in the real-world environment.
Camera Orientation. We adjust the camera orientation in the simulated environment. The camera is rotated 5 degrees counterclockwise around both the X and Z axes the simulated environment.
Camera Field of View. We adjust the camera orientation by modifying its parameters. Specifically, the focal length is set to 1.15 times the actual focal length to achieve the desired adjustments.
Object Pose. We adjust the object pose in the simulated environment by applying rotation of up 15 degrees and shifting their x-y positions by up to 5 centimeters.
Object Shape. We rescale the width, length, and height of the object in the simulated environment, with the scaling ranging from 0.9 to 1.1 times the object's size in the realistic environment.
A.3. Model Training Configurations
1. OBJECT GRASPING
Model Structure. We adopt the HumanPlus HIT model configuration and outline the following details:
The model employs ResNet18 backbones to process visual inputs and output visual latents. Action data is projected into action latents using an MLP. The visual and action latents are then fed into 6 BERT-style transformer decoders for further processing.
Model Architecture:
Number of Parameters: The total number of parameters in the model is 30.4M.
Activation Functions: GELU used the activation function throughout the network.
Dataset Details. The dataset consists of 2000 trajectories for each feature and each setting (gap, domain randomization, domain adaptation, domain randomization and adaptation), with each trajectory containing 50 steps. Each trajectory includes 21-dimensional action data (comprising 6-dim joint angles, 7-dim end-effector pose under the robot base frame, 1-dim gripper open state, and 7-dim end-effector pose under the camera frame) and two 512 x 640 rectified images. The dataset
15

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
split into 99% training and 1% validation. Data preprocessing involves normalization for action data, ImageNet-statiities normalization for images, and image rectification. During training, color-jitter data augmentation is applied to the images.
Training Hyperparameters. The model is trained with a batch size of 12 for 200, 000 epochs using the A optimizer with learning rate of 1 x 10 The training objective is guided by the mean squared error (MSE) loss for actions and a 1-dimensional error for image latents.
Training Environment. Training was conducted on 4 NVIDIA A800 GPUs, with each policy trained for 36 hours. PyTorch version 2.0.1 was used as the deep learning framework.
A.3.2. OPEN Box
Model Structure. We follow the network architecture of EquiBot (Yang et al., 2024), with several modifications to enhance sim-to-real transfer. Our model is based on the Eqqiivaaqi Diffusion Policy (Yang et al., 2024), with the following details:
Network Architecture: The original EquiBot processes entire scene point clouds as the observed input environment, with a sampling point number of 1024 or 2048 depending on the task type. In our modified version, we use the cropped point clouds of manipulated objects as input and always set the sampling point number to 1024.
Mask Detection: The 2D mask of manipulated objects is obtained automatically in simulation using the DexScale simulator. During real-world inference, the mask is detected using vision foundation models, such as Florence2 (Xiao al., 2024) and SAM2 (Ravi al., 2024).
Prediction and Observation Horizon: The prediction horizon is set to 72, and the observation horizon is Unlike the dynamic style used in the original EquiBot predditioo-oodduutionnoevvvaorrr we observe once and execute all predicted actions at once.
Stereo Vision: Our model uses binocular camera to capture left and right images simultaneously. The left image branch lifts 2D pixels into 3D using stereo matching algorithm (Xu et al., 2023). This approach significantly reduces domain gaps caused by hardware differences between real and simulated environments.
Number of Parameters: The total parameter size is 1 111.43 MB.
Dataset Details. The dataset consists of 50 demonstrations in the training set, with each demonstration containing 75 timesteps. The dataset is split into 50 demonstrations for training and 10 demonstrations for testing. Additionally, 100 evaluation trials are conducted in the simulator to assess performance.
Training Hyperparameters. The training process uses a batch size of and runs for 150,000 epochs. The model optimized using the training objective guided by the Chamfer Distance loss function for point clouds and the Mean Squared Error (MSE) loss for action predictions.
Adam optimizer with cosine learning rate scheduler, starting with a learning rate of 5 x 10 The
Training Environment. Training was performed on single NVIDIA A100 GPU for a total of 48 hours. Py PyTorch version 2.0.1 was used as the deep learning framework.
A.3.3. TABLEWARE REARRANGEMENT
Model Structure. We follow the original Roboties Diffusion Transfromer (RDT) model configuration (Liu et al., 2024b) and make the following modifications:
Vision Encoder: Replace SigLip with DINOv2 (base) with registers the vision encoder.
Image Conditions: Use one stereo camera along with two wrist-mounted RGB cameras, resulting in the following image conditions: 2 RGB images, 1 disparity image, and 2 additional RGB images.
Mask Predictor Module: Add small mask predictor module consisting 3 convolutional layers to predict the mask for each image. The updated image condition is then computed using the following equation:
img cond new = img cond + mask mask embed
16

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Transformer Block Settings: The transformer block is configured with hidden size of 512, depth of layers, and 16 attention heads.
Model Size: The total number of parameters is 116.66 MB.
Dataset Details. The dataset used in this work consists of multiple trajectories designed to capture diverse scenarios for training and evaluation. Specifically: 1) Number of Trajectories: The dataset includes 200 trajectories, each consisting of 200 steps. 2) Data Preprocessiing: The data is preprocessed using the same pipeline described in the original RDT setup.
Training Hyperparameters. The training process utilizes batch size of 8 and runs for 40,000 iterations. The model is optimized using the Adam optimizer with cosine learning rate scheduler, starting with a learning rate of 1 x 10-4. The training objective is guided by the Mean Squared Error (MSE) loss function.
Training Environment. The training was conducted on single NVIDIA A100 GPU using PyTorch version 2.0.1 the deep learning framework.
A.3.4. HARDWARE DETAILS FOR DIFFERENT TASKS
Table 4: Robot Specifications for Different Tasks
Task
Robot Name
DOF
Maximum Reach (mm)
Maximum Payload (kg)
Object Grasping
Rokae SR3
6
705
Open Box
AUBO 15
6
886.5
Tableware Rearrangement
WidowX 250 S
7
650
0.25
B. More Experiment Results
B.1. Human-Object Interaction Example
Figure 6 visualize the input image alongside the output hand-object interactions as a 3D mesh, which can be directly projected into simulated environment. To enhance the visualization of the 3D mesh details, we present observations from six different angles.
B.2. Examples of using DexScale
17

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
input image
3D Hand-Object Interaction
input image
3D Hand-Object Interaction
input image
3D Hand-Object Interaction
Figure 6: Visualizing the input image and the output hand-object interactions in 3D mesh.
Domain Gap: Camera Orientation
Domain Gap: Camera Field View
Duman
Demo.
(Action Trajectory Generated by DexSim (Size of Dataset)
(Action Trajectory Generated by DexSim x (Size of Dataset)
Sim2Rteal Dato Scaling
Deployment
Figure Visualizing the input demonstration (top row), examples of the scaled data (middle rows), and the realistic deployment of the robot policy (bottom row). The Sim2Real domain gaps camera orientation (left) and field of views (right).
18

---

DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Skills
Donain Gap: Light
Domain Gap: Object Texture
Demo.
(Action Trajectory Generated by DexSim (Size of Dataset)
(Action Trijectory Generated by DexSim x (Size of Dataset)
Sim2Real Data Scaling
Poliyy Deployment
Figure 8: Visualizing the input demonstration (top row), examples of the scaled data (middle rows), and the realistic deployment of the robot policy (bottom row). The Sim2Real domain gaps are lighting (left) and object texture (right).
Domain
Gag: Background
Human Demo.
(Action Trajectory Generated by DexSim (Size of Dataset)
(Action Trajectory Generated by DexSim x (Size of Dataset)
Sim2Real Data Scaling
Policy Deployment
Figure Visualizing the input demonstration (top row), examples of the scaled data (middle rows), and the realistic deployment of the robot policy (bottom row). The Sim2Real domain gaps are table texture (left) and background (right).
Domain Gap: Distractoes
Domain Gap: Camera Position
Deno.
(Action Trajectory Generated by DexSim Dataset)
(Action Trajectory Generated DexSim Dataset)
Sim2Real Dota Sealing
Policyy Deployment
Figure 10: Visualizing the input demonstration (top row), examples of the scaled data (middle rows), and the realistic deployment of the robot policy (bottom row). The Sim2Real domain gaps are distractors (left) and camera position (right).
19


Reviews:
[{"review_id": "Pyp9HKJAxu", "reviewer": "Reviewer_Fjdp", "summary": "This paper proposes DexSim, a pipeline for automating the learning of manipulation skills from human videos. Given an egocentric video, it first extracts human hand, wrist, and object pose trajectories. It then retargets the hand trajectory to a robot gripper and finds an object mesh closest to the object in the video. After that, it builds a simulation environment with objects, layouts, and scenes, and uses an LLM to randomize scene configurations. In this randomized environment, it then generates trajectories as the ground-truth demonstrations and learns a policy for deployment in the real world.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we sincerely appreciate your constructive feedback. We hope that the following response can address your concerns:\n\n> 1 \"*Only a few supplementary videos show different robots, there is no quantitative comparison in the main text or supplementary material.*\"\n\n**Response.** \nTable 4 in the appendix summarizes the various embodiments used to demonstrate their performance in real-world scenarios.\nTo better support our claim, we experiment with an additional robot for grasping and manipulating (e.g., grasping a bottle and pouring the liquid into a cup). This experiment is conducted on a dual-armed Cobot Magic robot, i.e., mobile ALOHA (Fu et al., 2024). \nThe success rates are as follows:\n\n| Setting | Skill | Skill+DR | Skill+DA | DexSim |\n| -------- | ------- | -------- | ------- | ------- |\n| Real-World | 1/10 | 2/10 | 3/10 | 7/10 |\n| Simulation | 30/100| 54/100 | 71/100 | 86/100 |\n\nFor the relevant videos for different embodiments, please check section 1.1 in our [anonymous link](https://anonymous.4open.science/w/dexscale/)\n\nMoreover, previous studies on the Sim2Real robotic simulation data engine primarily focus on a single type of embodiment. In contrast, our real-world experiment is conducted on a significantly larger scale.\n\n\n> 2. \"*How does automated DR compare with hand-crafted DR?*\"\n\nHandcrafted DR refers to selecting the type and scale of DR based on human expertise. We refer to the empirical analysis by Xie et al. (2024), which ranks the importance of various DR features. To evaluate model performance, we apply the top 1, 2, and 3 most important DR features to bridge the Sim2Real gap and report the results as follows. \n\n| Setting | Camera Orientation | Camera Orientation + Table Texture | Camera Orientation + Table Texture + Distractors | DexSim |\n| -------- | ------- | -------- | ------- | ------- |\n| Real-World | 1/10 | 3/10 | 4/10 | 7/10 |\n| Simulation | 62/100| 63/100 | 73/100 | 86/100 |\n\nFor the detailed videos, please check section 1.2 in our [anonymous link](https://anonymous.4open.science/w/dexscale/) for detailed results.\n\n(Xie et al., 2024) Xie, Annie, et al. \"Decomposing the generalization gap in imitation learning for visual robotic manipulation.\" ICRA 2024.\n\n> 3. \"*What is the contribution of individual components (e.g., AI-DR vs. SA-DR)? Similarly, what are the performance gains from object-oriented representations and pose-affordance representations in DA?*\"\n\nRegarding the comparison between AI-DR and SA-DR, their configurations are automatically determined by the LLM, and as such, there is no guarantee that they will always be included in the DR variants generated by DexSim. This makes it challenging to isolate and evaluate their individual effectiveness in empirical studies. Therefore, we choose to study DR as an integrated component in our analysis.\n\nThe same argument applies to object-oriented representations and pose-affordance representations in DA. There is no guarantee they will always be included in DexSim.\n\n\n> 4. \"*Figure 2 mentions image/teleoperation as data sources and RL for policy fine-tuning, but these are not used in the experiments.*\"\n\nWe apologize for the misunderstanding. Our Cobot Magic robot is equipped for teleoperation. Since the teleoperation signals are directly compatible with the robot, retargeting is unnecessary. Instead, we can directly transfer these control signals to the simulator to control the simulated robot within the data engine. \n\nRegarding RL fine-tuning, DexSim supports the automatic design of reward and goal functions by leveraging large language models (see lines 270–273, right column). Given these reward functions, applying RL algorithms to DexSim is straightforward.\nWell-established algorithms, such as PPO and SAC, can be seamlessly integrated into our engine to generate skills. \n\n> 5. \"*The reorientation task appears to be a simple pick-and-place...*\"\n\nThe re-arrangement task is significantly more challenging than simple pick-and-place operations for the following reasons:\n1. The robot arm must reorient the fork and spoon to ensure they face the front of the plate and position them correctly around it.\n2. To achieve smooth execution, both arms must work in coordination, ensuring the fork and spoon are placed in their correct positions simultaneously.\n\n> 6. \"*What are the performance contributions of each module?*\"\n\nIn section 5.1, we have conducted an ablation study by removing either the strategic Domain Adaptation (DA) or Domain Randomization (DR) components from our DexSim dataset. We have also added an experiment for comparing different DR methods (either hand-crafted or automated) as mentioned above.\n\n> 7. \"*What are the primary failure modes observed in real-world deployments?*\"\n\nWe have included a fail case study in the revised draft. Please see examples in section 1.3 in our [anonymous link](https://anonymous.4open.science/w/dexscale/) for detailed results.", "claims_and_evidence": "The paper makes several claims regarding the effectiveness of the proposed method, but not all are well supported:\n\n1. Scalability across different embodiments: The paper claims scalability, and while a few supplementary videos show different robots, there is no quantitative comparison in the main text or supplementary material. More analysis is needed in this part.\n\n2. Effectiveness of automated domain randomization (DR) and domain adaptation (DA): The experiments show that these techniques improve Sim2Real transfer, but additional ablations are necessary. For example, how does automated DR compare with hand-crafted DR? What is the contribution of individual components (e.g., AI-DR vs. SA-DR)? Similarly, what are the performance gains from object-oriented representations and pose-affordance representations in DA?\n\n3. Unverified claims in figures: Some figures include elements that are not backed up by experiments. For instance, Figure 2 mentions image/teleoperation as data sources and RL for policy fine-tuning, but these are not used in the experiments. While it is reasonable to illustrate pipeline flexibility, it may mislead readers if these elements are not actually part of the method.", "methods_and_evaluation_criteria": "The proposed method is reasonable for the problem setting.", "theoretical_claims": "No theoretical claims are presented in this paper.", "experimental_designs_or_analyses": "I checked the validity of all experiments, from the experimental design to the results. The existing designs are valid but would benefit from further analysis, especially regarding:\n\n1. Quantitative analysis across different embodiments.\n2. More detailed ablation experiments.\n\n(See Claims and Evidence for further details.)", "relation_to_broader_scientific_literature": "The paper builds upon prior work in Sim2Real transfer, domain randomization, and imitation learning. The discussion of related work is thorough, but a detailed comparison to baselines (e.g., naive randomization) would clarify the contribution.", "essential_references_not_discussed": "The paper covers most relevant references.", "other_strengths_and_weaknesses": "Most of my concerns are expressed below, here are some additional points:\n\n(+) The paper addresses an important challenge in Sim2Real learning and proposes an automated and scalable pipeline.\n\n(-) It lacks detailed ablation experiments on the effectiveness of different components.\n\n(-) This paper does not show quantitative results for different embodiments.\n\n(-) I find the current paper presentation can be improved. Some figure elements (e.g., teleoperation, RL fine-tuning) are not reflected in the method section. It would be helpful if the authors explicitly stated what is implemented and what is a possible future extension. Additionally, clearer explanations on how human hand trajectories are mapped to robot hands and how mesh refinement is obtained from video capture would improve clarity.", "other_comments_or_suggestions": "No additional comments. Please see my detailed points above.", "questions_for_authors": "1. Similar to mentioned above, what are the performance contributions of each module? And what are the performance for different\n2. How do you implement the retargeting from human hand to different robot embodiment?\n3. What are the primary failure modes observed in real-world deployments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAJ2nTPoV0", "reviewer": "Reviewer_ioHD", "summary": "The paper proposes a new data generation pipeline, that takes human video demo as input and generate retargeted data for robot manipulation. The pipeline involves different stages: scene projection, action-trajectory projection, scene simulation, action-trajectory simulation and various techniques to bridge the sim2real gap. The author conducts experiment on simple pick-and-place and open-box problems to suggest the proposed data generation pipeline produces better data quality to train manipulation policies.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, We sincerely appreciate your constructive feedback and thank you for recognizing the significance of our work. We have carefully considered your suggestions, and we hope that the following response can address your concerns:\n\n> 1. *\"...I only see simple tasks like grasping and open a box. These tasks are relatively fault tolerant comparing more dextrous task like opening a cabinet door via the handle. Could you provide evaluation on more dexterous tasks?\"*\n\nOur experiment comprises three distinct tasks, each involving a unique embodiment. To further support our argument—and in response to the reviewer's request—we additionally demonstrate DexSim's performance on two representative tasks: articulated object manipulation (e.g., opening a drawer) and combined grasping and manipulation (e.g., water pouring). Detailed results and examples can be found in Section 2.1 of our [anonymous link](https://anonymous.4open.science/w/dexscale/).\n\nAdditionally, we emphasize that the tasks demonstrated in our experiment are inherently challenging. For instance, the box-opening task requires the robot to sequentially open four flaps, demanding precise control and planning to complete the task successfully.\nEven more complex is the re-arrangement task, where the robot must reorient both the fork and spoon so they face the front of the plate, while also positioning them accurately around it. Smooth execution of this task necessitates coordinated movement between both arms, ensuring that the utensils are placed correctly and simultaneously.\n\n> 2. \"*Why not use existing platforms like MuJoCo or PyBullet?*\"\n\nChoosing a simulation platform involves balancing several factors: rendering quality, physics accuracy, overall simulation efficiency, and the flexibility and scalability of the simulator.\n\nUnfortunately, we have not found an existing open-source simulator that satisfies all of these requirements. For example, PyBullet and MuJoCo offer strong physics simulation capabilities, but their rendering quality is limited. On the other hand, Blender excels at rendering but is not designed for physics simulation—especially not for robotics applications.\n\nRecently, several simulators have emerged for robotics and embodied AI, such as AI2-THOR, RobotSuite, and SAPIEN/ManiSkill, but each comes with notable limitations:\n\nAI2-THOR uses Unity3D as its simulation backend. While Unity is a well-known game engine, it lacks support for GPU-parallel simulation and tiled rendering, which are crucial for large-scale robotics experiments.\nRobotSuite separates the physics and rendering engines and integrates them via a Python frontend. This architecture leads to inefficiencies and does not scale well for large simulations.\nSAPIEN/ManiSkill supports GPU-based simulation and tiled rendering, but it only handles rigid-body dynamics and lacks the flexibility needed to extend or incorporate new features.\n\nIsaacSim/IsaacLab comes closest to meeting all our requirements. However, it is not fully open-source, which limits our ability to modify or extend its low-level capabilities to fit our specific needs.\n\n> 3. \"*How would out-of-dataset objects effect the retarget trajectories?*\"\n\nDexSim includes an automatic domain randomization method to address mismatches between objects in simulated and real-world scenes. Specifically, the reviewer pointed out in the Scene Projection section (Section 4) that the targeted object is included in the asset dataset (OoD), and DexSim matches it with a similar object. Consequently, the skill is learned based on this matched object rather than the actual target object.\n\nHowever, because our domain randomization process modifies the size, shape, pose, and texture of objects, the learned skills can generalize across different objects. As a result, the model can successfully handle the real object, even if it differs from the one used in the simulation. We have experimentally validated this phenomenon in Section 5.1.\n\n> 4 \"*Add Discussion with paper like DexMimicGen: ...*\"\n\nWe have included this paper in our related works and discussed it in detail. The key differences lie in the format of input signals, the methods for skill generation, and the integration of automatic domain randomization and adaptation for Sim2Real generalization.", "claims_and_evidence": "The paper claims that the pipeline is more powerful in generating data. However, I only see simple tasks like grasping and open a box. These tasks are relatively fault tolerant comparing more dextrous task like opening a cabinet door via the handle. It is unclear whether the action retargeting algorithm will still provide useful trajectories to solve the problem.", "methods_and_evaluation_criteria": "The physical part is build on a custom physics engine in the current implementation. Why not using the current available platform like mujoco / pybullet?\n\nFor the scene / action projection, i would like to understand the impact of out-of-domain cases where one has to find similar objects in the set. How will this effects the quality of the synthetic data?", "theoretical_claims": "There is no theoretical claims.", "experimental_designs_or_analyses": "The experiment is supportive for the dexsim techniques on the given tasks. However, as mentioned above, it fails to support that dexsim is a valuable platform to generate valuable synthetic trajectories for other dexterous tasks, which is of ture interest of the community. For simple pick and place tasks, one can also design procedural ways to generate synthetic data easily.", "relation_to_broader_scientific_literature": "The paper proposes a way to generate synthetic data given input of real-world human demos. This aims to tackle the challenge in robot learning, where such data is scarce in practice.", "essential_references_not_discussed": "Discussion with paper like DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning, where it is also aims for synthetic data generation but with different types of inputs.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Why not using the current available platform like mujoco / pybullet?\n2. How would out-of-dataset objects effect the retarget trajectories?\n3. Could you provide evaluation on more dexterous tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s2Hpb99NCq", "reviewer": "Reviewer_dQaj", "summary": "The paper introduces a data engine that automatically simulates and scales skills for learning robot manipulation policies. In particular, DexSim presents a comprehensive pipeline for Sim2Real data scaling by automating domain randomization and adaptation processes. The authors claim that this approach not only achieves superior zero-shot Sim2Real performance but also enhances generalization across diverse tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we sincerely appreciate your constructive feedback. We hope that the following response can address your concerns:\n\n> 1. \"*...substantial manual adjustments are required to align the hand and object models within the simulation before retargeting them to the end-effector pose. ... Figure 7 further suggests that this approach is labor-intensive and may not be scalable.*\"\n\nDexSim features an automated pipeline for extracting human hand poses from videos and aligning them with objects **without any manual intervention**. The joint optimization process (see line 222, right column) between the hand and object models is fully automated. Specifically, similar to Liu et al. (2024), we utilize a learning-based approach to iteratively refine and synchronize hand-object interactions over multiple rounds until they are properly aligned. We have clarified this in the revised version of our study.\n\n(Liu et al., 2024)  Liu, Yumeng, et al. \"EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild.\" arXiv preprint arXiv:2411.14280 (2024).\n\n> 2. \"*...The performance differences between simulation and real-world results for skills+DR and DexSim are minimal ...*\"\n\nThank you for pointing this out. In our experimental results, we observed that the performance gap between Skill+DR and DexSim is less pronounced in the object grasping task. One major contributing factor is the limited number of trials conducted in both real-world and simulated environments. To address this, we have increased the number of experiment runs. The updated results are shown below:\n\n| Setting | Skill | Skill+DR | Skill+DA | DexSim |\n| -------- | ------- | -------- | ------- | ------- |\n| Real-World | 3/25 | 7/25 | 9/25 | 14/25 |\n| Simulation | 93/200| 122/200 | 151/200 | 157/200 |\n\nTo further support our findings, we have included the relevant experiment videos in Section 3.1 of our [anonymous link](https://anonymous.4open.science/w/dexscale/).\n\n> 3. \"*A more robust evaluation—potentially using large-scale simulation environments such as the Factor-world or The Colossuem Benchmark, which incorporate perturbation variations—would strengthen this claim.*\"\n\nOur study focuses on real-world experiments, demonstrating improvements in generalization performance on physical robots rather than in simulated environments. Specifically, our experiment on generalizability (Section 5.1) addresses the challenge of bridging the Sim2Real gap. We show that the automatic domain randomization (DR) and domain adaptation (DA) mechanisms in DexSim effectively transfer policies learned in simulation to real-world applications.\n\nIn contrast, benchmarks that study generalization across discrepancies between training and testing environments, such as the Factor-World or the Colosseum Benchmark, **still evaluate performance in simulated settings**. However, as we argue in Section 3, the Sim2Real gap presents a significant challenge. There is no guarantee that a policy performing well in these benchmarks can be successfully deployed in real-world scenarios.\n\n> 4. \"*The method itself is problematic due to its many moving parts, where a failure in any single component can undermine the entire process.*\"\n\nThe term \"moving parts\" in the review seems confusing. Generally, in methods implementing the Real-to-Sim-to-Real pipeline, the presence of multiple components is a fundamental characteristic. In the case of DexSim, these components are not only necessary but also integral to the pipeline's functionality. Similarly, methods like RoboGen and GenSim also adopt pipelines composed of multiple interconnected parts. Therefore, the presence of \"many parts\" is not unique to this method but rather a common feature of such approaches.\n\n> 5. \"*... there are alternative methods for improving sim-to-real transfer—such as ASID, SystemID, and GenAug...*\"\n\nWhile these studies are indeed inspiring, we found that their methods are not directly comparable to ours. Specifically:\n\n- ASID focuses on refining simulation models to better represent real-world dynamics. In contrast, DexSim interacts with a given dynamics model without access to its underlying function.\n- SystemID addresses system identification by recovering nonlinear models of dynamical systems from data. However, it does not consider simulated environments or skill discovery methods relevant to our study.\n- GenAug appears to be a data augmentation technique leveraging image-text generative models. Notably, its augmentation approach enforces action invariance. In contrast, DexSim can autonomously refine actions when the target object changes.\n\n> 6. \"*Figure 2 includes numerous elements that are not adequately explained in the caption.*\"\n\nThe elements in the figures have a one-to-one correspondence with the titles of the subsections and paragraphs in Section 4: Data Engine for Sim-to-Real Generalization. This relationship has been clarified in the revised version of our study.", "claims_and_evidence": "**The paper claims:**\n- DexSim can generate simulation trajectories based on real-world human demonstrations.\n- DexSim facilitates improved zero-shot Sim2Real transfer for manipulation policies.\n- DexSim supports generalization across a variety of tasks and multiple robot embodiments.\n\n**However, several claims appear weak and are not well supported:**\n- **Manual Adjustments and Pose Capture:**  \n  As illustrated in Figure 3, substantial manual adjustments are required to align the hand and object models within the simulation before retargeting them to the end-effector pose. Additionally, it remains unclear how the delta-pose is extracted from human hand video. Figure 7 further suggests that this approach is labor-intensive and may not be scalable.\n  \n- **Statistical Significance:**  \n  The performance differences between simulation and real-world results for skills+DR and DexSim are minimal. With additional experiments and the inclusion of standard deviation data, these differences might not be statistically significant.\n  \n- **Generalization with Perturbation:**  \n  The improvements in generalization achieved through perturbation appear marginal. A more robust evaluation—potentially using large-scale simulation environments such as the Factor-world or The Colossuem Benchmark, which incorporate perturbation variations—would strengthen this claim.", "methods_and_evaluation_criteria": "The evaluation criteria are well-suited for assessing zero-shot sim-to-real transfer. However, the method itself is problematic due to its many moving parts, where a failure in any single component—given its reliance on other models—can undermine the entire process. Consequently, scaling this approach appears infeasible.", "theoretical_claims": "Yes, the problem formulation make sense.", "experimental_designs_or_analyses": "The experimental design is sound. However, there are alternative methods for improving sim-to-real transfer—such as ASID, SystemID, and GenAug—that the authors could consider for comparison.", "relation_to_broader_scientific_literature": "Yes, the work could be more valuable with a broader discussion of existing sim-to-real methods beyond just domain randomization (DR) and domain adaptation (DA).", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Figure 1 does not effectively convey the core idea of the work, and the alignment of arrows and boxes appears to have been executed hastily. \n\nFigure 2 includes numerous elements that are not adequately explained in the caption, which hinders the reader's understanding of the essential pipeline.", "other_comments_or_suggestions": "Nill", "questions_for_authors": "Nill", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces DexSim, a tool for generating diverse large-scale demonstration data for robotic manipulation tasks in simulation based on human videos.

DexSim targets one of biggest problems in robot learning, data scarcity. The reviewers appreciate its significance, as well as DexSim's potential for alleviating it.

Originally, the reviewers had a number of concerns regarding DexSim's empirical evaluation, including the effectiveness of its domain randomization. However, the experiments done by the authors for the rebuttal have largely addressed these. The two remaining issues with this work are:

**Writing/presentation quality.** The reviews mention the specific details. None of these issues are grave, and all can be fixed in the camera-ready. The metareviewer strongly encourages the authors to do so.

**The limitations in tasks for which DexSim can generate data.** DexSim is largely limited to pick-and-place tasks. While the authors claims it goes beyond that and can handle tasks such as pouring and reorientation, the videos provided by the authors in the rebuttal don't support these claims (e.g., no liquid is actually being poured in the pouring task videos). Tasks that involve assembly, such as screwing a nut onto a bolt, also appear to be outside DexSim's scope. Nonetheless, this limitation is understandable and stems, among other factors, from the limitations of the underlying simulators. Other robot data generation frameworks such as DexMimicGen, are even more restrictive.

Thus, on balance the metareviewer believes that the paper makes a very valuable contribution to solving the data challenges of robot learning.
Decision: Accept (poster)