Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots
Authors: Minghuan Liu
Zhengbang Zhu
Xiaoshen Han
PengHu
Haotong Lin
Xinyao Li
Jingxiao Chen
Jiafeng Xu
Yichu Yang
Yunfeng Lin
Xinghang Li
Yong Yu
Weinan Zhang
Tao Kong
Bingyi Kang
Venue: ICLR 2026 Poster (2026)
Keywords: 3D perception; manipulation; sim-to-real; depth foundation model
Primary area: applications to robotics, autonomy, planning

Abstract: Modern robotic manipulation primarily relies on visual observations in a 2D color space for skill learning but suffers from poor generalization. In contrast, humans, living in a 3D world, depend more on physical properties-such as distance, size, and shape-than on texture when interacting with objects. Since such 3D geometric information can be acquired from widely available depth cameras, it appears feasible to endow robots with similar perceptual capabilities. Our pilot study found that using depth cameras for manipulation is challenging, primarily due to their limited accuracy and susceptibility to various types of noise. In this work, we propose Camera Depth Models (CDMs) as a simple plugin on daily-use depth cameras, which take RGB images and raw depth signals as input and output denoised, accurate metric depth. To achieve this, we develop a neural data engine that generates high-quality paired data from simulation by modeling a depth camera's noise pattern. Our results show that CDMs achieve nearly simulation-level accuracy in depth prediction, effectively bridging the sim-to-real gap for manipulation tasks. Notably, our experiments demonstrate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots on two challenging long-horizon tasks involving articulated, reflective, and slender objects, with little to no performance degradation. We hope our findings will inspire future research in utilizing simulation data and 3D information in general robot policies. We release the dataset, models for various depth cameras, along with an easy-to-use guide for sim-to-real transfer at https://manipulation-as-in-simulation.github.io/.

Published as a conference paper at ICLR 2026
MANIPULATION AS IN SIMULATION: ENABLING AC- CURATE GEOMETRY PERCEPTION IN ROBOTS
Minghuan Liu Lf Zhengbang Zhu 12, Xiaoshen Han .2,,, Peng Hu HHattng Lin Xinyao Liº, Jingxiao Chen 1.2 Jiafeng Xu Yichu Yang', Yunfeng Lin", Xinghang Li", Yong Yu?, Weinan Zhang?.i, Tao Kong'. Bingyi Kang-.
BByteDance Seed, "Shanghai Jiao Tong University, ZZhejingg University, "Tsinghua University ericliucf 978gmmf77mmmmmm. com, wnzhangesjtu. edu.. cn, bingykang@@gaill com
ABSTRACT
Modern robotic manipulation primarily relies on visual observations in a 2D color space for skill learning but suffers from poor generalization. In contrast, hu- mans, living in 3D world, depend more on physical properties-such as dis- tance, size, and shape-than 3D geometrie information can be acquired from widely available depth cam- eras, appears feasible to endow robots with similar perceptual capabilities. Our pilot study found that using depth cameras for manipulation is challeng- ing, primarily due to their limited accuracy and susceptibility to various types noise. In this work, we propose Camera Depth Models (CDMs) as a simple plugin on daily-use depth cameras, which take RGB images and raw depth sig- nals as input and output denoised, accurate metric depth. To achieve this, we develop a neural data engine that generates high-quality paired data from simu- lation by modeling depth camera's 's noise pattern. Our results show that CDMs achieve nearly simulation-lever accuracy in depth prediction, effectively bridg- ing the sim-to-real gap for manipulation tasks. Notably, our experiments demon- strate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real- world robots on two challenging long-horizon tasks involving articulated, reflec- tive, and slender objects, with little to no performance degradation. We hope our findings will inspire future research in utilizing simulation data and 3D in- formation in general robot policies. We release the dataset, models for vari- ous depth cameras, along with an easy-to-use guide for sim-to-real transfer at https::/ pu i on as- in- s mu at ionn wit....thhi...
texture when interacting with objects. Since such
INTRODUCTION
Manipulation a fundamental capability expected of robots, primarily involving skilled interactions with diverse objects, and thus necessitating visual observations. Recent advances show that robots can perform various tasks using 2D color images from single or multiple viewpoints (Chi al. 2023 Zhao et al.) 2023 Fu et al.) 2024 Li et al., 2024a Wu et al.) 2024 Team et al.) [024] Kim et al., 2025 Black al. 2024). While color images provide rich semantic information, humans operate in 3D world and rely geometric objects (e.g., bottles versus bowls) and comprehend the skills required. This reliance on geometry. rather than texture, enables functional inference and precise interaction with objects.
as shape and spatial relationships to distinguish
With the widespread availability of depth cameras, acquiring 3D geometric information appears to be straightforward, suggesting that robots could be endowed with similar perceptual capabili- ties (Fang et al. 2023 Yan et al. [0044] [iiu et al., 2024, Ze et al. 2024b). However, their unreliable output, frequent mode failures, and sensitivity noise pose significant challenges. Although recent studies have integrated 3D representations into robotic manipulation, performance remains limited by the poor quality of depth data produced by such devices. Consequently, evaluations are typically
Equull contribution. 'Corresponding authors.

---

Published as a conference paper at ICLR 2026
Depth Comera
Sim Scene
Camera Depth
Model
Unproject
Unproject
Unproject
Geometrically
Aligned
Clean
X Noisy
Accurate
Train
Inaccurate
Enable Direct
Sim-to-Real Transfer
Monipulation Policy
Real Scene
Figure 1: How the proposed camera depth model (CDMs) makes real-worid manipulation in simulation. The illustration made on the RealSense D435 camera with CDM-D435, With CDM, the manipulation policy learns from accurate geometric information, which is aligned between the simulation and the real world.
where clean and accurate Hua et al.] [2024) (Ze et al.) 2024b) to mitigate noise in real-worla scenarios. As illustrated in Fig. real-world depth camera data often contains significant and characteristic noise artifacts, resulting in inaccurate perception of objects and environments by robots.
restricted simulation environments (Zhen al. 2024 Zhu 2024
depth is available; or rely on downsampled point clouds (Ze et al. 2024a
To mitigate the fundamental problem of depth perception and bring accurate geometry into robotic manipulation, this paper proposes camera depth models (CDMs), plug-in solution for depth cam- eras that enhances geometric accuracy, as illustrated in Fig. T A CDM processes RGB images and noisy depth signals from specific depth camera to produce train such models, developed multi-camee mount and collected dataset of RGB-depth pairs from seven cameras across ten depth modes. Leveraging both this dataset and open-source simulated data, designed neural data engine that models the noise patterns of depth cameras generate high-quality paired data in simulation. To address the scale mismatch in synthesized noise, we pro- pose a novel guided filter approach for noise augmentation. CDMs achieve nearly simulation-level 3D accuracy, effectively bridging the sim-to-real geometry gap from the real-world perspective.
denoised metric depth. To
Our experiments evaluate CDMs in real-world imitation and sim-to-real manipulation tasks. We show that CDMs enable robot policies to learn generalizable skills from accurate geometric infor- mation. Notably. we demonstrate, for the first time, that a policy trained raw simulated depth, without the need for adding noise or real-world fine-tuning, can transfer seamlessly to real robots on two challenging oong-hoiizon tasks involving articulated, reflective, and slender objects. These results highlight the potential of CDMs to leverage simulation and underscore the importance of accurate geometric data in developing robust, generalizable robot policies.
In nutshell, the contributions of this paper are:
We introduce ByteCameraDepth, real-world multi-camera depth dataset comprising over 170,000 RGB-depth pairs captured by seven depth cameras.
We proposed and released a family of camera depth models (CDMs), plug-in solution that enhances depth perception accuracy for widely used depth cameras.
3. Through CDMs, we demonstrate how the sim-to-real geometry gap can be bridged, highlighting the critical role of accurate geometric information in robotic manipulation tasks.
2 RELATED WORK
Metric depth prediction. Recent advances hepth-fundamentee models, such as the Depth Any- thing (DA) series YYang al. 2024a b). has significantly advanced scene geometry estimation and
2

---

Published as a conference paper at ICLR 2026
high-resolution relative depth prediction across diverse open-workd images, showing strong gener- alization. However, most real-world applications require metric depth rather than relative depth. Simply fine-tuning DA models to predict metric depth (Yang et al. 2024b) remains constrained by
fixed depth scale and is susceptible to scale ambiguities (Yin et al. | Hu et al., 2024). Although recent approaches (Wang et al.) [024bb (025b) introduce affine-invarian techniques to train relative models on large-scale datasets and achieve improved metric depth estimation via post-processing with a prompt depth, the fundamental scale ambiguity in monocular images, especially for scenes with large depth ranges, remains unresolved. To address this, many recent approaches choose to incorporate explicit scale cues for predicting metric depths. For instance, Guizzilini al. (2023) and Instance,,Guizilin
Piccinelli et al. (2024) introduce camera intrinsics into the model. Lin et al. (2025) and Wang et al. (2025c) proposed a more straightforward way that directly integrates scale information by prompt- ing paradigms, Le., low-quality depth images or sparse LiDAR signals, into the model's architecture of the pre-trained DA model, and finetuned the model on RGBD datasets with handcrafted prompt depth images synthesized data. Nevertheless, they are limited in prompt images made with the style of handcrafted rules and are hard to work well on diverse sensor configurations for dynamic scenes. In addition to these solutions with depth prompt, there are some works focused recov- ering metric depth (disparity) from stereo RGB images (Wen et al. 2025), which provide implicit depth cues through disparity, but they often require careful calibration and are limited in diversity. Such methods are limited to working on stereo cameras (and with RGB only) and require the precise camera intrinsics (baseline distance, focal length, and so on) to obtain the depth.
Manipulation with 3D representation. Robotic manipulation requires accurate perception of ob- ject states. Classical planning- based approaches typically depend on a calibrated perception module to identify the 3D positions of relevant objects, which are then used to plan feasible manipulation paths (Fang et al. 2023). Learning-based methods, on the contrary, focus on modeling autonomous robot policies using neural networks. Most recent works rely RGB images, ranging from single-
view (Chi et al. 2023] 2024 Kim et al. (2025) to multi-view setups (Zhao et al. 2023 Fu et al.) 2024), and from single-tast policies (Chi al.) 2023 Zhao et al. (002) to multi-task generalist policies (Li et al. 2024a Wu et al. 2024| Kim et al. 2025 Black et al. 2024). While these methods achieve strong results, they struggle to generalize across visual conditions. To better leverage scene geometry, recent works incorporate 3D representations into policies in different ways: some use point clouds to improve generalization across objects with similar shapes but varying textures and backgrounds (Ze et al. 20244a b. Hua et al. 2024)) though these approaches typically require careful calibration, downsampling, and cropping to mitigate sensor noise; some train depth-only policies that segment objects and mitigate sim-to-real discrepancies (Liu et al.] 2024); and others propose 3D-aware foundation models that jointly process depth, point clouds, or bounding boxes, though these are often limited to simulation with perfect 3D perception (Zhen et al.) [0224] Zhu al. 2024).
Furthermore, some works explore more complex representations, such as neural radiance fields (Li
al. 2022; Yan al., 2024) and dense voxels (Shridhar al. [0023] Ze et al. (023); however,
all of these 3D representations ultimately depend on transforming the original camera depth into real-world scenarios.
3 CAMERA DEPTH MODELS
Existing depth foundation models can estimate proper relative depth without a geometric prior. However, for real-world manipulation tasks, models must predict absolute metric depth. This re- quires two key capabilities: 1) identifying semantically meaningful local regions for objects and backgrounds in RGB images, and 2) assigning accurate metric depths these regions using coarse depth prompts from camera depth images. Notably, this task extends beyond simple denoising or depth completion, raw depth readings from various depth cameras exhibit diverse working ranges, failure modes, noise patterns, and biases. Fig. provides a glimpse of the noisy depth images pro- duced by consumer-grade depth cameras.
3.1 NOISE FROM DEPTH CAMERAS
We categorize two general types of noise, i.e., the value noise and the hole noise, depicted in Fig. left. Intuitively, hole noise manifests as missing data in depth readings, often caused by depth estimation algorithms (e.g., stereo matching) or environmental factors, such as lighting material
3

---

Published as a conference paper at ICLR 2026
Sensor Depth Generation
Camera Depth Model
Dapth Tokens
Multi-Head Attention
Voluum No Modal
Hole No Mode!
Takens
Concottip Tourrs
Lovar-oER
VIT Encoder
Token
Rod image
DPT Decodar
Recl Camera
Fusion
Row Dagith
Figure 2: Overview of camera depth models. Left: camera depth generation synthesize the datasets for training camera depth models, where the value/hole noise models are trained using the collected dataset. (Sec. (3.4p Right: the camera depth model, which built on two ViT en- coders (Dosovitskiy et al., (0200) and fine-tuned from depth foundation model Yang et al. 2024b); the RGB and depth tokens are fused before being given to DPT decoder (Ranftl et al. 2021). y. Such structure allows the model to receive sparse depth from sensors for prediction without any pre- processing like hole-filling. (Sec. 3.2)
properties. Value noise encompasses all other inaccuracies, including biases specific to each camera, as well blur, jitter, and other distortions. For instance, stereo matching-basce cameras often produce holes around object boundaries, while LiDAR-based cameras struggle with black or highly reflective surfaces. Both types perform poorly transparent mirror-like objects, such glass. These noise patterns depend on the camera's ' intrinsic parameters and physical installation.
Therefore, develop effective metric depth model for specific camera, the model must 1) refine coarse, low-quality depth prompts from the camera into precise metric depth estimates, while 2) correcting faulty depth readings by leveraging semantic information from RGB images. Balancing reliance on sensor data with skepticism of its inaccuracies poses a significant challenge, making generalizable solution nontrivial. This motivates the development of camera-specific depth models (CDMs) tailored to individual depth cameras.
3.2
MODEL DESIGN
We designed our CDMs M for specific depth cameras take a pair of an RGB image I 3 and depth image D E TR H from the depth camera, and predict high-quality metric (absolute) depth image D E H The proposed model structure of CDMs is illustrated in Fig. right. In particular, we design dual-branch ViT (Dosovitskiy al. 2020) architecture to achieve the above- mentioned capabilities by separately capturing semantic information from the RGB and the depth images, as well as scale information that cross-modae but aligned in feature tokens X:
x = (I) x" = VD)
(1)
where X branch and the depth branch, separately.
(X.....,XX x and X (xP, ... X are feature tokens encoded by the RGB
Subsequently, we fuse these two types of information through feature token fusion module. Since the token fusion module primarily serves to augment semantic information with scale information, only necessary to fuse tokens corresponding to the same spatial locations. Based on this, the fusion module only performs self-attention corresponding tokens to accomplish bidirectional feature fusion, and results in depth features X imbued with scale information:
X
Σ [HA(([X):xPIE)
where MHA stands for multi-head attention, and [:] is the concatenation operation. The entire fusion process occurs across multiple levels of feature tokens, allowing for deeper integration and the ability to incorporate both global- and local-scale information, especially when the camera depth prompt has large missing regions, where global-scale information is particularly needed.
Additionally, we concatenate the original RGB feature tokens into the fused feature tokens. These fused feature tokens, along with the RGB feature tokens, concatenated to prevent loss of semantic
4

---

Published as a conference paper at ICLR 2026
information, and then passed through DPT head (Ranftl et al. 2021) to produce scale-aware depth estimation results D.
D = DPT((X);
Compared with previous works that fuse the prompted depth information simply in the shallow decoding phase (Wang et af. [2025c] Lin et al. (025), our proposed CDM structure provides a much more informative representation of the depth feature and its alignment with the RGB feature, and thus can perceive the raw depth image directly, without preprocessing such as hole-filling |Wan et al. (202cc); Lin et al. 2025). Through the simple but representative structure design, CDM simplities the inference procedure, provides the metric depth, and works as a simple plugin after the camera input, thereby fulfilling the three desiderata mentioned in the beginning.
3.3 BYTECAMERADEPTI A MULTI-CAMERA DEPTH DATASET
To train our CDMs, we will need dataset that contains triplets. i.e., RGB image I, low-quality depth image D. and ground-truie depth image D. However, the low-quality depth images, although they have usually been handcrafted by adding typical noise patterns to ground-truth depth images that are simulated by basic depth measurement principles, other factors are hard to be accurately modeled in the simulation, for instance, the camera parameters, the implementation and optimization details in each depth camera hardware and software are case by case, resulting distinct noise behaviors. On the contrary, we can easily collect low-quality depth data from real sensors, but it is hard to get perfect depth data. Therefore, naturally, we propose learn the noise pattern with neural networks automatically from real-world data, and then synthesize the noisy low-quality depth image with the learned noise models.
To this end, we collect typical depth patterns and construct a dataset for various depth cameras that are commonly used in daily robot experiments. Specifically, our dataset spans 10 depth modes from 7 different depth cameras, including different stereo and lidar cameras. To achieve highly effi- cient data collection, we design a multi-camera mount device to capture data simultaneously, details shown in Appendix[[] Our datasets contain more than 17,000 images for each camera, sampled from videos at 5Hz, covering 7 different scenes, including kitchens, living rooms, markets, bedrooms, bathrooms, offices, and breakrooms. A visual glimpse of the dataset is shown in Appendix
3.4 DATA SYNTHESIS WITH NOISE MODELS
We train two noise models our collected depth dataset for each camera, which are then used for generating stylized low-quality depth images on open datasets to train CDMs.
Hole noise model. We treat the hole noise prediction as binary-class prediction given the RGB image 1, thereby training the hole noise model Nhole with a pretrained DINOv2 backbone (Oquab) al. 2023) with a DPT head (Ranftl al. 2021 I) to predict the valid mask (i.e., hole/non-hole) for each pixel on the camera depth D. Formally, this corresponds to optimizing the objective:
i=H.j=W Σ i=0,j=0
E(Nhole (T))
[ui log (x4g)( + (1 - 9.,)) log(1 (xx1)))
(4)
where Z4,5 = Noole (D)s denotes the i,j pixel on the mask image predicted by the hole noise model Noole- Yh.. T(D:; 0) denotes if the i, j-th pixel corresponds to hole, and is the
sigmoid function.
Value noise model. Motivated by the fact that depth foundation models are predicting clean style depth image, we regard the value noise prediction as stylized relative depth prediction problem, thereby turning the help of Depth Anything V2 YYang al., 2024b) (DAV2) by taking the low- quality depth image as the labels for prediction. The training objective of the value noise model Nvatue is an L1 loss of the predicted depth Dvalue = Nvatue (I) and a normalized ground truth depth D:
((Nvatue) = In (J(),, D Dvalue)
(5)
where f is the normalization function, in our project, we use the affine-invarl invariant transformation proposed in Wang et al. (2025a). To make sure the value noise learn proper relative scales
5

---

Published as a conference paper at ICLR 2026
during the data synthesis stage, we also fine-tune the DAV2 model on the synthesized dataset before turning it into a value noise model.
synthesize the noisy low-quality data on open synthesized datasets with clean ground truth depth. Denote the hole noise model as H and the value noise model as V, we cured the synthesized noisy data given the RGB image I via:
Synthesizing camera depth. Having the noise models,
D (V((T) (H < (.5),
(6)
where js is the affine-invarian unscaling function (Wang et al. 2025a recovering the metric of the predicted relative value noise, referring to the metric of the ground truth depth.
3.5 CDM TRAINING
Although we can synthesize training data for specific cameras, and these two types of noise models can learn similar noise patterns compared to the raw depth obtained from the real depth camera, we observed several problems, especially with the value noise models.
Guided filter for value noise. A key challenge of the value noise model its struggle to maintain the correct metric scale on synthesized datasets after fine-tuning the ByteCameraDepth dataset. This leads to metric discrepancies between the synthesized camera depth and ground-truth depth, causing the trained CDMs to underutilize the metric information in the camera depth prompt. To address this, we propose using the guided filter (He et al.] 2012). which assumes the output image B is local linear transformation of the guided image G: pixel of B and G, respectively; Zg, Yk are the scale and shift parameters the kernel window. These parameters are optimized by minimizing the error of the transformation between the input image A and the output image B:
F&Zi + Vl where b;, 9r are the i-th
Σ ((129) + yk az)² +cxx)
where is regularization term. In approach, the guided filter uses the value noise the guidance image G and the ground-truth depth as the input image to be filtered A. The resulting image B preserves the geometry and structure of the value noise while approximating the correct metric scale across the image. As the kernel size k increases, the output image retains more of the noise structure and less of the ground-truth metric information. To balance this, we employ randomized kernel size & (ranging from small to large) an augmentation strategy for the value noise before adding hole noise, which yields optimal results. Additionally, adjusting the maximum value of l controls the model's reliance on the prompt depth: smaller k aligns the prompt closer to the ground truth, encouraging the model to depend more on the prompt. Furthermore, both noise models struggle to capture high-frequency noise patterns due to the neural network's 's limitations and our DAV2-like training strategy. To address this, we introduce high-frequency noise via handcrafted rules as additional augmentation strategy.
Training loss. Referring to Lin et al. (2025), we use the L1 loss combined with the gradient loss for better edge depth to train CDMs M, given image I and its raw depth D:
((M) = T,, (D,D) + fgral (D.D),
(8)
where Egod DD = aaD D
8ND D
dæ
By
During our training, we use disparity as the training target. The weights of the ViT encoder in the
RGB and the depth branch are both initialized from DINO-v2 (Oquab et al. 2023), and the decoder is trained from scratch. For the single-channe depth images, by default, they are copied three times before being fed into the network. We synthesized our training data on four simulated datasets:
HyperSim (Roberts et al.] 202T), DREDS (Dai et al. 2022). HISS (Wei et al. 2024), and IRS (Wang al. 2019). total of 280,000+ images.
3.6 SIM-TO-REAL MANIPULATION THROUGH CDMS
Our camera depth models (CDMs) allow us to obtain a 'simulation-lin depth image in the real world, which provides accurate geometry information. Therefore, simulation data can fully uti- lized to learn manipulation policy, which be seamlessly transferred to the real world. To

---

Published as a conference paper at ICLR 2026
Split
Filled / Holed /
L1
RMSE L
AbsRel
do.s
6,
Ours (CDM-1435)
0,0258
0,0404
0.0311
0.9842
0,9951
Ours (CDM-L515)
0.0182
0.0338
0,0217
0.9877
0.9956
PromptDA
0.0434
0.0666
0.0599
0,9459
0.9770
Promp(DA*(515)
0.1830
0.2387
0.2750
0.8802
0,9186
(IR Stereo)
D435
PrumpIDA
0.1703
0.2971
0.2437
0.6704
0.7229
PriorDA
1.2031
0.6856
1,2030
0.0837
0.1717
PrumpiD2A
0.0396
0.0691
0,0484
0,9503
0.9772
PrionDA
0.0388
0.0754
0.0461
0,9632
0,9880
Raw Depth
0,0550
0.1458
0.0708
0,9179
0.9543
Ours (CDM-L5515)
0,0156
0.0297
0.0229
0.9754
0.9919
Ours (CDM-D435)
0.0165
0.0349
0.0246
0.9613
0.9855
PrompiDA*(115)
0.0235
0.0666
0.0349
0.9291
0,9730
LS15
PrompiDA*(435)
0.0254
0.0438
0.0379
0,9234
0,9640
PrompiDA
0.0483
0,0400
0.0612
0.8867
0.9259
(D>Toff)
PriorDA
0,5412
0.6134
0.9211
0.0850
0.1794
PrumpIDA
0.0207
0.0515
0.0304
0,9480
0,9699
PriorDA
0.0177
0.0385
0.0274
0,9502
0.9763
Raw Depth
0.0312
0.0813
0.0475
0,9098
0.9429
Ours (CDM-L515)
0.0248
0.0403
0.0334
0.9468
0,9871
(CDM-D435)
0.0272
0.0457
0,03722
0.9297
0,9806
Helios
PrompiDDA
0.0207
0.0515
0.0304
0,9480
0.9699
(I-Tof)
PriorDA
0.0324
0.0597
0.0461
0.8984
0.9638
Raw Depth
0.0312
0.0813
0.0475
0.9098
0,9429
Table 1: Quantitative comparisons of metric depths prediction on Hammer (Jung et al., 2023
dataset (zero-shot evaluation). The terms Filled. and Holed. refer to whether the low-quality depth is filled directly given to the model for prediction. *(split) denotes fine-tuning on our synthesized datasets with the same augmentation strategy. Raw depth refers to the metric directly using a low-quality depth image without model. CDMs named as the camera type, which are trained on the corresponding synthesized noise of that camera. All results are computed directly from the output of these models, without any alignment postprocessing.
evaluate the power of CDM and its benefits to robot manipulation, develop geometry-base based sim-to-real pipeline that contains four main stages: scene construction, camera alignment, simulated data collection, and imitation learning. After, we directly deploy the trained policy onto real-world robots, with the corresponding CDM as an inference plug-in. In particular, we choose depth instead of pointclouds as the observation for the following reasons: 1) human relies on single-view stereo visual observations and can do many things; 2) pointclouds fused from multi-view cameras require careful camera calibrations and are more sensitive to irrelevant backgrounds. Details can be further referred to Appendix D
4 EXPERIMENTS
The experiments involved are mainly threefold. 1) Does the camera depth model achieve better performance than other methods? 2) How does the accurate geometry information benefit real-world robot manipulation? 3) How the "sim-like" geometry contribute to zero-shot sim-to-real robotics manipulation? Note that our goal is not to identify if depth is better visual modality than color, but to validate whether accurate geometry information contained in a more precise depth image can benefit manipulation. Therefore, the policies designed in experiments depth-only, excluding the effect of color information.
4.1 DEPTH PERFORMANCE
We mainly evaluate the trained camera depth models (CDMs) on the Hammer dataset (Jung et al. 2023), real-world dataset that contains warped depth data paired with RGB images collected by three depth sensors: the RealSense D435 (stereo depth based active structure light and IR im- ages), L515 (a D-Tof camera), and a Lucid Helios (an I-Tof camera). Note that the dataset is not used for training, showing a zero-shot performance. We compared our CDMs against two baseline methods, PromptDA (Lin et al. 2025) and PriorDA (Wang et al., 2025c). both of which are metric depth prediction methods using prompt depth images and require hole-filling preprocessing during inference time. Since our CDM directly takes a prompt depth image as it is, we test two cases, i.e., Filled and Holed, denoting whether the low-quality depth is filled or directly given to the model
7

---

Published as a conference paper at ICLR 2026
D435
L515
URZ
(a) Overall setup (sim).
(c) Kitchen randomization (sim).
(e) Canteen randomization (sim).
D433
LOTS
(h) Overall setup (real).
(d) Kitchen task distribution (real). (f) Conteen task distribution (real).
Figure 3: UR-5 manipulation experiment setup. (a)(d) The overall setup for simulation and the real-world experiments, where we test two different cameras (D435 and L.51) and train single- view policies for both cameras. In simulation, we add small randomization to the camera pose when generating demonstrations. (c)e)The training environments of the kitchen and the canteen tasks in the simulation, where the dotted frame and the arrow denote the randomization range of objects. (f)(f) The test distributions of the two tasks, where we compose the position of the bowls and microwaves, following the randomization boundaries in the simulation.
for prediction. We also compared PromptDA fine-tuned on our synthesized dataset to show the advantage of the structure design.
The results are shown in Tab. [ where we can observe and conclude several things. Both PromptDA and PriorDA failed to obtain good depth prediction without hole-filling preprocessing, which requires additional computation time in real-time robot experiments. 2) Even with hole- filling, our CDMs achieve the state-of-the-ard performance on corresponding data splits. 3) With the same training data and augmentation strategy, our CDMs still perform better than PromptDA, showing the advantage of our designed structure. 4) The model trained specific synthesized cam- era noise data should work better on the same camera data split, like PromptDA; however, to our surprise, the CDM-L515 generalized well the 435 datasets and even achieve slightly better results than the CDM-D435 model, indicating the test cases of the D435 camera in the Hammer dataset can be generalized by the CDM-L515, 5) Both CDMs have better zero-shot generalization ability on the data split with different depth sensor (the I-Tof Lucid Helios camera), potentially because CDMs solve some common noise problems among depth cameras.
4.2 IMITATION LEARNING WITH ONLY DEPTH
We aim to investigate how the accurate geometry information produced by the camera depth models benefits robot manipulation tasks. To this end, we design pilot study that includes two pick-and- place tasks (Toolkpaste-and-Co) and Stack-Bowls) using a daily-use depth camera, the RealSense D435. We manually collect 50 trajectories for each task through teleoperation of real robot and conduct the test at five different positions, with three trials at each position. In particular, for the Stack-Bowls task, we trained our policy on normal-sized bowl and tested on bowls of five different sizes, including four unseen sizes. Without CDMs, the robot almost entirely failed in Toothpaste- and-Cup (0% success) and achieved only 20% success in Stack-Bowie With CDM-D435, success improved markedly, reaching 40% in Toothpaste-and- the CDM-enabled policy generalized to bowls of unseen sizes, achieving consistent non-zers zero success across four novel sizes, while the baseline policy without CDMs failed completely. This highlights that accurate geometry not only boosts task performance but also enables robust generalization to previously unseen object scales. Detailed setup and results provided in Appendix
and 60% in Stack-Bowls. Furthermore,
8

---

Published as a conference paper at ICLR 2026
Table 2: Zero-shot sim-real results using CDMs as the plugin in a real-world robot pipeline.
Kitchen Task
Canteen Task
Depth Model
Bowl
into Microwave
Close Microwave
Total
Fork
Fork
Plate
Dump Plate
Plate
Total
Sim (D435-View)
None
43/50
33/50
32/50
30/50
40/50
28/50
47/50
45/50
33/50
21/50
None
0/30
0/30
0/30
0/30
0/30
0/30
0730
0/30
D435
PrompiDA
PricetAA
16/30
11/30
8/30
5/30
7/90
0230
1130
7730
30/30
17/30
30/30
16/30
7/30
1/30
0730
2/30
(V.0
6/3
(X30
1/30
CDM-D435
29/30
26/30
26/30
26/30
30/30
30/30
15:30
14/30
14(30
14/30
CDM-L515
22/30
16/30
0730
Sim (Lew)-View)
None
43/50
34/50
37/50)
32/50
40/50
26/50
46/50
43/50
31/50
None
0:30
(U30)
0X30
D/30
0/30
0/30
0/30
0/30
(13))
(X30
LS15
PromptDA
PriceDA
3/30
(130
0X50
D730
3/30
(X50
3/30
0730
0/30
0X30
CDM-D435
22/30
11/30
9/30
9/30
13/30
11/30
11/30
10030
9/30
CDM-L5515
25/30
18/30
18/30
18/30
24/30
24/30
22/30
22/30
22/30
22/30
4.3 ZERO-SHOT SIM-TO-REAL MANIPULATION
Robot and task setup. We construct our sim-to-real pipeline using tabletop UR5 robot arm equipped with a Robotiq gripper, as illustrated in Fig. 3bb As mentioned before, the visual observa- tion of the policy is the depth image from a single third-view camera. We design two long-horizon manipulation tasks: the kitchen task and the canteen task. 1) The kitchen task tests the ability to utilize articulation objects: the robot is required to pick up a bowl the table, put it into the mi- crowave, and then close the door of the microwave. Note that the microwave door is glass, which is seen as hole from the original camera depth, and poses additional challenge for depth capturing. 2) The canteen task requires recognizing and accurately grasping a slim fork and thin plate, which rather noisy, and the fork is even unseen from the original camera depth: the robot should pick up the fork and put it into the box in front of it, then pick up the plate and dump the trash into the left box, at last place the plate into the front box. We collect 680 demo trajectories for the kitchen task and ~ 800 for the canteen task in simulation, train policy by imitation learning, and directly deploy the policy in the real world by plugging in depth model. The test setup illustrated in Fig. 33d and Fig. 31 where we test 10 positions and each for 3 times, resulting in 30 tests in total for both tasks. the real world, we test two cameras: one is the RealSense D435 (IR Stereo) camera, and the RealSense L515 (D-Tof lidar) camera. For both cameras, we gather their intrinsics and calibrate the extrinsics using the method as mentioned Section D
The zero-shot sim-to-real results are collected in Tab. 2 where we compare the policy per- formances using our CDMs against the same policy using two state-of-the-ar prompt-based depth models, and directly using the raw depth image. We have several interesting observations: 1) The CDM works better under their specific camera type, although in the previous section, the CDMs showed generalization under the depth metric on static dataset. 2) CDMs work better than previ- ous baselines, even on different camera types, showing the advantage of the training dataset and the structure design. 3) The real-world policy performance matches the simulation performance, and some is even higher. The reason may be that the randomized position in the simulation is bigger, and some of them are difficult to complete all tasks.
Results.
Total latency. On a single RTX 4090 with a RealSense D435 providing the prompt depth. we measure end-to-end latency (pre-processing
Float32 model inference + post-processing) for different depth models: D3RoMa (Wei et al. 2024) (1.531+0.004:, PriorDA (0. 0..055)) PromptDA (0. 000005,s,,,,s and CDMs (0.15T=0.0022 timization or quantization, CDMs achieve the lowest latency, enabling control at over 6 Hz. In comparison, FoundationStereo on the Zed camera (limited in stereo RGB cameras) only achieves Hz(0.319-0.002s).
Without any additional engineering op-
5 CONCLUSION AND FUTURE WORKS
This work introduces camera depth models (CDMs), designed to provide high-quality geometric information by enhancing depth perception for specific depth cameras. By delivering accurate depth predictions, CDMs enable robust robotic manipulation in real-world settings with accurate geometry information, effectively bridging the sim-to-real geometry gap. Integrating CDMs with real-world depth cameras, we successfully transferred depth-only visuomotor policies, trained solely in sim-

---

Published as a conference paper at ICLR 2026
ulation, to real robots long-horizon manipulation tasks, achieving high success rates. These results underscore the critical role of accurate geometric information, provided by CDMs, in en- abling generalizable and effective robotic manipulation. Although this study demonstrates CDMs in a depth-only imitation and sim-to-real pipeline, their potential extends far beyond this application. Future work could consider leveraging CDMs to relahel RGB-D data, enhancing policies with robust 3D representations. Moreover, by achieving simulation-level 3D perception in the real world and aligning sim-real geometry gaps, CDMs enable seamless integration of simulation and real-world 3D data. This approach could lead to more efficient data utilization strategies, fostering the develop- ment of large-scale robotic foundation models with human-level generalization ability for complex manipulation tasks.
ACKNOWLEDGEMENTS
We would like to thank all members of the ByteDance Seed Robotics team for their support through- this project. We also want to extend gratitude to Tao Wang for his kind support of the depth camera devices and to Hang Li for his leadership of this team.
REFERENCES
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. no. A lasion-laguuage-actton flow model for general robot control. arXiv preprint 2rXiv:2410.24166 2024.
Linghao Chen, Yuzhe Qin, Xiaowei Zhou, and Hao Su. Easyhec: Accurate and automatic hand-eye calibration via differentiable rendering and space exploration. IEEE Robotics and Automation Letters, 1)124333447 2023.
Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309. 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, page 02783649241273666 2023.
Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching with- out wn-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024.
Qiyu Dai, Jiyao Zhang, Qiwei Li, Tianhao Wu, Hao Dong, Ziyuan Liu, Ping Tan, and He Wang. depth simulation and restoration for perceiving and grasping
Domain randomization-eniee specular and transparent objects. In European Conference on Computer Vision (ECCV), 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and tem- poral domains. IEEE Transactions on Robotics, 3965):392-33945 2023.
Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning (CoRI.), 2024.
Alessandro Gasparetto and V Zanotto. A new method for smooth trajectory planning of robot manipulators. Mechanism and machine theory, 42(4):455 471, 2007.
Alessandro Gasparetto and Vanni Zanotto. A technique for time-jerk optimal planning of robot trajectories. Robotics and Computer-Integrated Manufacturing, 24(3):415-426, 2008.
Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, and Adrien Gaidon. Towards zero-shot scale-aware monocular depth estimation. arXiv, 2023. In ICCV, pages 9233-9243.
10

---

Published as a conference paper at ICLR 2026
Xiaoshen Han, Minghuan Liu, Yilun Chen, Junqiu Yu, Xiaoyang Lyu, Yang Tian, Bolun Wang, Weinan Zhang, and Jiangmiao Pang. Re Rimsim: Generating high-fidelity simulation data via 3d- photorealistie real-to-sim for robotic manipulation. arXiv preprint arXiv:2502.08643, 2025.
Jesse Haviland, Niko Sünderhauf, and Peter Corke. A holistic approach to reactive mobile manipu- lation. IEEE Robotics and Automation Letters, 7(2):3122-33) 2022.
Kaiming He, Jian Sun, and Xiaoou Tang. Guided image filtering. IEEE transactions on patlern analysis and machine intelligence, 35(6):1397-1409) 2012.
Tairan He, Chong Zhang, Wenli Xiao, Guanqi He, Changliu Liu, and Guanya Shi. Agile but safe: Learning collision-free high-speed legged locomotion. arXiv preprint arXiv:2401.17583, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2024.
Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, and Lirui Wang. Gensim2: Scaling robot data generation with multi-modal and reasoning llms. arXiv preprint arXiv:2410.03645, 2024.
HyunJun Jung, Patrick Ruhkamp, Guangyao Zhai, Nikolas Brasch, Yitong Li, Yannick Verdie, Jifei Song. Yiren Zhou, Anil Armagan, Slobodan Ilic, et al. On the importance accurate geometry data for dense 3d vision tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 780-791, 2023.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source assion-lngguage-attin model. In Conference Robot Learning, pages 2679-2713. PMLR, 2025.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems (RSS), 2021.
Hang Lai, Jiahang Cao, Jiafeng Xu, Hongtao Wu, Yunfeng Lin, Tao Kong, Yong Yu, and Weinan Zhang. World model-based perception for visual legged locomotion. arXiv preprint arXiv:2409. 1677444409.66784 2024.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models effective robot imitators. In ICLR, 2024a.
Xinhai Li, Jialin Li, Ziheng Zhang, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Kuo-Kun Tseng, and Ruiping Wang. Robogsim: A real2sim2real robotic gaussian splatting simulator. arXiv preprint ar arXiv22411.1183,, 2024b.
Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference Robot Learning, pages 112-123. PMLR, 2022.
Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng. Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng. Xiaowei Zhou, and Bingyi Kang. Prompting depth anything for 4k resolution ac- curate metric depth estimation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17070-17080, 2025.
Minghuan Liu, Zixuan Chen, Xuxin Cheng, Yandong Ji, Rizhao Qiu, Ruihan Yang, and Xiaolong Wang. Visual whole-body control for legged loco-mamipulation. The 8th Conference on Robot Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Haozhe Lou, Yurong Liu, Yike Pan, Yiran Geng, Jianteng Chen, Wenlong Ma, Chenglong Li, Lin Wang, Hengzhen Feng, Lu Shi, et al. Robo-gs: physics consistent spatial-temporal model for robotic arm with hybrid representation. arXiv preprint arXiv:2408. 14873 2024.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023.
NVIDIA. Nvidia isaac sim, 2021. BRLLLttps://deeeepperr. www.eer.iiiii... com/ www.o///amssiiniia.aa.a
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 20/8 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. arXiv, 2024. In CVPR.
Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, and Ab- hishesh Silwal. Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting. arXiv preprint arXiv:2409. 10161, 2024.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12189-1288, 2021.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021, 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv: /22 2024.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
Jun Wang, Yuzhe Qin, Kaiming Kuang, Yigit Korkmaz, Akhilan Gurumoorthy, Hao Su, and Xiao- long Wang. Cyberdemo: Augmenting simulated human demonstration for real-world dexterous manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17952-17963, 2024a.
Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu. Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. arXiv preprint arXiv:1912.0966 2019.
Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for domain images with optimal training supervision, 2024b. URL URLnttps://axxv..oo wwwpaxiwwi.oo..aaaa.aa wbs/ 2410. 19115
Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Unlocking accurate monocular geometry estimation for apma-d images with optimal training supervision. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5261-5271, 2025a.
Moge:
12

---

Published as a conference paper at ICLR 2026
Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. Moge-2: Accurate monocular geometry with metric scale and sharp details, 2025b. URLhttps: /ttetps://arii...o org/ abs 200./2250... 02546
Zehan Wang, Siyu Chen, Lihe Yang, Jialei Wang, Ziang Zhang, Hengshuang Zhao, and Zhou Zhao. Depth anything with any prior. arXiv preprint arXiv:2505. 10565, 2025c.
Songlin Wei, Haoran Geng, Jiayi Chen, Congyue Deng, Cui Wenbo, Chengyang Zhao, Xiaomeng Fang, Leonidas Guibas, and He Wang. D'roma: Disparity diffusion-basee depth sensing for material-agnoste robotic manipulation. In ECCV 2024 Workshop on Wild 3D: 3D Modeling, Reconstruction, and Generation in the Wild, 2024.
Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foun- dationstereo: Zero-shot stereo matching. arXiv, 2025.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In JCLR, 2024.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang. Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Yufei Xue, Wentao Dong, Minghuan Liu, Weinan Zhang, and Jiangmiao Pang. A unified and general humanoid whole-body controller for fine- ane-grained locomotion. In Robotics: Science and Systems (RSS), 2025.
Ge Yan, Yuch-Hua Wu, and Xiaolong Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint arXiv:2403.0411 2024.
Lihe Yang, Bingyi Kang, Zilong Huang. Xipark2004ddephhaggan Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-21911, 2024b.
Wei Yin, Jianming Zhang, Oliver Wang. Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. arXiv, 2023. In CVPR, pages 9043-9053.
Yanjie Ze, Ge Yan, Yuch-Hua Wu. Annabella Macaluso, Yuying Ge. Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In Conference on robot learning, pages 284-301. PMLR, 2023.
Yanjie Ze, Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng, and Jiajun Wu. Generalizable humanoid manipulation with 3d diffusion policies. arXiv preprint arXiv:2410. 10800, 2024a.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In Proceedings of Robotics: Science and Systems (RSS), 2024b.
Xiaoshuai Zhang, Rui Chen, Ang Li, Fanbo Xiang, Yuzhe Qin, Jiayuan Gu, Zhan Ling, Minghua Liu, Peiyu Zeng, Songfang Han, Zhiao Huang. Tongzhou Mu, Jing Xu, and Hao Su. Close the op- tical sensing domain gap by thysics-grounder active stereo sensor simulation. IEEE Trunsactions on Robotics, (9(3):2429-244 2023. doi: 10.1 00.1109/TR0.2023335555
Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine fine-graine iieegrained bimanual ma- nipulation with low-cost hardware. Robotics: Science and Systems XIX, 2023.
13

---

Published as a conference paper at ICLR 2026
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A saiio-language-actionn generative world model. arXiv preprint arXiv:2403.00631, 2024.
Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces on robot learning. Advances in Neural Information Processing Systems, 37:77799-77830, 2024.
Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL), 2023.
Ziwen Zhuang, Shenzhe Yao, and Hang Zhao. Humanoid parkour learning. In 8th Annual
Conference on Robot Learning, 2024. URL https:://ppnree.ee...ett net. formm
Es7 Es71a3FqUM
14

---

Published as a conference paper at ICLR 2026
A DECLARATION OF LLM USAGE
Large Language Models (LLMs) were used in the preparation of this submission. Specifically, they assisted in editing and polishing the writing for grammar and clarity. All technical ideas, experimental designs, and results were developed by the authors. No LLMS were used to generate research content or experimental results.
B EXTENDED RELATED WORK FOR VISUAL SIM-TO-REAL
Sim-to-real transfer requires policies to overcome both the observation gap and the physics gap between simulated and real-world environments. Physics gaps, such as discrepancies in dynamics or friction, are often addressed by using domain randomization (Tobin et al. (017 Peng et. 2018).
which trains policies generalize across a range of physical parameters that encompass real-world variability. While this method is generally effective for locomotion tasks in legged robots (Xue) al. 2025 Kumar al., 202T), manipulation tasks require more precise modeling because they depend heavily on accurate visual observations of objects, typically provided by RGB and/or depth images. Achieving robust sim-to-real policy transfer for manipulation tasks using RGB images requires high-fidelitg simulation rendering to minimize the visual gap. Relying solely on simulator- generated images often demands extensive curriculum and augmentation design to ensure that the learned policy is effectively transferred to the real world (Wang al. 2024a). Although advances in simulator rendering technologies (Xiang et al. 2020 NVIDIA 2021 | can help, recent real-to-sim approaches using neural rendering techniques demonstrate that reconstructing photorealistic scenes
from real- world data can further reduce the visual gap (Han et al. 2025 Lou et al. 2024 Qureshi
et al. 2024 Li al. 2024b).
In addition to RGB images, some works utilize colorless 3D representations, such as point clouds and depth images, to reduce visual discrepancies between simulation and reality. For example, He et al. (2024) predicts the ray distances from simulated depth images to learn the reach-avoid value networks; Cheng et al.] (2023). Zhuang al.] (2023), Zhuang et al. (2024), and Lai et al. (2024) employ depth images to train quadrupeds for collision avoidance and high dynamic locomotion; and Liu al. (2024) uses depth images from two camera views for mobile manipulation tasks on quadruped robot, which requires object segmentation to narrow the sim-to-real gap further. These approaches typically add noise and augmentations simulated depth images and require post-processing of real-world depth data, such as clipping, hole filling, and temporal filtering, to address sensor imperfections. Alternatively, Hua al. (2024) uses point clouds as visual input, but still adds noise in simulation and applies cropping and downsampling in the real world. Besides, Zhang et al. (2023) introduces a computation-cost method to simulate the depth with typical noise patterns rendered by real-world stereo camera. However, simulating real-world camera adding noise in the simulation is last resort, as it may deteriorate the rich geometry information and precise manipulation.
C DATASET
We design a multi-camera mount device to achieve highly efficient data collection, so that we can capture data simultaneously, as shown in Fig. [ In detail, we mount seven cameras, including five RealSense cameras (D405, D415, D435, D455, L515), a ZED camera, and an Azure Kinect camera. For the ZED camera, we record the raw data and replay it with its 4 modes (performance, ultra, quality, neural) offline.
The collected dataset contains more than 17,000 images for each camera, sam- pled from videos at 5Hz, covering different scenes, including kitchens, living rooms, markets, bedrooms, bathrooms, offices, and breakrooms, as shown in Fig.
D SIM-TO-REAL MANIPULATION THROUGH CDMS
We describe the details of the four stages in sim-to-real manipulation pipeline with CDMs.
15

---

Published as a conference paper at ICLR 2026
Kitchen
Living Room
Market
Bedroom
Bathroom
Office
Breakroom
Scene
RGB
Depth
RealSense D405
RealSense D415
RealSense
RealSense
RealSense LS15
ZED 21
D435
DAS5
Kinect
3.79
9%
10.%%
Bathrcom
Becroom
ZED Modes
Living Room
Depth
13.7%
Kitchen
13.6%
36.9%
Office
Markat
Quality
Figure 4: Illustration of the collected ByteCameraDeptti datasets, which contains the raw depth data from cameras, 10 modes (including modes the ZED 2i camera) in 7 different scenes.
D655
D405
D435
D415
L515
ZED
Camera Mount Device
Figure 5: Multi-camera mount device for capturing the color-depth image pairs from multiple depth cameras all at once. In practice, we use two computers to capture all data due to the USB bus bandwidth limits.
Scene construction. Since we rely on depth-only visual transfer, without any color information, the geometry-based sim-to-real pipeline does not require rigorous alignment of the exact appear- ance between objects and backgrounds in the simulation and the real world. Instead, we introduce geometrically similar objects and construct simple geometry as the background in the simulation. After building the environment, we set up the camera to obtain visual observations. We note that manipulation also requires reasonable approximation of interactions between robots and objects, yet this is beyond the focus of this project. Thereafter, we manually assign the physical attributes of objects and modify the open-source robot description files to ensure plausible interactions, without aiming for absolute physical accuracy.
Camera alignment. To align the camera pose between the simulation and the real scene, we adopt differentiablle-enneiiin camera calibration method Chen et al. (2023) estimate the cam-
16

---

Published as a conference paper at ICLR 2026
extrinsics in the real scene with minimal human effort, which only requires a few corresponding masks of the robot arm between real and virtual scenes. However, the calibrated poses are hard to perfectly align due to the differences in camera models between simulation and real-world sensors. To mitigate the gap from such misalignment, we slightly randomize camera poses during data col- lection in simulation, which helps the policy be robust small discrepancies in viewpoint and better work in real-world deployments.
Data generation. To generate demonstrations efficiently in simulation, we develop an extension of MimicGen (Mandlekar et al. 2023), with whole-body control (WBC) (Haviland al. 2022 gen- erate smoother, high-quality demonstrations. Our method introduces several data generation features like random reset generate retrying demonstrations, controllable velocity for fine-ad ane-adjustment, and so on. With WBC, the algorithm can be further extended to wheeled robots for mobile manipulation tasks. Details of the algorithm can be further referred to Appendix M
Imitation learning with depth. We adopt policy structure similar to the one used in Hua et al. (2024) and Lin et al. (2025). Although some previous works have shown some robustness using a point cloud based policy (Ze al., 2024b Hua al., 2024). their point clouds are also trans- formed from the depth image captured by the depth camera and require cropping and downsampling to alleviate noises. Since now we can obtain an accurate depth where the geometry information
complete, choose use the depth image directly the input to policy. The policy encodes the depth image with a pre-trained ResNet, where the first layer the network is replaced with a I-channel convolutional layer; the proprioceptive states are encoded by from-scratch MLP: dif- fusion head (Chi et al. 2023 (o et al. 2020) is adopted to predict the action sequence. We directly the one-step single-vier view depth image rendered in simulation for training the policy, without adding any noise, but with only the RandomSh Stacalllllllc SccleRotate augmentation to alleviate the camera calibration biases. Besides the depth image, the observation space also includes the joint position and the gripper status. In real-world deployment, we make our CDM plugin between the camera and the policy, which predicts clean depth image based on the raw depth image and the RGB image from the depth camera. The predicted depth image is then used for real-time policy inference.
E DEPTH-ONLY IMITATION LEARNING SETUP AND RESULTS
The real-world depth-only imitation experiments are conducted on tabletop UR5 robot arm with Robotiq gripper, using depth observations from single RealSense D435 camera. The two pilot tasks are Stack -owly (pick bowl, stack bowl) and Touthpaste-and-CCu (pick toothpaste, put toothpaste into cup). The two tasks are illustrated in Figure which depicts the success state of their subtasks. For each task, we collect 50 telcoperated demonstrations, and evaluation is performed across five different positions with three trials per position. In the Stack-Bowi task, the policy is trained on one normal-sized bowl and tested on five different sizes, including four unseen ones, to examine generalization.
(a) Stack-bowls task.
(b) Toothpaste-and-ee task.
Figure 6: The experiment setup of the depth-only imitation learning tasks.
17

---

Published as a conference paper at ICLR 2026
Results show that CDM-processee depth substantially improves success compared to depth (Table 3). Without CDMs, the robot almost entirely fails Toothpaste-ans and-Cup and achieves low success in Stack-Bowls, With CDM-D435, success rates increase markedly across both tasks. More- over, the CDM-enabled policy generalizes to bowls of unseen sizes, consistently achieving non-zero success across all four novel sizes, whereas the baseline without CDMs fails completely (Fig. 7
Success Rotte
Table 3: Dnpy imitation results with and without CDMs (15 trials per subtask; 50 demos per task for training).
Stock bowt
DAT
0.13 0.06
0.M6
0.20
Medium (Trainl
Depth Moddl
Thodkpattr-nd---ng Cup
Stack-Bowll
Fig Figure Generalization over different bowl sizes. With CDM, success remains non-zero across unseen sizes; without CDM, success
Bostpaae
Psi Toctipaic into Cup
Stadi Rowl
0/15
3/55
CDM D435
10/15
6/15
11/5
W15
is zero.
F ADDITIONAL COMPARISONS WITH D3ROMA AND FOUNDATIONSTERIO
In this section, we provide additional comparisons between our CDMs, D3RoMa Wei et al. 2024 and Sterd (Wen al., 2025). We report quantitative results on the Hammer (Jung al. 2023) dataset only for D3RoMa, using the same splits and metrics as in Tab Note that Foundation- Stereo specifically designed for stereo RGB cameras (e.g., ZED), and Hammer does not provide stereo RGB image pairs, so it cannot be evaluated quantitatively on this benchmark. Instead, we in- clude qualitative comparisons with both D3RoMa and FoundationStereo on our ByteCameraDeephh dataset samples to illustrate how different methods behave on real scenes.
F.1 QUANTITATIVE COMPARISON WITH D3ROMA ON HAMMER DATASET
Tab. reports the quantitative comparison between CDMs and D3RoMa on the D435, L515 and Helios splits of the Hammer dataset. We use exactly the same evaluation protocol as Tab. [ in the main paper. Our CDMs consistently achieve better results than D3RoMa.
Split
Method
LI 4
RMSE +
AbsRel
80.5 1
81
Ours (CDM-D435)
0.0258
0.0404
0.0312
0.9842
0.9951
Ours (CDM-L515)
0.0182
0.0338
0.0217
0.9877
0.9956
D435 (IR Stereo)
D3RoMa
0.0338
0.0610
0.0432
0.9243
0.9740
Raw Depth
0.0550
0.1458
0.0708
0.9179
0.9543
Ours (CDM-L.155)
0.0156
0.0297
0.0229
0.9754
0.9919
Ours (CDM-D435)
0.0165
0.0349
0.0246
0.9613
0.9855
L515 (D-ToF)
D3RoMa
0.0282
0.0522
0.0428
0.9329
0.9652
Raw Depth
0,0312
0.08
0.0475
0.9098
0.9429
Ours (CDM-L515)
0.0248
0.0403
0.0334
0.9468
0.9871
Ours (CDM-D435)
0.0272
0.0457
0.0372
0.9297
0.9806
Helios (I-ToF)
D3RoMa
0.0400
0.0612
0,0523
0.8934
0.9690
Raw Depth
0.0312
0.0813
0.0475
0.9098
0.9429
Table 4: Additional comparison with D3RoMa the Hammer dataset. We report results on the three camera splits (D435, L515, Helios) using the same metrics and protocol as Tab. [ Our CDMs achieve better geometric accuracy than D3RoMa on all splits.
F.2 QUALITATIVE COMPARISONS ON BYTECAMERADEPTH DATASET
We further provide qualitative comparisons on our ByteCameraDepth dataset. Specifically, we vi- sualize both depth maps and reconstructed point clouds for CDMs, D3RoMa and FoundationStereo.
18

---

Published as a conference paper at ICLR 2026
compare CDMs with D3RoMa. The former shows depth map comparisons, while the latter shows the corresponding reconstructed point clouds. Similarly, Fig. T0 and Fig. [I compare CDMs with FoundationSterro in terms of depth maps and point clouds, respectively.
together with the sensor outputs. Fig. 8 and Fig. 2
Camera RGB
Camera Depth
D3ROMA Depth
CDM-D435 Depth
Figure 8: Depth prediction comparison between CDMs and D3RoMa.
Rendered from depth
Rendered from the model depth predicted by
Rendered from the model depth predicted by
Figure 9: Reconstructed point cloud comparison between CDMs and D3RoMa.
19

---

Published as a conference paper at ICLR 2026
Camera RGB
Camera Depth
FoundationSteree Depth
CDM-Zed2I-Quallty Depth
Figure 10: Depth prediction comparison between CDMs and FoundationSterco.
Rendered from the
Rendered from the model depth predicted FoundationStereo
Rendered from the model depth predicted by ZDM-ZedZi-Qualiyy
Figure 11: Reconstructed point cloud comparison between CDMs and FoundationStereo.
20

---

Published as a conference paper at ICLR 2026
Camera RGB
Camera Depth
Depth
Camera RGB
Camera Depth
Depth Diff
Depth
Depth
D435
Kinect
LS15
C400
Zed2i Neural
Zed2i Quality
Figure 12: Depth predictions visualization of extended camera depth models, including CDM- D435, CDM-D405, CDM-L515, CDM-Kinect, and CDM-Zed2i (Neural & Quality mode) on their corresponding camera captures. Note that the depth difference visualized in range of 0-0.2m.
G EXTENDED CAMERA DEPTH MODELS
Except for the two daily-used camera depth models (CDMs) used in the robot manipulation ex- periments, i.e., RealSense D435 and RealSense I.515, further train three CDMs for RealSense D405, Azure Kinect, and Zed2i (Neural mode). Since we do not have the corresponding real-world dataset with the ground-trui truth depth label quantitatively evaluate them, we simply visualize their predictions on representative scenes from the ByteCameraDepth dataset, shown in Fig. [2] Since RealSense D435 uses a similar depth technology to RealSense D415 and RealSense D455, they share a similar noise mode, so we visualize CDM-D435 predictions upon these two cameras and find that CDM-D435 can also provide good predictions. We open all model weights to allow the community to further test and improve them.
21

---

Published as a conference paper at ICLR 2026
Value Noise Filter
Depth Diff w/ Filter
Value Noise w/ Filter Hole Noise
Camera RGB
Value Noise
Depth Diff
Hole Noise
GT Noise
D405
D435
L515
Figure 13: Synthesized noise of RealSense D405, D435 and L,55.
H VISUALIZATION OF SYNTHESIZED NOISE
We illustrate the synthesized noise samples from the simulation datasets in Fig. 13 and Fig. 14 pro- duced by the noise models, including the hole noise and the value noise, learned from the collected ByteCameraDepth dataset. We observe that the noise models leam typical noise patterns of the depth camera, e.g., the failures on transparent objects. Additionally, the guided filter fills the seale gap between the value noise and the ground truth.
22

---

Published as a conference paper at ICLR 2026
Value Noise w/ Filter
Depth Diff w/ Filter
Value Noise w/ Filter + Hole Noise
Camera RGB
Value Noise
Depth Diff
Hole Noise
Noise
Kinect
Zed2i Quality
Zed2i Neural
Figure 14: Synthesized noise of Azure Kinect and modes of ZED2i.
23

---

Published as a conference paper at ICLR 2026
1 DEPTH ACCURACY W.R.T DISTANCE
Before leaving the factory and being sold to customers, depth camera will evaluated at various distances fully examine its desired working range. To understand the work range of CDMs and as a reference to help people use them, we also provide the depth accuracy w.r.t the distance of CDMs (CDM-D435 and CDM-L515) on the Hammer dataset, in terms of absolute and relative error and the LI eтror, shown in Fig. [S From the accuracy curves, we observe that the raw depth has larger error than the camera producer claims, for example, the RealSense D435 should have less than 2% error rate when working under 1~2 meters, which may be the bias of the dataset. Upon this dataset, CDMs can achieve high accuracy, whose trend follows the accuracy of the original prompted depth.
24

---

Published as a conference paper at ICLR 2026
CDM-L515
CDM-D435
Raw Depth
Absolute Relative Error
L1 Error
2000
2000
1.000
1,000
0.700
0.700
0.500
0.00
Volue (Log Scale)
0.300
Value lLog Scale!
0300
0.00
0,100
0.070
0.050
0.050
0.030
0.030
0.010
Depth Range
Depth Range
(a) Depth accuracy on D435 data split.
Absolute Relative Error
L1 Error
1.000
1,000
0.700
0500
Value (Log Scale)
0.300
0.000
0150
Volue LLog
0.100
0.070
0.030
0.030
0.000
0.000
0.10
0.010
0.07
0.005
0.003
10
15
2.0
25
0.5
10
20
25
Depth Range
Depth Range
(b) Depth accuracy L515 data split.
Absolute Relative Error
L1 Error
1.000
0.700
0.500
(Log Scaiel
0.300
0.300
0.200
0.000
0150
0.100
0.070
0.070
0.050
0.000
0.200
0.00
0.007
0.055
0.005
0.033
05
15
2.0
0.5
10
25
Depth Range
Depth Range
(c) Depth accuracy Helios data split.
Figure 15: Depth accuracy evaluation of CDMs and other models on the Hammer dataset.
25

---

Published as a conference paper at ICLR 2026
J DEPTH COMPARISON
We provide a detailed visual comparison between the raw camera depth and the predicted depth by the proposed CDMs, shown Fig. [6 We can easily observe that both of these two representative depth cameras have their typical noise and failure modes. For example, both cameras fail to recog- nize the glass of the microwave and the metal fork; D435 has noisy depth on the plaid tablecloth; L515 has problems with the reflective outside part of the microwave, and the gripper fingers. In comparison, CDMs can provide accurate and complete geometry information.
RGB
image
from (d) RGB
image from
(g) RGB
image
from
(j) RGB image from L,55,, Canteen.
D435, Kitchen.
L515, Kitchen.
D435, Canteen.
(b) Raw depth from D435, (e) Raw depth from L515, (h) Raw depth from D435, (k) Raw depth from L515, Kitchen.
Kitchen.
Canteen.
Canteen.
(c) Depth by CDM-D435, (0) Depth by CDM-L515, (i) Depth by CDM-D435, (I) Depth by CDM-L515, Kitchen.
Kitchen.
Canteen.
Canteen.
Figure 16: Detailed real-world cases of two representative depth cameras, RealSense D435 (active IR stereo camera) and RealSense L515 (lidar camera), including color images (first row), camera depth images (second row), and depth predicted by camera depth models proposed this project (third row).
26

---

Published as a conference paper at ICLR 2026
K RENDERED POINT CLOUD COMPARISON
We further compared the rendered point clouds transformed from the raw camera depth and the ones predicted by the camera depth models (CDMs), shown fig:ed-reell (two real-world imitation tasks), Fig. 18 and Fig. 19 (two sim-to-real tasks), From the rendering results, we can easily observe that the point clouds rendered by the raw depth camera are much noisier, where the objects are distorted and convey wrong geometry information. In comparison, the CDM provides clean point cloud where the objects maintain most of their original geometry. It is worth noting that in the Canteen task, the geometry of the fork from the raw camera depth is integrated within the plate; the one predicted by the CDM is better, but still inaccurate. This is because the raw camera depth does not provide any useful information about the fork, and the model has to predict the whole from the semantic information of the color image, which may be confusing. We encourage readers to further visit the project page for more interactive point cloud rendering demos
27

---

Published as a conference paper at ICLR 2026
Rendered from the raw camera depth
Rendered from the model depth predicted by CDM-D435
Point cloud comparison of the Stack Bowl task.
Rendered from the raw camera depth
Rendered from the model depth predicted by CDM-D435
(b) Point cloud comparison of the Toothpaste task.
Figure 17: Rendered point cloud comparison on two real-world imitation tasks between the raw camera depth and the predicted depth of CDM-D435, upon the RealSense D435 camera.
28

---

Published as a conference paper at ICLR 2026
Rendered from the camera depth
Rendered from the model depth predicted by CDM-D435
(a) Point cloud comparison of the Kitchen task.
Rendered from the raw camera clepth
Rendered from the model depth predicted by CDM-D435
(b) Point cloud comparison of the Canteen task.
Figure 18: Rendered point cloud comparison sim-to-real tasks between the raw camera depth and the predicted depth of CDM-D435, upon the RealSense D435 camera.
29

---

Published as a conference paper at ICLR 2026
Rendered from the raw camera depth
Rendered from the model depth predicted by CDM-L515
(a) Point cloud comparison of the Kitchen task.
Rendered from the raw camera depth
Rendered from the model depth predicted by CDM-L515
(b) Point cloud comparison of the Canteen task.
Figure 19: Rendered point cloud comparison on sim-to-real tasks between the raw camera depth and the predicted depth of CDM-L5155, upon the RealSense 1.515 camera.
30

---

Published as a conference paper at ICLR 2026
Kitchen Rollout D435
Kitchen Rollout L515
Sim
Camera Depth
Camera RGB
Real
Camera
Depth
Model
Depth
Canteen Rollout
D435
Canteen Rollout L515
Camera RGB
Sim
Depth
Real
Depth
Depth
Figure 20: Sim-Real rollouts comparison on two tasks in the sim-to-real experiments, including two views (D435 view and L515 view). For the simulation, show the rendered RGB and depth images; as for the real experiments, we visualize the RGB images and the depth images from the depth camera, with the predicted depth from the corresponding camera depth model.
L COMPARISON OF SIM-REAL ROLLOUTS
We visualize the policy rollouts on two tasks in the sim-to-real experiments in Fig. 20 where we compare the key frames from the simulation and the real world separately. It is readily apparent that the camera depth model provides high-quality, simulation-like depth, offering accurate geometry information in the real world and thus bridging the geometry gap between simulation and reality.
31

---

get more safe, smooth and Published as a conference paper at ICLR 2026
M DATA GENERATION WITH WBCMIMICGEN
M.1 ALGORITHM
To generate demonstrations efficiently in simulation, inspired by Haviland et al. (2022), we pro- pose WBCMimicGen, which extends with whole-body control (WBC). Compared to the original MimicGen algorithm, which utilizes linear interpolation of end-effector poses generate trajecto- ries, WBCMimicGen optimizes target joint velocities with WBC, considering the manipulability, joint limits, joint velocity limits together the QP problem and thereby generating smoother, high- quality demonstrations. This approach can be further extended to wheeled robots for mobile manip- ulation tasks. In the simulation, the advantage of precise perception, without considering any error, helps utilize classical control methods, such as WBC, and thus controllable trajectories.
Specifically, regard the data generation problem solving trajectory of whole-body joint velocities that enables the end-effector to move with a specific velocity. Formally, denote joint velocities æ, this problem be modeled as quadratic programming (QP) problem (Haviland et al. 2022):
min fo(x) = 2"
Qx - CT
(10)
subject to Jx Ove.
Ax < B
X S x xx
where (Abaser Qactive; 81, 62, 6.) and x is the limits; Obase is the velocities of the robot base; Qactive is the velocity of the joints related to the end-effectors in the QP (so called the active joints): 6; are slack variables that can help construct solvable QP. Without loss of generality, suppose there are end-effectors and joints, these variables can be expressed as:
@ = diag(>a: A61.-.. Aoe) E ((n+6k)
= Jm +e
E R((++6))
O
A (Inx(x+6k)) (2=x(++6)))
(II)
Og
Pi Pi-Ps
pp PP=
Pe-Pa
Here jm is the manipulability Jacobian, c is the base to end-effector angle, and p is the distance to the nearest joint limit, encouraging the joint not to stay too close to the limit.
M.2 COMPARISON RESULTS
To evaluate the data quality generated by WBCMimicGen, we compare the trajectory smoothness, measured by the mean absolute acceleration and the root mean square (RMS) jerk (i.e., the averaged rate of acceleration change) metrics against the data generated by the original MimicGen Mandlekar et al. (203)) algorithm. Previous works Gasparetto and Zanotto (2007 2008) use metrics like these as objectives for better smoothness. As shown Tab. 5] W WBCMimieGen consistently generates smoother trajectories with significantly lower acceleration and jerk values across all joints. This improvement stems from the quadratic programming formulation WBC, which incorporates ve- locity regularization and enforces joint velocity limits. We encourage readers further visit the project page for a direct visual comparison of the generated trajectories.
Simulation experiments further validate these improvements. As detailed in Tab. models trained on WBCMimieGen data achieve higher success rates (72% vs 56% for Kitchen, 42% vs 24% for Canteen) while maintaining substantially lower RMS jerk and acceleration. The baseline approach
32

---

Published as a conference paper at ICLR 2026
Table Comparison of the generated demonstrations over every robot joint (UR5).
Joint 1
Joint 2
Joint 3
Joint
Joint 5
Joint
Task
Method
Mean absolute acceleration (rad/s")
MimicGen
1.284
1.253
0.910
8.115
4.457
6.702
Kitchen
WBCMimieGen
0.183
0.495
0.311
4.925
2.810
6.539
MimicGen
1.180
1.792
1.310
6.487
2.961
1.177
Canteen
WBCMimicGen
0.028
0.064
0.080
2.200
0.433
0.709
RMS jerk (rad/s")
MimicGen
461.374
313.714
277.185
1047.850
852.199
1252.070
Kitchen
WBCMimicGen
58.774
112.983
84.063
767.230
516.074
1020.545
MimicGinn
435.487
384.092
425.695
1004.206
592.221
282.824
Canteen
WBCMimicGen
8.610
19.879
25.812
547.281
111.007
207.548
exhibits much larger acceleration at action chunk boundaries, which increases the likelihood of dropping objects. In contrast, WBCMimicGen's 's smoother trajectories enhance both task reliability and safety, making them more suitable for real-world deployment.
MimicGen
WBCMimicGen
3000
2688
10a
1040.6
Success Rate
0.72
(rad/s")
294.1
0.6
0.56
2000
Acceleration
10*
19.5
0.4
101
0.2
572
2.8
204
100
Success
Rate
RMS Jerk
RMS Jerk
Acceleration
Acceleration
(a) Kitchen Task
MimicGan
WBCMimiccGe
Acceleration (rad/s")
239.1
0.8
Success Rate
Jork (rad/=")
10
0.6
1000
0.42
0.4
101
6.4
500
4.0
0.2
292
100
Success Rate
RMS Jerk
RMS Jerk
Acceleration
Acceleration
(b) Canteen Task
Figure 21: Performance comparison between policy models trained demonstrations gener- ated by MimicGen and our WBCMimicGen. on (a) Kitchen Task and (b) Canteen Task. The policy learned from WBCMimieGen reflects a higher success rate while keeping smooth on the rollout policy trajectory (lower RMS jerk and acceleration indicate).
33

---

Published as a conference paper at ICLR 2026
Figure 22: A typical failure case of CDM-L515, caused by the wrong prompt and the less infor-
mative semantic information contained in the RGB image.
N LIMITATIONS AND FAILURE CASES
Although CDMs can fix many errors of the source depth cameras due to the semantic information in the RGB image, they still easy be effected by the prompted depth image and fall into some failure cases when the monocular semantic information is not enough to fix the eπror. In other words, when the prompted camera depth image has wrong metries for large region, the predicted depth can be misled. Here we provide typical case CDM-L-515 in Fig. 22 where the red dashed line highlights the area where the prompted camera depth hints that a hole. This due the metal plane causes the failure perception for the RealSense L515 is LiDAR depth camera. Additionally, the RGB image does not bring informative semantic information for the CDM to fix that error.
34


Reviews:
[{"review_id": "tPmWBzptrI", "reviewer": "Reviewer_o2Ze", "summary": "The authors propose Camera Depth Models (CDMs): Lightweight neural plug-ins for existing depth cameras that convert raw RGB + noisy depth input into denoised, metric-accurate depth — effectively making real-world perception as accurate as simulation.\n\nDataset contribution: it proposes ByteCameraDepth, a real-world multi-camera depth dataset comprising >170,000 RGB-depth pairs.\n\nCompared with baseline methods, this pipeline presents better performance on benchmarks.\n\nThere are also some limitation and concerns regarding the generalization scope and model capability.", "questions": "Figure 5 in supp materials has typesetting issues.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CDMs act as post-camera modules, requiring no changes to robot control code or simulators. Fast (< 0.15 s latency) and generalizable across camera types.\n2. By producing metric-accurate depth aligned with simulation, CDMs remove the need for domain randomization and reduce reliance on real fine-tuning, unlocking large-scale sim-data reuse.\n3. Large real-camera dataset + multi-sensor evaluation. Demonstrates not only numeric improvement (RMSE ↓ 40 %) but task-level transfer (0 % → 90 % success).", "weaknesses": "1. Generalization Scope, There are multiple aspects for generalization, for example, scene generalization, robot generalization and so on. The model is trained on some collected dataset on specific robot and scenes, can it generalize to other scenarios? Please provide more analysis about zero-shot generalization capability.\n\n2. The previous method d3roma and FoundationStereo has different settings but also valuable to compare, can you compare the generalization capability of the proposed method and these two? especially on cases like transparent and specular area.\n\n3. Some template issues, like Fig.5 in supp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdBTMJoh6d", "reviewer": "Reviewer_5fyz", "summary": "This paper proposes to enhance the depth perception ability of different real-world depth sensor. This paper introduces the Camera Depth Models(CDMs) as a simple plugin on daily-use depth cameras, which is trained with high-quality paired data from simulation. Concretely, to generate paired depth image,  this paper first collects large-scale real-world depth-rgb images, and utilize these images to pretrain the hole noise model and value noise model to predict the low-quality depth image. With the hole noise model and value noise model, this method synthesizes camera depth for open-source simulation depth image to get paired simulation-real depth images to train the CDMs. The results prove the effectiveness of this method for depth-based manipulation policy sim-to-real adaptation.", "questions": "To what degree do these two noise models generate real-world-camera style depth images? For example, in the case of a transparent water cup, what effect does each of these models have?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper proposes a large-scale real-world RGB-Depth image pairs captured by seven depth cameras, which is a valuable depth-based dataset for depth prediction in the real world.\n\n2. This paper provides a good solution to generate paired simulation-real depth images. This model enables a simulation policy to be transferred directly to a real robot for downstream tasks, achieving a zero-shot sim-to-real transfer, by simply using the CDM to clean the real camera's noise.\n\n3. This paper provides sufficient experiments to prove that this model could achieve nearly simulation-level accuracy in depth prediction, making it possible to transfer a long-horizon simulation policy into the real-world manipulation task.", "weaknesses": "1. This article provides extensive visualization results and sim-to-real outcomes to validate the model's effectiveness, but it lacks some quantitative experimental results. For example, it lacks quantitative experiments on the predictive performance of the two pre-trained noise models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5PuZfYhL8", "reviewer": "Reviewer_ScyS", "summary": "This paper introduces Camera Depth Models (CDMs), a neural network plugin designed to improve the accuracy of depth readings from consumer-grade depth cameras (like RealSense). The authors develop a large multi-camera depth dataset (ByteCameraDepth) and a novel data engine to train the CDMs by modeling camera-specific noise in simulation. By generating accurate, \"simulation-like\" depth data in the real world, the CDMs effectively bridge the sim-to-real gap for geometric perception. The main contribution is demonstrating that a depth-only manipulation policy, trained purely on clean simulated depth data (without real-world fine-tuning or adding noise), can transfer seamlessly to real robots and achieve high success rates on challenging long-horizon tasks.", "questions": "1. CDM Robustness on Novel Scenes/Objects: The sim-to-real tasks test policies on randomized positions (Fig. 3), but the objects themselves seem consistent across the real and sim environments. How does the camera depth models perform when deployed on objects or scenes drastically unseen during its noise model training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-structured and easy to follow. The problem is clearly defined, and the figures (especially Figure 1 and 2) effectively illustrate the proposed architecture and the core contribution of bridging the sim-to-real geometry gap.\n2. This work provides a generalized, plug-and-play solution that allows researchers to fully leverage clean simulation depth without the current necessity of laborious real-world noise modeling or fine-tuning, dramatically lowering the barrier for generalizable, geometry-based robot policies.", "weaknesses": "1. In Section 3.4, subsection \"Value noise model\", the notation should be $N_{\\text{value}}$ and $\\hat{D}\\_{\\text{value}}$, instead of $N_{\\text{hole}}$ and $\\hat{D}\\_{\\text{hole}}$.\n2. In robotic manipulation, color information is also crucial, as in practical applications, we rarely rely solely on depth as input. I'm interested in experiments demonstrating how camera depth models benefit robot manipulation tasks when RGB information is included (i.e., using RGB-D data).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgz60npcko", "reviewer": "Reviewer_b9og", "summary": "This paper addresses the challenge of transferring robotic manipulation policies from simulation to the real world, identifying the noise and inaccuracies of real-world depth sensors as a primary obstacle.The authors propose \"Camera Depth Models (CDMs),\" a plug-in module designed to take raw RGB and noisy depth signals from a specific camera and output a clean, accurate, \"simulation-like\" metric depth map.", "questions": "1. The core of the paper focuses on the CDM models. The actual manipulation policy, which is critical to the paper's claims, is barely discussed in the main text. Section 4.3 briefly mentions \"imitation learning\" , but the specific architecture (a ResNet encoder with a diffusion head) is relegated to Appendix D. This makes the manipulation aspect of the paper feel like an afterthought and difficult to evaluate properly.\n2. The paper's title and abstract frame it as a manipulation paper, but the experimental validation for manipulation is limited. The zero-shot evaluation rests on two tasks (Kitchen and Canteen) presented in one table (Table 2). While the tasks are well-chosen, this is a small number from which to conclude that this method \"enables generalizable and effective robotic manipulation\"[L 880].", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a highly relevant and practical problem in robotics: the sensor-reality gap that plagues sim-to-real transfer. The proposed approach of explicitly de-noising the real-world perception to match the clean simulation environment, rather than the other way around, is a clear and interesting contribution.\n2. The introduction of the ByteCameraDepth dataset, which captures data from 7 different depth cameras across 10 modes and multiple scenes, is a valuable contribution to the community for studying and modeling depth sensor noise", "weaknesses": "1. **Missing Critical Baseline:** The paper's primary motivation is that depth noise breaks sim-to-real transfer. The proposed solution is to clean the real-world depth (Sim-Clean -> Real-Clean). However, the paper fails to compare this against the most obvious and widely-used baseline: making the policy robust to noise via domain randomization (i.e., Sim-Noisy -> Real-Noisy). It is common practice to train policies on simulated data with added noise patterns precisely to make them robust to real-world sensor imperfections. The paper only compares its depth *model* (CDM) against other depth *models* (e.g., PromptDA) but does not compare its *manipulation pipeline* against this standard sim-to-real baseline. This is a significant omission that weakens the central claim.\n2. **Questionable Premise:** Related to the first point, the premise that policies *cannot* be trained on noisy data is debatable. While the paper shows that a policy trained on *clean* data fails on *noisy* data (Table 2, \"None\" model), this is expected. It does not prove that a policy trained on noisy data would also fail. Many existing works successfully train policies on real-world, noisy data (for imitation learning) or on noise-augmented simulation data (for sim-to-real), as the policy learns to filter or ignore the noise implicitly. The paper overstates the necessity of perfectly clean depth for manipulation.\n3.  **Lack of Robustness Analysis:** For a paper focused on real-world manipulation, there is no analysis of the policy's robustness. The experiments are run in a controlled setting. There are no ablations involving physical perturbations, dynamic clutter, or other disturbances during policy execution. It is possible that the policy's success is brittle and highly dependent on the (now clean) geometric information, but lacks the robustness that might be learned from training on more varied (i.e., noisy) data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)