Manipulation as in Simulation: Enabling Accurate Geometry Perception in
  Robots
Authors: name: Minghuan Liu, name: Zhengbang Zhu, name: Xiaoshen Han, name: Peng Hu, name: Haotong Lin, name: Xinyao Li, name: Jingxiao Chen, name: Jiafeng Xu, name: Yichu Yang, name: Yunfeng Lin, name: Xinghang Li, name: Yong Yu, name: Weinan Zhang, name: Tao Kong, name: Bingyi Kang
Organization: ByteDance Seed

Modern robotic manipulation primarily relies on visual observations in a 2D
color space for skill learning but suffers from poor generalization. In
contrast, humans, living in a 3D world, depend more on physical properties-such
as distance, size, and shape-than on texture when interacting with objects.
Since such 3D geometric information can be acquired from widely available depth
cameras, it appears feasible to endow robots with similar perceptual
capabilities. Our pilot study found that using depth cameras for manipulation
is challenging, primarily due to their limited accuracy and susceptibility to
various types of noise. In this work, we propose Camera Depth Models (CDMs) as
a simple plugin on daily-use depth cameras, which take RGB images and raw depth
signals as input and output denoised, accurate metric depth. To achieve this,
we develop a neural data engine that generates high-quality paired data from
simulation by modeling a depth camera's noise pattern. Our results show that
CDMs achieve nearly simulation-level accuracy in depth prediction, effectively
bridging the sim-to-real gap for manipulation tasks. Notably, our experiments
demonstrate, for the first time, that a policy trained on raw simulated depth,
without the need for adding noise or real-world fine-tuning, generalizes
seamlessly to real-world robots on two challenging long-horizon tasks involving
articulated, reflective, and slender objects, with little to no performance
degradation. We hope our findings will inspire future research in utilizing
simulation data and 3D information in general robot policies.