D$^3$RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation
Authors: Songlin Wei
Haoran Geng
Jiayi Chen
Congyue Deng
Cui Wenbo
Chengyang Zhao
Xiaomeng Fang
Leonidas Guibas
He Wang
Venue: CoRL 2024 (2024)
Keywords: Depth Estimation
Diffusion Model
Stereo Vision
TL;DR: We propose D3RoMa, a diffusion model-based depth estimation framework on stereo image pairs for robotic manipulation.

Abstract: Depth sensing is an important problem for 3D vision-based robotics. Yet, a real-world active stereo or ToF depth camera often produces noisy and incomplete depth which bottlenecks robot performances. In this work, we propose D3RoMa, a learning-based depth estimation framework on stereo image pairs that predicts clean and accurate depth in diverse indoor scenes, even in the most challenging scenarios with translucent or specular surfaces where classical depth sensing completely fails. Key to our method is that we unify depth estimation and restoration into an image-to-image translation problem by predicting the disparity map with a denoising diffusion probabilistic model.  At inference time, we further incorporated a left-right consistency constraint as classifier guidance to the diffusion process. Our framework combines recently advanced learning-based approaches and geometric constraints from traditional stereo vision. For model training, we create a large scene-level synthetic dataset with diverse transparent and specular objects to compensate for existing tabletop datasets. The trained model can be directly applied to real-world in-the-wild scenes and achieve state-of-the-art performance in multiple public depth estimation benchmarks. Further experiments in both simulated and real environments show that accurate depth prediction significantly improves robotic manipulation in various scenarios.

RoMM Disparity Diffusion-based Depth Sensing for Material-Agnostia Robotic Manipulation
Songlin WeilA, Haoran Geng" Jiayi Chen Congyue Deng', Wenbo Cui
Chengyang Zhao Xiaomeng Fang". Leonidas Guibas", He Wang
CCFCS, School of Computer Science, Peking University,
University of California, Berkeley, Stanford University, *Galbot,
University of Chinese Academy of Sciences BBijing Academy of Artificial Intelligence https://PKU-EPIC. wwt.i....tbb........ io Lo/D3RoMa
(a) RealSense D415
(b) RealSense D435
Figure 1: Generalizability of D' in the real world. Our method robustly predicts transparent (bottles) and specular (basin and cups) object depths in tabletop environments and beyond. RGB image, pseudo colorized raw disparity map, our prediction, and point cloud are displayed for each case of a total of 6 frames captured by camera RealSense D415 and D435, RGB and depth images are not aligned for the D435 cameru for better visualization.
Abstract: Depth sensing is important problem for 3D vision-based robotics. Yet, real-world active stereo or ToF depth camera often produces noisy and in- complete depth which bottlenecks robot performances. In this work, we propose D*RoMa, a learning-baas based depth estimation framework on stereo image pairs that predicts clean and accurate depth in diverse indoor scenes, even in the most chal- lenging scenarios with translucent or specular surfaces where classical depth sens- ing completely fails. Key our method is that unify depth estimation and restoration into image-to-image translation problem by predicting the dispar- ity map with a denoising diffusion probabilistic model. At inference time, we further incorporated a left-right consistency constraint as classifier guidance to the diffusion process. Our framework combines recently advanced learning-ban based approaches and geometric constraints from traditional stereo vision. For model training, we create a large scene-level synthetic dataset with diverse transparent and specular objects compensate for existing tabletop datasets. The trained model can be directly applied real-world in-the-wild scenes and achieve state- of-the-art performance in multiple public depth estimation benchmarks. Further experiments in real environments show that accurate depth prediction significantly improves robotic manipulation in various scenarios.
Keywords: Depth Estimation, Diffusion Model, Stereo Vision

---

Introduction
With the extensive use of stereo cameras, stereo depth estimation has been one of the most widely studied problems in robotics for determining the target object position acquiring 3D) information of the environment [1, 2, 3, 4]. However, the depth maps provided by existing stereo cameras suffer from severe noise, inaccuracy, and incompleteness issues, bottlenecking robot performances regardless of their well-developed recognition and manipulation algorithms.
Traditional stereo-to-depth algorithms such as SGM [5] have fundamental issues: (i) In principle, they cannot tackle -amberertian surfaces due to the intricate light paths: (ii) Occlusion and out- of-view areas prohibit the computation of pixel correspondences. Recent works have leveraged learning-basee techniques for acquiring restoring better depth maps [6, 7]. While they alleviate the above issues to certain extent, predicting the depth for transparent and specular objects re- mains challenging as their image features from RGB pixel values are inherently ambiguous due to foreground-backeee
color blending and thus can be misleading to correspondence estimation [8].
In this work, we propose R*RoMa. Instead of building our network with cost volumes in most prior works, we transform the dense matching problem in depth estimation into image-to-imagee translation problem by predicting the disparity map with denoising diffusion probabilistic model. Such paradigm does not rely on low-level feature matching but rather unleashes the power of gen- erative models to directly translate the left and right frames into the target disparity image. More concretely, our method brings twofold benefits: (i) Unlike the regression models as in prior works, the diffusion model in our framework enables generative modeling of tions for transparent or translucent surfaces. (ii) The multi-step denoising process resembles the iterative solver, replacing prior iterative networks such RAFT-Stereo [7] and HitNet [9].
depth distribu-
Additionally, at inference time, we further incorporate the geometric constraints from traditional stereo vision by introducing a left-right consistency loss. The loss is integrated into the diffusion sampling process as a classifier guidance. The whole paradigm combines learning- maaeed predic- tions and traditional geometric modeling through a simple summation of their gradients in the score function of the diffusion model.
To train the network, we craft synthetic dataset HSSD-IssacSSimSSTD (HISS) of around 10, 000 stereo image pairs simulating real active stereo infrared patterns, including more than 350 trans- parent and specular objects in more than 160 indoor scenes [10]. Our dataset greatly extends the existing datasets that are limited to near-diffusive materials, table-top settings, or without realistic depth sensor simulation [11, 12, 13, 14, 15, 16). Trained on our synthetic dataset, our model can be directly applied to real-world in-the-wild scenes (Figure 1) and achieve state-of-the-art perfor- mances not only traditional stereo benchmarks but also on datasets targeting specular, transparent, and diffusive (STD) objects. To further validate our effectiveness in robotic manipulation, we con- duct experiments on both simulated and real environments ranging from tabletop grasping to mobile grasping in indoor scenes. We observe that with the high-quality depth maps and 3D point clouds predicted by our method, the success rates of robotic manipulation can be significantly improved in diverse settings.
To summarize, our contributions are: (1) A diffusion model-based stereo depth estimation frame- work that can predict
depth and restore noisy depth maps for transparent and specu- lar surfaces; (2) An integration of stereo geometry constraints into the learning paradigm via guided diffusion; (3) A new scene-level STD synthetic dataset that simulates real depth sensor IR patterns and photo-realistic renderings; (4) Significant improvements robotic manipulation tasks with our higher-quality depth maps and 3D point clouds.
Related Work
Stereo Depth Estimation and Completion. Modern deep-learning stereo methods [7, 17] typi- cally have the following structures. First, feature encoder used to extract the left and right

---

image features. The feature encoders are either pre-trained and frozen or trained end-to-end. Sec- ond, a cost volume is built by enumerating all the possible matching. Some works incorporate 31) CNN or attention networks to increase the receptive field of convolution layers which have proved to be beneficial [18, 19J. Finally, detection head is added to regress the disparities. The most success- ful ingredient is the iterative mechanism which is proposed by the seminar work RAFT [6] [7]. On the other hand, Weinzaepfel et al. [20] uses cross-view completion pertaining and achieves impres- sive results. Despite all the progress that has been made, in the real world, transparent and specular objects are ubiquitous and the RGB features of the surfaces are inherently ambiguous to be used to search for correspondence because the foreground and background colors are blended.
Previous work [13] tried restore the missing depths with the help of neighboring raw depth values and their RGB color clues [12] [21] but with limited generalizability. Another line of work directly fine-tunes trained deep stereo network [14] on transparent surfaces allowing the feature encoders to learn to match the transparent surfaces. However, the aforementioned foreground and background ambiguities confuse the feature-matching matching -based pipeline when dealing with regular diffuse objects.
Diffusion Model for Depth Estimation. Recently, researchers have employed diffusion models estimate optical flow [22] and predict depths with single RGB image input [23] [24] [25] [26]. Such monocular methods estimate the depth map up to unknown absolute scale. Bhat et al. [27] proposed to train extraneous network to predict the scale and achieve decent accuracy. How- ever, the monocular methods either lack the absolute scale or have inferior accuracy for robotic manipulation tasks. Another line of work has pioneered adjusting the diffusion model to stereo settings. Nam et al. [28] proposed to learn matching based on cost volume in a diffusion manner, which could not handle matching ambiguities. Shao et al. [29] proposed to refine raw map for high quality human reconstruction. The authors designed novel linear scheduler and condition on all the stereo- stereo-related information. Nonetheless, found that using the default DDPM [30] scheduling works well and only needs to condition on stereo images and raw disparity if necessary. Addition- ally, we combined the stereo matching loss gradient with the learned gradient by the diffusion model. The stereo matching loss is obtained by checking the left-right image photometric consistency in an unsupervised manner [31] [32]. Such guided diffusion model [33] achieves the best results in our experiments.
3D Vision-Based Based Robotic Manipulation. 3D vision is becoming increasingly critical for robotic manipulation [34]. Most basically, depth perception enables robots to comprehend the size, shape, and position of objects within three-dimensional space, thereby facilitating more sophisticated and reliable interactions [35, 36]. Morcover, numerous works [37, 38, 39, 40] utilize RGB-D point clouds as input. However, the substantial domain gap between simulated and real RGB-D images can result in significant sim-to-real gap [41]. Additionally, transparent and specular objects ex- acerbate this issue, leading poor depth-sensing performance. Policies trained in simulators often struggle to transfer effectively to real-world scenarios, particularly in the context of mobile ma- nipulation. Our proposed high- high-performance depth estimation network is promising direction to improve existing 3D vision-based tasks.
Method
In this section, we introduce D*RoMa, a disparity diffusion-based depth sensing framework for material-agnostie robotic manipulation. Our framework focuses on improving the accuracy of dis- parity map in depth estimation, especially for transparent and specular objects which are ubiquitous yet challenging in robotic manipulation tasks. Given an observation of the scene, our framework takes the raw disparity map D and the left-right stereo image pair I, I, from the depth sensor as input, and outputs a restored disparity map zo, which will be converted into the restored depth map.
3

---

Loop
Denoising Network
Va, log p( 4r DD)
I
Reverse Sampling
IT (0,
Final Prediction .o
Com
Matching
6---!
Figure 2: Disparity difflusion with stereo-geometry guidance. Our disparity diffusion- basod depth sensing framework takes the raw disparity map D and the left-right stereo image pair 4, I, as input. With the geometry prior from the stereo matching between I and I, as guidance for the reverse sampling, our diffusion model can gradually perform the denoising process conditioned on D to predict the restored disparity map To.
3.1 Preliminaries
Stereo Vision and Depth Estimation. Once the disparity map for the observed points between a pair of stereo cameras is known, the depth map for the points can then be calculated using the camera intrinsic parameters through d = (f b)/2, where f and b are the camera focal length and the stereo baseline, respectively. The estimation of the disparity map æ is traditionally modeled as a dense matching problem, which can be solved within the image domain. Thus, stereo depth estimation can be studied independently from different camera devices.
Denoising Diffusion Probabilistic Model. Diffusion models [30] [42] are special latent variable models that reverse the diffusion (forward) process which gradually diffuses the original data wo through a Markov process. When the network Sp is trained converged. the gradient of the noise distribution also called the score function [43]
Vz,log
VT- 80 t;
During inference, data samples be generated through ancestral sampling which resembles the stochastic gradient Langevin dynamics (SGLD) [44]
1 1 V Br
(zt + B.V.. log + Beg, ~ ()..
(2)
3.2 Disparity Diffusion for Depth Estimation
In this work, formulate the stereo depth estimation problem image-to-image translation problem in the diffusion model. One important design choice is what to condition on. The model usually formulated to condition on the stereo image pairs I,, T, for stereo depth estimation. Our experiments found that conditioning additionally on raw disparity D makes the network converge faster during training and generalizes more robustly out-of-distribution scenarios. The raw dis- parity can be easily obtained either from a traditional stereo matching algorithm SGM [5] or from the real camera sensor outputs. For real active stereo depth sensors like RealSense, the left and right images are captured by infrared (IR) cameras with special shadow patterns projected by IR pro- jector. As result, conditioning on the left and right images and the raw disparity map 1), we train a conditional diffusion model to learn the distribution of the disparity map
Po (woll) / pa ( p( p(xT) Il (o 1
where {Ii, D). Empirically, this conditional denoising network has been shown to be successful [45] [46]. Batzolis et al. [47] further provided proof (see Theorem that the conditional

---

score V ((rl())) can be learned through the same training objectives as in the unconditional case even though the condition does not appear the training objectives. After the network is trained, the estimated disparity can then be sampled through
1
Tt-
(xe + B a. logp(z.lu)) + Brer, ~ (O0,
(4)
5
3.3 Reverse Sampling Guided by Stereo Geometry
Inspired by the classier guidance to image generation tasks [48] [49], propose guide the dis- parity diffusion process with the model-based geometry gradient. The guided reverse process is illustrated in Figure 2. Specifically, the conditional score function is perturbed with gradient com- puted from stereo-matching
Vx, log p(gap()))) = VI- a sp- (It, t, y: sV., L.m Ir.
(5)
where Cam is the similarity loss function which compares the left image with the warped left image. The warped left image is obtained by warping the right image with the estimated disparity. s E R+ controls the geometric guidance strength and balances the learned gradient from diffusion model and geometric gradient from stereo models. A detailed derivation of Equation 5 is provided in Appendix B. To mitigate the gradient locality in stereo matching, downsample the stereo images into multiple different lower resolutions when computing the gradient of stereo matching. More specifically, we have
Lum [1, Ir,
E (I,, Rt) + + Leamooth (J1,
where the index of the layer for different resolutions and E R+ a weighting constant balances the two photometric and smoothness losses. The Lasim the structural similarity index (SSIM) [50] which computes the photometric loss between the left image and warped image Iett
Lessim (I,, Ir, x2.) = SSIM(I, Tt). iu v) I, + 22,
(8)
where u, v are the pixel coordinates in the image plane and () is linear sampling operation. Camooth an edge-aware smoothness loss [31] [32] defined as Lsmooth (4, x.) 00,2 1 10,41 10,44
which regularizes the disparity by penalizing large discontinuity non-edge areas. Here 0, means partial derivative in (horizontal) direction in the image plane. Then we predict the disparity map xo with the perturbed gradient from Equation 5 following the sampling process introduced in Equation 2. Finally, We can convert the disparity to depth once we know the camera parameters.
3.4 HISS Synthetic Dataset
We create our synthetic dataset HISS based on Habitat Synthetic Scenes Dataset (HSSD) [10]. We leverage the 168 high-quality indoor scenes from HSSD to increase scene diversity. For objects, we include total of over 350 object models from DREDS [13] and GraspNet [34]. The scene and randomly selected object CAD models are rendered in Isaac Sim [51]. During rendering, object materials and scene lighting are specifically randomized in simulation to mimic the transparent or specular physical properties of objects (cups, glasses, bottles, etc.) in the real world. To obtain the correct depth values for transparent surfaces, we adopt a two-pass methodology. We first render RGB images and depth maps of the scenes with object materials set to diffuse. The lightings are all turned to enable photorealistic rendering. In the second pass, we turn the normal lighting off and project a similar shadow pattern on the scenes to mimic the RealSense D415 infrared stereo images. Using the intrinsics of the RealSense D415 depth camera, we render over 10,000 photo-realistic stereo images with simulated shadow patterns. Experiments demonstrate that our dataset is the key enabler of our method's 's excellent generalizability in the real world.
5

---

Experiments
4.1 Depth Estimation in Robotic Scenarios
DREDS. We evaluate our method on DREDS [13], tabletop-level depth dataset with both syn- thetic and annotated real data for specular and transparent objects. We compare our method with several state-of-the-ard baselines: 1) NLSPN, 2) LIDF, 3) SwinDR, and 4) ASGrasp, which are four methods that have been shown effective on depth estimation for transparent or specular objects. We also compare another stereo depth SoTA methods: RAFT-Stereo, IGEV-Stereo, and CroCo- Stereo. We use mean absolute enror (MAE), relative depth error (REL), root mean square error (RMSE), along with 3 other metrics related to depth accuracy as metrics for evaluation. We include detailed descriptions for the baselines and the metrics in the Appendix C.
As shown in Table I, on the DREDS-CatKnon data split (synthetic data), all variants of our method surpass all baselines on all metrics. Further, the results of our ablations show that the performance of our method can be steadily improved with more information provided, especially the integration of the raw disparity.
Table I: Comparisons of Depth Estimation Results on DREDS Dataset (DREDS-CatKnove) split, synthetic). We also studied different combinations of conditioned images for the denosing network. Best shown in bold and second best shown in underlined.
Methods
RMSE +
REL 4
MAB
6 1
61.10
61.25
NLSPN (Park al. [21])
0,010
0.009
0.006
97.48
99.51
99.97
LIDF (Zhu et al. [12])
0.016
0.018
0.011
93.60
98.
99.92
SwinDR (Dai
[13))
0.005
98.04
99.62
99.98
ASGrasp (Shi et al. [14)
0.007
0.0016
0.004
RAFT (Lipson
(7])
Itev-Stereo (Xu et al. [18])
0.006
0.007
0.002
98.19
99.66
99. 97
CroCo-Stereo (Weinzacpfel et al. [52])
0.008
0.010
0.002
94.49
98.32
99.87
Dd DoRoMaCoond on RGB+Raw)
0.0045
0.0016
0.0011
99.64
99.88
99,99
D*RoMa(Cond. on RGB+LcfteRighr)
0.0070
0.0048
0.0032
99.
99,
99 98
D D*RoMa(Cond. on RGB+Loffe-Rggh-RRaw)
0.0040
0.0014
0.0010
99.71
99.90
99.99
More ablation studies regarding the geometry- based guidance, network sapee-earameters and archi- tectures are provided in Appendix F.
SynTODD. SynTODD [15] is another synthetic dataset for transparent objects by using Blender. contains 87512 train images and 5263 test images. The authors proposed a novel multi-view method (MVTrans) to estimate object depths, poses, and segmentations. Because the dataset pro- vides neither simulated raw disparity nor correct camera intrinsic for stereo images, We compare with MVTrans[15] using our monocular variant, ie., we modify the network to condition only on RGB images. We do scale alignment after the prediction of the scale-invariant depth following Mi- Das [53]. As shown in Table 2, Our method achieves better performance than all the variants of MVTrans including 2-views, 3-views, and 5-views.
ClearPose ClearPose [16] is large-scale real-world RGB-D benchmark for transparent and translucent objects. The dataset contains 350,000 real images captured by the RealSense L515 depth camera. The authors collected a set of very challenging scenes including different back- grounds, heavy occlusions, objects in translucent and opaque covers, on non-planar surfaces, and even filled with liquid. We evaluate our method on ClearPose in all settings. Our method D RoMM outperforms two previous SoTA ImplicitDepth [54] and TransCG [55] by a large margin as shown in Tabbl More detailed statisties of each scene are provided Table 11 and Figure 10 Appendix H. ClearPose is captured with RealSense L515 camera, the depth noise large when point distance is larger than 5 meters. We mask out the noise depth values out of the range [0.2,5] meters for all the experiments.

---

Table 2: Depth estimation results on Syn-TODD.
Table 3: Results on Depth completion
benchmark ClearPose.
Methods
RMSE 4
4
Methods
RMSEJ
REL
MAE
Simnee((Laskey et al. (56])
1.229
0.975
1.020
MVTrans images)
0.134
0.135
0.089
DmplichtDepth
0.133
0.120
0.101
MVTIrans (3 imuges)
0.125
0.125
0,083
TranCG
0.077
0.065
0.060
images)
D3RoMa
D Ronda on RGB)
0.065
0.079
0.040
(Cond. On RGB+Raw)
0.093
0.031
0.412
RGB
D RoMA (Ours)
Raft-Stereo
Raft-Stereo Ft.
ASGrasp
Figure 3: Qualitative Comparisons with SOTA methods in the Real World. Each row (from left to right) shows the RGB image and disparity results of our method, pre-trained Raft Stereo, Raft Stereo fine-tuned on our dataset, and ASGrasp.
4.2 Comparisons with SOTA Stereo Matching Methods in General Scenarios
We further demonstrate the effectiveness of our method for stereo matching in general scenarios. We compare our method with state-of oftttrrrrt mattoon basclines on SceneFlow [57], synthetic dataset containing more than 39,000 stereo frames in 960x540 pixel resolution. The dataset contains three challenging scenes, FlyingThiie Things3D, Driving, and Monkaa, which makes it a high-quality dataset for pre-training [7, 18, 20]. train our model from scratch using 35,454 stereo pairs for training and leave the rest as testing split. We also resize the images in the dataset into 480x270 to be consistent with our robotic perception settings. Following the previous work [18], the ground truth disparity is normalized using the maximum disparity value 192, which is also used to crop the test data. More implementation details are provided in Appendix E. As shown in Table 4, we achieve the best results compared to existing state-of-the-art methods.
Table Comparisons with SOTA Stereo Matching Methods on SceneFlow Dataset. *Ours is not built Cost Volumes.
Methods
GA-Net [58]
LEAStereo [59]
EdgeSStrre [60]
EPE
0.84
0.78
0.74
ACVNe1(61] 0.48
[GEV-Steroo[88] 0.47
HitNet [9]
D²RoMa" 0.36
0.36
4.3 Robotic Manipulation
In the grasping experiments, first acquire the depth map by D (f b)/X. Then back project the depth into point cloud p DK-1P, where K R3x3 is the camera intrinsics and P the homogeneous points in the image plane corresponding to each pixel. With the restored point cloud, we leverage GSNet [62] to predict DoF grasping poses. To increase the grasping success rate for all baselines, we filter the grasping pose which has the angle between the grasping pose and the (up) direction less than 30 degrees. We always select the grasping pose with the highest core and transform into the robot base frame.
Environment Setup. We set up tabletop grasping, articulated object manipulation and mobile grasping environment in the real world, shown in Figure 4, In the tabletop grasping experiments,
7

---

Table-top Grasping
Articulated Object Manipulation
Mobile Grasping
Robotic Manipulation. We examine our approach on challenging robot manipulation tasks and our performance significantly outperforms all baselines.
Figure 4:
we use a Franka 7-DoF robotic arm. We place STD objects on surfaces with bumps and pits, which are challenging setups for both depth sensing and robot grasping. The objects have non-diffusive surface materials such as glass, porcelain, glass, etc.
In the mobile grasping experiments, in Appendix E.3, The robot is arbitrarily placed near the target objects omitting the navigation phase which is beyond our scope. We place the objects in clusters at different tables and furniture with both flat and non-flat surfaces and different backgrounds, bringing challenges to depth sensing with the complex scene environments. More details about the articulated object manipulation experiments can be found in Appendix
customized wheeled robot with DoF arms as shown
Results and Analysis. We compare our method with two other baselines. All baselines use the same motion planner CuRobo [63] but different depth sensing. We also compare with ASGrasp [14] which was mainly designed for table-top grasping of STD objects. We report the results for different objects (STD) respectively and overall success rate in Table 6. The quantitative results for three different scenes of mobile manipulation are provided in Table 5. While ASGrasp and RoRoMa both improved over the raw sensor outputs, our method outperforms ASGrasp with a large margin.
Table 5: Mobile grasping success rate of different baselines with the same motion planner in real environments. Each cell shows SR on specular. transparent, and diffusive objects separated with /.
Table Comparisons of tabletop grasp- ing Success Rate (SR) with different depth sources. specular, trans- parent, D. = diffusive.
Baselines
Table
Sofa
Baselines
S.
Overall
Raw
0.400.22/0.67
0.67/0.63/1.00 1.00
0.50X1.75/0.6 67 0.67/0.8750..8
Raw
0.33
0.25
0.70
0.45
ASGrasp
0.600.899. 89/0.83
2.67/0.550..8
ASGrasp
0.63
0.43
0.50
0.50
D RoMa
1.80/1.00/1.00
1.67/088//1.0
0.83 0.875 0.83
D'RoMa
0.83
0.63
0.78
0.77
5
Conclusion
In this work, we proposed novel geometry gradient guidance to diffusion model in disparity space to predict depth for stereo images. Conditioning on stereo image pair and raw disparity map, our network achieves SOTA performance on existing benchmarks. Both disparity evaluation on pure synthetic datasets and depth evaluation on depth datasets demonstrate the efficiency of our method. Our key observations include data diversity can make big impact on generalization in the wild and guidance helps in more challenging real-world scenes. Current 3D vision- based robotics manipu- lation pipelines including grasping and part manipulation can be significantly improved simply by improved depth perception. Especially, we found that depth estimation for challenging transparent objects could be better dealt with generative models than traditional stereo methods.
Limitations and Future Work One limitation inherited from the diffusion model is the iterative inference with many denoising steps. This issue can be mitigated by incorporating diffusion sam- pling acceleration techniques studied in variety of existing works. Another limitation is that the EPE is linear to the resolution of the input image. Future research could use coarse-to-fire or simple sliced techniques to split high resolution images into smaller patches to resolve this issue.

---

Acknowledgments
This work supported by the National Natural Science Foundation of China(No.62306010)
References
[1] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.W. Chao, and D. Fox. RVT: Robotic view transformer for 3d object manipulation. In 7th Annual Conference Robot Learning, 2023, URL https: // //operreview net/forumidd-oeeeee id=OhPkttoGAf..
[2] Yi. A. Zeng, and S. Song. Rearrangement planning for general part assembly. In 7th Anmual Conference Robot Learning, 2023.
[3] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu. Robocook: Long-horizon elasto-plastic object manipulation with diverse tools. arXiv preprint arXiv:2306.14447 2023.
[4] Grannen, Y. Wu, B. Vu, and D. Sadigh. Stabilize to act: Learning coordinate for bi- manual manipulation. In 7th Annual Conference Robot Learning, 2023. URL https: /openreview net/forumm id=86aMP Jn6hX9F.
[5] H. Hirschmuller. Accurate and efficient stereo processing by semi-global matching and mutual information. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume 2, pages 807-814. IEEE, 2005.
[6] Z. Teed and J. Deng. Raft: Recurrent all all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th Europeun Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part II 16, pages 402-419. Springer, 2020.
[7] L. Lipson, Z. Teed, and J. Deng. Raft-stereo: Multilevel recurrent field transforms for stereo matching. In 2021 International Conference on 3D Vision (3DV), pages 218-227. IEEE, 2021.
[8] A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. Coded time of flight cameras: sparse deconvolution to address multipath interference and re- cover time profiles. ACM Transactions on Graphics (ToG), 32(6):1-10, 2013.
[9] V. Tankovich, C. Hane, Y. Zhang, A. Kowdle, S. Fanello, and S. Bouaziz. Hitnet: Hier- archical iterative tile refinement network for real-time stereo matching. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14362-14372, 2021.
[10] M. Khanna, Y. Mao, H. Jiang, S. Haresh, B. Schacklett, D. Batra, A. Clegg, E. Undersander, A. X. Chang, and M. Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoff's for objectgoal navigation. arXiv preprint arXiv:2306. 11222000..110. 11900, 2023.
[11] Sajjan, M. Moore, M. Pan, G. Nagaraja, Lee, A. Zeng. and S. Song. Clear grasp: 3d shape estimation of transparent objects for manipulation. In 2020 IEEE international conference on robotics and automation (ICRA), pages 3634-3642. IEEE, 2020.
[12] L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. van Eenbergen, S. Debnath, and D. Fox. Rgb- d local implicit function for depth completion of transparent objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4649 4658, 2021.
[13] Q. Dai. Zhang, Q. Li, Wu, H. Dong, Liu, P. Tan, and H. Wang. Domain randomization- enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects. In European Conference Computer Vision, pages 374-391. Springer, 2022.
[14] Shi, Y. Jin, D. Li, H. Niu, Z. Jin, H. Wang, al. Asgrasp: Generalizable transparent object reconstruction and grasping from rgb-d active stereo camera. arXiv preprint arh1::22500:0 2024.

---

[15] Y. R. Wang, Y. Zhao, H. Xu, S. Eppel, A. Aspuru-Guzik, F. Shkurti, and A. Garg. Mv- trans: Multi-view perception of transparent objects. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3771-3778, 2023. doi:10.1 109//CAA48899 2022.. 10161089.
[16] X. Chen, H. Zhang, Z. Yu, A. Opipari, and O. C. Jenkins. Clearpose: Large-scale transparent object dataset and benchmark. In European Conference on Computer Vision, 2022.
[17] J. Li, P. Wang, P. Xiong, T. Cai, Z. Yan, L. Yang, J. Liu, H. Fan, and S. Liu. Practical stereo matching via cascaded recurrent network with adaptive correlation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16263-16272, 2022.
(18) G. Xu, X. Wang, X. Ding, and X. Yang. Iterative geometry encoding volume for stereo match- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11919-21928, 2023.
[19] Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li. Flowformer: A transformer architecture for optical flow. In European conference on computer vision, pages 668-688 Springer, 2022.
[20] P. Weinzaepfel,
T. Lucas, V. Leroy, Y. Cabon, V. Arora, R. Brégier, G. Csurka, L.. Antsfeld, B. Chidlovskii, and J. Revaud. Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17969-17980, 2023.
[21] J. Park, K. Joo, Z. Hu, C.-K. Liu, and So Kweon. Non-local spatial propagation network for depth completion. Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XIII 16, pages 120-136. Springer, 2020.
[22] S. Saxena, C. Herrmann, Hur, A. Kar, M. Norouzi, D. Sun, and D. J. Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. Advances in Neural Information Processing Systems, 36, 2024.
[23] Y. Duan, X. Guo, and Z. Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint #xXiv:2303.05021 2023.
[24] S. Saxena, A. Kar, M. Norouzi, and D. J. Fleet. Monocular depth estimation using diffusion models, 2023.
[25] Y. Ji, Z. Chen, E. Xie, L. Hong, X. Liu, Z. Liu, T. Lu, Z. Li, and P. Luo. Ddp: Diffusion model for dense visual prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21741-21752, 2023.
[26] B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler. Repur- posing diffusion-based image generators for monocular depth estimation. arXiv preprint arXiv:2312.02145, 2023.
[27] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
[28] J. Nam, G. Lee, S. Kim, H. Kim, H. Cho, S. Kim, and S. Kim. Diffmmatc: Diffusion model for dense matching. arXiv preprint arXiv:2305.19094 2023.
[29] R. Shao, Z. Zheng, H. Zhang, J. Sun, and Y. Liu. Diffustereo: High quality human reconstruc- tion via diffusion-based stereo using sparse cameras. In European Conference on Computer Vision, pages 702-720. Springer. 2022.
[30] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 2020.
10

---

[31] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270-279, 2017.
2017.
[32] S. Wei, G. Chen, W. Chi, Z. Wang, and L. Sun. 3d object aided self-supervised monocu- lar depth estimation. 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10635-10642, 2022. doi:10. 109/1R0547612..222.222 1590.
[33] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Gold- stein. Universal guidance diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
[34] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet-Ibilioon: A large-scale benchmark for general object grasping- In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pages 11444-11453, 2020.
[35] H. Geng, H. Xu, Zhao, C. Xu, L. Yi, S. Huang, and Wang- Gapartnet: Cross-category domain-geneeralizbble object perception and manipulation via generalizable and actionable parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pages 708 1-7917 2023.
[36] H. Geng, S. Wei, C. Deng, B. Shen, H. Wang, and L. Guibas. Sage: actionable parts for generalizable articulated manipulation under language instructions. arXiv preprint arXiv:23/2.01307, 2023.
Bridging semantic and
[37] M. Bajracharya, J. Borders, R. Cheng, D. Helmick, L. Kaul, D. Kruse, J. Leichty, J. Ma, C. Matl, F. Michel, et al. Demonstrating mobile manipulation in the wild: A metrics-driven approach. arXiv preprint arXiv:2401.01474 2024.
[38] S. Patki, E. Fahnestock, T. M. Howard, and M. R. Walter. Language-guidde semantic map- ping and mobile manipulation in partially observable environments. In Conference on Robot Learning, pages 1201-1210. PMLR, 2020.
[39] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine. Fully autonomous real-world reinforcement learning with applications to mobile manipulation. In Conference on Robot Learning, pages 308-319. PMLR, 2022.
[40] J. Lyu, Y. Chen, T. Du, F. Zhu, H. Liu, Y. Wang, and H. Wang. Scissorbot: Learning gen- eralizable scissor skill for paper cutting via simulation, imitation, and sim2real. In 8th An- nual Conference on Robot Learning, 2024. URL pettps:/oppereeee net/forum?idt PAtsxVZOND.
[41] Zhang, N. Gireesh, J. Wang, X. Fang. C. Xu, W. Chen, Dai, and H. Wang. Gamma: Graspability-wware mobile manipulation policy learning based on online grasping pose fusion. arXiv preprint arXiv:2309. 2023.
[42] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265 PMLR, 2015.
[43] Y. Song, C. Durkan, I. Murray, and S. Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021.
[44] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681- 688. Citeseer, 2011.
[45] C. Saharia, Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE transactions pattern analysis and machine intelligence, 45 (4):4713-4726, 2022.

---

[46] Y. Tashiro, J. Song, Y. Song, and S. Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-2485 2021.
[47] G. Batzolis, Stanczuk, C.-B. Schönlieb, and C. Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606, 2021.
[48] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Infor- mation Processing Systems, volume 34, pages 8780-878 18944774. Curran Associates, Inc., 2021.
[49] Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12599. 2022.
[50] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from eπror visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[51] Liang, Makoviychuk, Handa, N. Chentanez, M. Macklin, and D. Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning, 2018.
[52] P. Weinzaepfel, T. Lucas, V. Leroy, Y. Cabon, V. Arora, R. Brégier, G. Csurka, L. Antsfeld, B. Chidlovskii, and J. Revaud. Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17980, 2023.
[53] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
[54] L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. van Eenbergen, S. Debnath, and D. Fox. Rgb- d local implicit function for depth completion of transparent objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 4649 444994668,, 2021.
[55] H. Fang, H.-S. Fang, S. Xu, and C. Lu. Transeg: A large-scale real-world dataset for transpar- ent object depth completion and grasping baseline. IEEE Robotics and Automation Letters, (13):7383-7390, 2022.
[56] M. Laskey, B. Thananjeyan, K. Stone, Kollar, and M. Tjersland. Simnet: Enabling robust unknown object manipulation from pure synthetic data via stereo. In 5th Annual Conference on Robot Learning, 2021. URL https: http::::///pppeeveeeeee
[57] N. Mayer, E. Ilg. P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4040, 2016.
[58] Zhang, Prisacariu, Yang, and P. Torr. Ga-net: Guided aggregation net for end-to-end stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 185-194, 2019.
[59] X. Cheng, Y. Zhong, M. Harandi, Y. Dai, X. Chang, H. Li. T. Drummond, and Z. Ge. Hier- archical neural architecture search for deep stereo matching. Advances in neural information processing systems, 33:22155-22169 2020.
[60] X. Song, Zhao, L. Fang, H. Hu, and Y. Yu. Edgestereo: An effective multi-task learning network for stereo matching and edge detection. International Journal of Computer Vision, 128(4):910-8)) 930 2020.
12

---

[61] G. Xu, J. Cheng, P. Guo, and X. Yang. Attention concatenation volume for accurate and efficient stereo matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12981-12990, 2022.
[62] C. Wang, H.-S. Fang, M. Gou, H. Fang. J. Gao, and C. Lu. Graspness discovery in clutters for fast and accurate grasp detection. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 15964-15973, 2021.
[63] B. Sundaralingam, S. K. Hari, A. Fishman, C. Garrett, K. V. Wyk, V. Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, N. Ratliff, and D. Fox. curobo: Parallelized collision-fre minimum-jerk robot motion generation, 2023.
[64] O. Shingo, T. Akihiro, and H. Takaaki. libsgm. httpss//gtaabb.com/sasssss////53 hatau///ccmm/xsstts/11S/t 2018.
[65] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj. D. Nair, S. Paul, W. Berman, Y. Xu, S. Liu, and Wolf. Diffusers: State-of-the-art diffusion models. https://ga thub. cwm/ www//huginggaaaeeeooooooooooeeee seeeediffuseerrr 2022.
[66] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. Interna- tional conference on machine learning, pages 8162 8171. PMLR, 2021.
[67] R. Rombach, A. Blattmann, D. Lorenz, Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models, 2021.
Supplementary Materials
A Overview
In this supplementary material, we will give the derivation for geometry guided diffusion model for stereo vision in Appendix B, introduce different metrics in Appendix C, study some interesting properties of our method in Appendix D, provide more implementation details in Appendix E, ana- lyze different guidance modes in Appendix F, study alternative guidance in Appendix G, show more results of our methods in Appendix H. provide more data samples our dataset HISS in Appendix and perform more experiments of mobile part manipulation in Appendix
B Geometry Guidance for Stereo Vision
To complement the main body of the paper, we provide the detailed derivation of the geometry guided diffusion model which appears in Equation 9 in the main text.
B.1
Stereo Vision
We define I, represents the conditioning stereo image pair and x, is the noisy depth at time step t. By Bayes' theorem, have
p( (u) = p(xs)p(ylxr)
(10)
p(v)
log (y) logp((;) + log p(gl())) log p(y)
(11)
Task derivative with respect to 2, on both sides of Equation 11:
Vx, log logg((p;l)))) Vx, log p(x,) Va, log lyg(())
(12)
Now, partition the second term log (0gg(()2))))) as log p(v(2;)
logp (x))
logp(I (x) + [og
= logp(x;(T)) + log)(T)) - log ( + ((,
(13)
13

---

where we apply Bayes' theorem again in the third equation. Substitute Equation 13 back to Equation 12, we have
Vx, pog y) = Vx, log (( + Vx, pog
(14)
The first term learned by the denoising network and the second term is the geometric guidance which can be calculated by stereo matching. the experiments, we leverage more available data such I, and D in addition to into the network during training:
Vx, log p(r
v1 az so- (#t, t, y: 0) 2V,, Tr)
(15)
Here empirically scale the geometry gradient with E IR+ and set it to in the experiments.
B.2 Extend to Active Stereo Vision
In addition to the left and right IR images, active stereo cameras provide another color image Ie captured from a third color camera. While the above derivation directly applies to active stereo cameras if we ignore the color image, we found that further feeding the color image into the network slightly improves the performance in DREDS [13]. However, most stereo datasets are passive and do have additional color images. Therefore, during mixed dataset training, this additional color image is dropped. Here, we provide active stereo version of derivation analogous to Equation 13:
log p(yla)) = (,
logp(1e [xx) + logg(()) + Io le,
logp(Ic [x) + pog.((1,)) x4)
logp(z; [T) + log p(Tc) log + logp(I,(
(16)
The and already aligned and the only difference is the shadow pattern projected from the camera IR projector. The shadow pattern is irrelevant to the depth. Therefore, Ie is approximately the sufficient statistic of 7,. For the same argument, we have log .F) = p(l, /4, Lo, xT). Likewise, the guidance for the active stereo camera can then be obtained by substituting Equation 16 into Equation 12:
where the third equation assumes p(1i [e,
Vx, log p(x y) V., logp(z; Va, logp(I,) v2)
(17)
In active stereo vision scenarios, we further train the network by conditioning it also other avail- able images. We set y (1, Ir. D}.
C Baselines and Metrics
Baselines. NLSPN [21] is a depth completion work that uses an end-to-end non-local spatial prop- agation network to predict dense depth given sparse inputs. LIDF [12] proposes to learn an implicit density field that can recover missing depth given noisy RGB-D input. SwinDR [13] proposes a depth restoration framework based on SWIN transformer and trained on proposed table-top dataset with STD objects (DREDS). ASGrasp [14] proposes a stereo-depth estimation method based on Raft-Stereo to predict two-layer depths for tabletop grasping. Raft-Stereo [7] is the seminal deep stereo network. To this day, it is still the most adopted architecture in stereo vision.
Disparity Metric. End-Point Error (EPE) H*W Σ X is the mean absolute difference for all pixels between the ground truth and estimated disparity map.
Depth Metries. We use the following depth metrics: 1) RMSE V NXW D D12 is the root mean square enror between ground truth and predicted depths, 2) MAE Hx D D the mean absolute depth error, 3) REL = D - DI/D is the mean absolute relative dif- ference, and 4) accuracy metric ò; is the percentage of pixels satisfying max (4, 4 < 6, where 6, E {1.05, 1.10, 1.25).
14

---

D Interesting Properties of Generative Stereo Vision
D.1 Uncertainty Estimation
Because our method diffusion model based, we inherited the stochasticity in the reverse sampling process. To visualize the stochasticity, we run the same input 10 times. The uncertainty obtained as the variance of the output disparity map. We conduct the experiments on DREDS and show the results in Figure 5. We observed that high uncertainty area corresponds to object edges where depth dramatically changes between foreground and background. Flat surfaces have lower uncertainty as the geometry is simpler. Such uncertainty could be used to filter outliers.
Predicted Disparity
Ground Truth
Raw Sensor
RGB
Uncertainty
Std- Std-CatNovel
Std-CatKnown
Figure 5: We visualize sample variance as uncertainty in the last column.
D.2 Generalization Comparisons with Monocular Methods
While our method works only in stereo cases, there are seminar works predicting depth given single RBG images. The attractive part of monocular depth estimation (MDE) is that more data is available for training. Therefore, these methods can be generalized well in the wild. While some monocular methods like ZeoDepth [27] propose to recover metric depth after a special training procedure, most monocular methods predict relative depth. The relative depth can be recovered with an absolute scale which can obtained via other sensors like lidar or prior knowledge. However, our experi- ments (Figure 6) found that most monocular methods produce inferior quality depth even without considering the absolute scale.
E
Implementation Details
E.1
CUDA accelerated Semi-Global Matching
We used libSGM[64], CUDA-accelerated, widely adopted implementation of the Semi-Global Matching (SGM)[5] algorithm. To seamlessly integrate libSGM into our pipeline, we utilized py- bindl1 encapsulate the original codebase within our Python-based framework. This integration allows the adapted version of libSGM achieve performance of approximately 55 frames per second (FPS) at input resolution of 960 540, with around 380MB of memory allocated on an NVIDIA RTX 4090 GPU.
15

---

Raw Depth
(*Ros)(Ours)
DepthAnything
Murigold
ZoeDepth
Figure 6: Generalization comparisons with State-of-the-art monocular depth estimation methods. All the results except ours are taken from their official web demo. Different methods used different color maps.
E.2 Network HyperParameters and Training
We implement our network using Hugging Face Diffusers [65] and pre-compute raw disparity maps using libSGM [64]. The network trained 600 epochs with the batch size 6x8 and a constant learn-
ing rate 0.0001. All the images are randomly cropped 320x240 and no other data augmentation used during training. We use cosine scheduler [66] with 128 denoising time steps for 8, starting at 0.0001 and ending at 0.02. We use UNet we have 6 downsampling ResNet blocks each layer has 128, 128, 256, 256, 512, and 512 channels. The second-to-last channel is a downsampling block with spatial attention. We use MSE as our loss function. For the SceneFlow experiment, we scale down the original image resolutions from 960x 640 into x80. We use multi-resolltion pyramid noise strategy as [26]. further use pretrained StableDiffusion v2 [67] in the grasping experiments and adapt the input Conv block accordingly to the conditioning inputs [26]. We also train the mixed datasets including DREDS, HISS, and SceneFlow at the batch level.
denoising network. In the DREDS experiments,
E.3 Grasping Implementation and Hardware Setup
In the grasping experiments, we mount the RealSense D415 on the wrist of the arm. After the camera captures frame, we first acquire the depth map by D = (f-b)/ bb/ Then back project the depth into point cloud P = DD- P. where K E 33x3 is the camera intrinsics and P are the homogeneous points the image plane corresponding to each pixel. With the restored point cloud, we leverage GSNet [62] predict 6 DoF grasping poses. To increase the grasping success rate for all baselines, filter the grasping pose which has the angle between the grasping pose and the z (up) direction less than 30 degrees. We always select the grasping pose with the highest and transform it into the robot base frame. Then we grasp the object with motion planner like CuRobo [63]. We did not perform workspace point cloud cropping operation as in the baseline ASGrusp [14] hence leading to an overall success rate drop in the main text compared with the numbers reported in ASGrasp.
We a wheeled mobile base mounted with DoF customized experiments. Each arm attaches parallel gripper. We only the left arm in the experiments. Figure displays the robot and the workplace.
in the real mobile grasping
16

---

RealSense stereo camera
DoF Arm with parallel gripper
Wheeled mobile basc
Transparent and specular objects
One random table
Figure 7: The robot used in the real mobile grasping experiments.
F
Ablations Studies
F.1
Ablations on Geometry-Guidance
Since DREDS does provide IR images for the STD-CatKnown and STD-CatNovel data split (real data), we train the variant of our framework which only conditions RGB image and raw disparity to compare with SwinDR.
We further perform ablation studies for the geometry-basee guidance on the STD-CatKnown and STD-CatNovel data split to validate its effectiveness for diffusion depth estimation in real- world scenarios. As Table 7 shows, our geometry-based guidance can significantly boost perfor- mance, especially for out-o--distribution scenarios. More ablation studies regarding network hyper- parameters and architectures are provided in Appendix F.
Table 7: Evaluations of Geometry Guidance on DREDS Dataset (STD-CatKnown and STD- CatNovel split, real). Ground truth depth cropped range (0.2, 2].
Methous
Guidance
RMSE
REL
MAE
61.05
61
81
SwinDR (Dai al. I13))
0.015
0.013
0.008
96.66
99.03
99.92
D RoMadConnd. RGB+Raw)
0.0109
0.0051
0.0036
98.41
99.46
99.94
D RoMa/Cond. on RGB+Raw)
0.0101
0.0044
0.0030
99.03
99,57
99.93
STD-CaiNovel
SwinDR [13])
Dd RoMadCood. on RGB+Raw)
0.0390
0.0177
0.0104
91.19
96.17
99.51
D RoMaaCondd on RGB+Raw)
0.0397
0.0158
0.0092
92.78
97.13
99.61
F.2 Ablations on Network HyperParameters and Architectures
We provide ablation studies on the DREDS dataset in Table 8. The baseline is conditioned on the left, and right image and raw disparity. Its hyperparameters and network architecture are described in Appendix E.2. We also trained variants with different network architectures, loss functions, and noise strategies. We reduce the channels from 512 to 256 of the last two layers denoted as reduced channels. We also changed the loss function from MSE to L1 and used the default standard Gaussian noise.
17

---

Table 8: Ablation Studies on Hyperparameters and Network Architectures.
Methods
MAE
61.05
10 7
61.25 25
Baseline
0.0040
0.0014
0.0010
99.71
99.90
99.99
D RoMa (reduced channels)
99.98
Da (I.1 loss)
0.0047
0.0008
0.0012
99.60
99.83
99.98
D RoMa (cando noisc)
0,0048
0.0017
0.0012
99.64
99.87
99.98
F.3 Ablations Different Samplers and Inference Time
The main factors of run time are the input image resolution and the number of denoising steps. To better understand the computation-accuraay tradeoff, list the runtime of our method and other SoTAs in Table 9. We also evaluate the effects of different samplers in the real experiments where we used pretrained StableDiffusion [67]. We report the inference time of our method in Table 10. The network has about 865M parameters in total for this variant. We fixed the number of time steps during training to 1000 and used the same standard cosine scheduler [66], and all the samplers take denoising steps during inference. We perform reverse sampling using different schedulers implemented by Diffusers [65]. All the samplers achieve similar qualitative results except Euler Ancestral. The results shown in Figure 8. Empirically, select DDPM with 10 denoising steps and resolution of 640x360 360 in our real experiments.
Table 9: Runtime and memory consumption comparisons. Our method has 113M parameters and takes 10 denoising steps during inference.
NLSPN
SwinDR
ASGrasp
IGEvoStereo
RoMa
Runtime
15ms
37ms
10ms
40ms
162ms 2.84G
188ms 2.78G
670ms
Peak Memory Usage
4,79G
1.16G
L.7G
3.14G
281G
Table 10: Runtime and memory consumption of method during reverse sampling for single input. All times are reported on NVIDIA A100.
Disparity Resolutions
1280x
640x360
480xx70
126
Denoising stops
5.53
2.56
2.31
1.95
1.96
Denoising Steps
8.82
3.19
2.91
2.25
2.17
50 Denoising Steps
34.56
8.45
5.79
4.28
3.86
Peak Memory Usage
18.62G
7,87G
7.57G
6.94G
6.89G
G Alternative Guidance with Raw Disparity
This section will study alternative guidance to the diffusion model during the reverse sampling processes. In the stereo vision case, the gradient of the photometric loss is obtained by checking the consistency of the left and right images. Mathematically, the gradient should also have the same direction with Ty 24 where Zo is the ground truth disparity. In test time, Ty is unknown but can be approximated by external less noisy measurement source such as a Lidar. The external depth measurement can be converted to the disparity space To and is multiplied with a mask if it is sparse:
V log (w) V.. log l V., log T, x,)
= V v. log aw(u, v)sign( To xx)
(18)
We here experiment with raw depth guidance. The guidance ão is approximated by camera raw sensor depth. Therefore the sign(ãg - vz) is the approximate gradient. We set mask w(u, v) = (ão 0) and is constant controls the guidance strength. We qualitatively study the guidance of the approximate gradient in Figure 9. The benefits of guidance by the approximate gradient limited when depth is highly noisy.
18

---

(4) (b)
(c) Ialler
(d) Ealer Ancestral
(e) Heun
(0)
(g)
Figure 8: Comparisons of different samplers used during reverse sampling. We use DDPM [30] sampler with 10 steps in the experiments.
RGB
Raw Disparity
Results
RGB
Row Disparity
Results
Figure 9: Guidance with raw disparity.
H More Experimental Results
H.1 Detailed Comparisons on ClearPose Dataset
We provide the results our method on ClearPose compared with two other baselines as shown in Table There are a total different scenes, including different backgrounds, heavy occlusions, objects translucent and opaque covers, on non-planar surfaces, and even filled with liquid.
Table 1I: Results on Depth completion benchmark ClearPose. Our method RoRMaa consis- tently outperforms both ImplicitDepth and TransCG on 6 different test scenarios.
Pessect
Metric
RMSE]
REL1
MAEL
01.05
$1.10
81.25
ImplicnDepth
0.07
0.04
67.00
87.03
New Background
TransCGG
0.03
0.03
0.02
86.50
97.02
99.74
D "RoMaaCood RGB+Raw)
0.05
0.01
0.01
96.71
98.84
99.74
ImplicnDepti
41.43
Heavy Occlusion
TranCG
0.06
0.04
0.04
72.03
90.61
98.73
D RoMddCond On RGB+Raw)
0.10
0.03
0.04
83.97
93.69
98.79
ImpliciDepth
0.16
22.85
Translucett Cover
TranSG
0.16
0.15
0.14
23.44
39.75
67.56
D RoModCodd On RGB+Rm))
0.13
0.07
63.07
82.78
95.80
ImplicnDepth
55.39
Opaque Distractor
TransCG
0.08
0.06
0.06
$2.43
75.52
97.53
D Roma On RGB+Raw)
0.11
0.03
0.05
82.46
91.46
97.99
ImpliciDepth
0.14
0.11
32.84
53.44
84
Filled Liquid
TranSCG
0.04
0.06
0.03
77.65
93.81
99.50
D Roma On RGB+Raw)
0.09
0.03
0.04
87.58
94.885
99.15
ImplicnDepth
0.18
0.15
20.34
38.57
74.02
Non Planar
TranCGG
0.09
0.07
0.07
55.31
76.47
94.88
D Comd On RGB+Raw)
0.08
0.03
0.04
84.67
92.85
98.21
H.2 Results on HISS Test Split
In this section, we train other SOTA stereo methods from scratch and compare with our method on the HISS dataset. We further rendered 300 images in new scenes different from our training dataset the test set. The results are given in Table 12. We also show more real depth estimation results in Figure 12 and more comparisons in Figure 13, which consider also attributed to the joint training our dataset.
19

---

RGB
Raw Depth
GT Depth
TransCG
ImplicitDepth DoMM
Heavy Background Occlusion
New
Opaque Distractor
Transluent
Cover
Filled Liquid Non Planar
Figure 10: Qualitative depth completion results on ClearPose. From left to right, there are RGB image, raw depth, ground truth depth rendered using object CAD models, completed depth by Tran- sCG, ImplicitDepth, and our method RoRoMa.
Table 12: Quantatives evaluations on HISS dataset.
Methods
EPE
RMSE
REL
MAE.L 4
61.08
61.25
Rafi-Stcroo D D'RoMa
0.0721
0.0521
0.0092
0.0164
95.26
99.10
0.0579
0.0378
0.0067
99.76
H.3 Deonising Process
One of the motivations for using the diffusion model predict depth the multi-step reverse sam- pling process. It resembles the iterative solver which has been proven successful in RAFT [6] and its successors. In figure 11 show example of the denoising process trained HISS dataset. The total denoising steps is set to 128 and we visualize every 32 timesteps.
I HISS Dataset
HISS. We further evaluate the effectiveness of our dataset for transparent and specular object depth estimation. We compare method with the previous state-of-the-an methods. As shown in Figure 13, compared with RAFT-Stereo [7], which is trained on large-scale datasets for stereo- matching, our method can predict better depth, especially on transparent bottles. To ensure fair comparisons, we further fine-tune RAFT-Stereo on HISS for 400,000 epochs. Compared to the original model, the fine-tuned RAFT-Stereo can recover the missing depth of transparent objects better but the object shapes are still inaccurate. We also compare our method with ASGrasp [14] which specially designed to detect and grasp transparent objects based depth estimation. has similar performance to the fine-tuned RAFT-Sterco but has blurred object boundaries. Our methods can provide the best depth for all STD objects, with significantly clearer object boundaries
20

---

= 95
63
31
Ground Truth
Figure II: Visualization of the denoising process the HISS dataset. The left columns show the denoising steps every time steps. The 2 right columns show the final output and ground truth disparity map respectively.
and accurate shapes. More quantitative and qualitative results for the effectiveness of our dataset are provided in Appendix H.
One aspect that characterizes dataset is scene-level and peoto-realistic rendering of the specular, fransparent, and diffuse objects. We rendered over 350 objects in 168 different HSSD [10] scenes. The objects randomly fall onto the furniture, ground, and tables to simulate real-world object place- ments. The infrared (IR) images are rendered properly with seeing-through or specular lighting effects on Non-Lambertian surfaces. We provide some data samples in Figure 14.
J Part Manipulation
J.1 Interaction Policy
Following [35, 36), first do part segmentation and pose estimation using the perception method. Based on the predictions of the part poses, we move the robot arm toward the target part and turn the gripper in the direction suitable for grabbing. Finally, we move the gripper along the proposed trajectories toward the target position, following our GAPart pose definition.
J.2 Experiment Setup
In the experiments, use the Franka Emika Panda robot arm with CuRobo[63] motion planning and the end-effector trajectory just like GAPartNer(35). For manipulation tasks in the real world, partial point cloud of the target object instance is acquired from method. With the proposed network and manipulation heuristics in [35], the pose trajectory of the end-effector can be predicted. Then we use cuRobo(63) to solve the pose of Franka to follow our end-effector trajectory.
21

---

RGB
D'RuMa(Ours)
Disparity
D D'RoMa(Ours)
Raw Disparity
Figure 12: More in-the-wild examples. Each example consists of an RGB image, a raw disparity map, and our prediction.
1
0
RGB
Raw Disparity
D'RoMa(Ours)
RAFT-Stereo
RAFT-Stereo.ft
ASGrasp
Figure 13: Qualitative comparisons with other state-of-the-arts.
22

---

Figure 14: RGB Data samples from Our dataset HISS except the bottom row which shows group of rendering RGB image, (left) IR image, and normal.
23


Reviews:
[{"review_id": "3QzAaLOsYs", "reviewer": "Reviewer_SQrs", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a denoising diffusion probabilistic model method to estimate accurate depth, even for translucent objects from stereo images for indoor scenes by casting the image restoration and depth estimation problem into an image-to-image translation problem. They collected a synthetic dataset with specular objects to train and better evaluate the proposed method. Various experiments are performed in both simulation and the real world to prove the effectiveness of the proposed method.", "summary_of_recommendation": "Overall this is a well written paper with good evaluation and ablation studies. However, I have some concerns and questions. I would like to see the response from the authors during the rebuttal process.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please see the concerns/weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vOXzc2eM4", "reviewer": "Reviewer_jawu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the problem of depth perception of transparent objects with a stereo camera. This is an understudied topic, usually monocular or depth completion is the focus. The contributions are: (1) a diffusion-based stereo depth estimation frame-work specifically designed for transparent and reflective objects. On the method side, a geometry-guided diffusion, a new synthetic dataset, evaluation on robot tasks.", "summary_of_recommendation": "Sufficient novelty and improvement over SoA.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See comments above, consider adding ablations and more information/data on the real-world experiments. Also consider adding mode limitations and failure cases in the limitations section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tWwNJWY0p", "reviewer": "Reviewer_Jtaw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Depth estimation is a crucial skill for robot control with vision feedback. However, existing depth sensor often reports incorrect raw depth, which limits robot performance. This paper proposes a new method for depth estimation using stereo image pair and raw disparity map either from sensor or traditional model peridtion. It uses a diffusion based approach to iteratively improve the disparity map conditioned on the stereo image pair. The performance of the proposed method is been validated on both real-world and simulated datasets. And it also perform well on real robot for vision feedback based manipulation task.  On top of that, experiment results show that the model is effective for challenging cases where objects are transparent or highly reflective. And a new synthetic dataset, HISS is created for training and testing the proposed method.", "summary_of_recommendation": "Depth estimation is a crucial skill for robot control with vision feedback. However, existing depth sensor often reports incorrect raw depth, which limits robot performance. This paper proposes a new method for depth estimation using stereo image pair and raw disparity map either from sensor or traditional model peridtion. It uses a diffusion based approach to iteratively improve the disparity map conditioned on the stereo image pair. The performance of the proposed method is been validated on both real-world and simulated datasets. And it also perform well on real robot for vision feedback based manipulation task.  On top of that, experiment results show that the model is effective for challenging cases where objects are transparent or highly reflective. And a new synthetic dataset, HISS is created for training and testing the proposed method.  However, there are still several questions about the method and dataset introduced in this paper. Firstly one of the novelty claimed in this paper is using diffusion model for disparity prediction. And the author considers this method to be an image-to-image translation instead of disparity prediction and claims this to be another novelty point. However, using end to end model to predict whole disparity map is not a new idea and can be found in many publications. claiming that to be a novel idea is questionable. Furthermore, using diffusion model for disparity/depth prediction is also not new, papers like DiffusionDepth and DDP already shows diffusion model can be used for this task.   Another question is related to the novelty of HISS dataset, compared to existing transparent object datasets, both real and rendered, it is lacking in terms of number of objects and scenes. What is the necessity to create a new dataset when existing dataset like ClearPose and SynTODD are already there.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. justify why using diffusion for disparity prediction is novel when existing works already did depth prediction using diffusion.\n2. what is the impact of the original disparity map on the final prediction, especially when raw map is wrong or incomplete for transparent and specular objects. what is the difference between disparity from sensor and from other model prediction?\n3. For rendering synthetic dataset, what is the method used to mimic the raw disparity? how do you make sure the simulated raw disparity is a good estimation of real-world, especially for transparent and specular objects.\n4. justify why not using existing transparent objects datasets instead of creating a new one, especially when existing datasets are better.\n5. Provide your model performance on existing transparent object datasets like ClearPose and SynTODD. \n6. For the claimed sim2real performance, report model eval result on ClearPose", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of Strengths:
* Important problem of estimating depth for translucent objects with an interesting approach
* A new dataset
* Experiment with a real robot
*  Clear improvement over SOTA

Summary of Weaknesses:
* Comparison with other SOTA - addressed  and to be added to the paper
* Needs to clarify novelty and parts of the method -  addressed and to be added to the paper
* Missing explanation about why it works on translucent objects -  addressed and to be added to the paper
Decision: Accept