D$^3$RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation

Authors: songlin  wei
haoran  geng
jiayi  chen
congyue  deng
wenbo  cui
chengyang  zhao
xiaomeng  fang
leonidas  guibas
he  wang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:49:07

Abstract: Depth sensing is an important problem for 3D vision-based robotics. Yet, a real-world active stereo or ToF depth camera often produces noisy and incomplete depth which bottlenecks robot performances. In this work, we propose D3RoMa, a learning-based depth estimation framework on stereo image pairs that predicts clean and accurate depth in diverse indoor scenes, even in the most challenging scenarios with translucent or specular surfaces where classical depth sensing completely fails. Key to our method is that we unify depth estimation and restoration into an image-to-image translation problem by predicting the disparity map with a denoising diffusion probabilistic model.  At inference time, we further incorporated a left-right consistency constraint as classifier guidance to the diffusion process. Our framework combines recently advanced learning-based approaches and geometric constraints from traditional stereo vision. For model training, we create a large scene-level synthetic dataset with diverse transparent and specular objects to compensate for existing tabletop datasets. The trained model can be directly applied to real-world in-the-wild scenes and achieve state-of-the-art performance in multiple public depth estimation benchmarks. Further experiments in both simulated and real environments show that accurate depth prediction significantly improves robotic manipulation in various scenarios.

Main paper: 
 introduction: 
 With the extensive use of stereo cameras, stereo depth estimation has been one of the most widely studied problems in robotics for determining the target object position or acquiring 3D information of the environment [1, 2, 3, 4] . However, the depth maps provided by existing stereo cameras suffer from severe noise, inaccuracy, and incompleteness issues, bottlenecking robot performances regardless of their well-developed recognition and manipulation algorithms.Traditional stereo-to-depth algorithms such as SGM [5] have fundamental issues: (i) In principle, they cannot tackle non-Lambertian surfaces due to the intricate light paths; (ii) Occlusion and outof-view areas prohibit the computation of pixel correspondences. Recent works have leveraged learning-based techniques for acquiring or restoring better depth maps [6, 7] . While they alleviate the above issues to a certain extent, predicting the depth for transparent and specular objects remains challenging as their image features from RGB pixel values are inherently ambiguous due to foreground-background color blending and thus can be misleading to correspondence estimation [8] .In this work, we propose D 3 RoMa. Instead of building our network with cost volumes as in most prior works, we transform the dense matching problem in depth estimation into an image-to-image translation problem by predicting the disparity map with a denoising diffusion probabilistic model. Such a paradigm does not rely on low-level feature matching but rather unleashes the power of generative models to directly translate the left and right frames into the target disparity image. More concretely, our method brings twofold benefits: (i) Unlike the regression models as in prior works, the diffusion model in our framework enables generative modeling of multi-modal depth distributions for transparent or translucent surfaces. (ii) The multi-step denoising process resembles the iterative solver, replacing prior iterative networks such as RAFT-Stereo [7] and HitNet [9] .Additionally, at inference time, we further incorporate the geometric constraints from traditional stereo vision by introducing a left-right consistency loss. The loss is integrated into the diffusion sampling process as a classifier guidance. The whole paradigm combines learning-based predictions and traditional geometric modeling through a simple summation of their gradients in the score function of the diffusion model.To train the network, we craft a synthetic dataset HSSD-IsaacSim-STD (HISS) of around 10, 000 stereo image pairs simulating real active stereo infrared patterns, including more than 350 transparent and specular objects in more than 160 indoor scenes [10] . Our dataset greatly extends the existing datasets that are limited to near-diffusive materials, table-top settings, or without realistic depth sensor simulation [11, 12, 13, 14, 15, 16] . Trained on our synthetic dataset, our model can be directly applied to real-world in-the-wild scenes (Figure 1 ) and achieve state-of-the-art performances not only on traditional stereo benchmarks but also on datasets targeting specular, transparent, and diffusive (STD) objects. To further validate our effectiveness in robotic manipulation, we conduct experiments on both simulated and real environments ranging from tabletop grasping to mobile grasping in indoor scenes. We observe that with the high-quality depth maps and 3D point clouds predicted by our method, the success rates of robotic manipulation can be significantly improved in diverse settings.To summarize, our contributions are: (1) A diffusion model-based stereo depth estimation framework that can predict state-of-the-art depth and restore noisy depth maps for transparent and specular surfaces; (2) An integration of stereo geometry constraints into the learning paradigm via guided diffusion; (3) A new scene-level STD synthetic dataset that simulates real depth sensor IR patterns and photo-realistic renderings; (4) Significant improvements in robotic manipulation tasks with our higher-quality depth maps and 3D point clouds. 
related work: 
 Stereo Depth Estimation and Completion. Modern deep-learning stereo methods [7, 17] typically have the following structures. First, a feature encoder are used to extract the left and right image features. The feature encoders are either pre-trained and frozen or trained end-to-end. Second, a cost volume is built by enumerating all the possible matching. Some works incorporate 3D CNN or attention networks to increase the receptive field of convolution layers which have proved to be beneficial [18, 19] . Finally, a detection head is added to regress the disparities. The most successful ingredient is the iterative mechanism which is proposed by the seminar work RAFT [6] [7] . On the other hand, Weinzaepfel et al. [20] uses cross-view completion pertaining and achieves impressive results. Despite all the progress that has been made, in the real world, transparent and specular objects are ubiquitous and the RGB features of the surfaces are inherently ambiguous to be used to search for correspondence because the foreground and background colors are blended.Previous work [13] tried to restore the missing depths with the help of neighboring raw depth values and their RGB color clues [12] [21] but with limited generalizability. Another line of work directly fine-tunes a trained deep stereo network [14] on transparent surfaces allowing the feature encoders to learn to match the transparent surfaces. However, the aforementioned foreground and background ambiguities confuse the feature-matching-based pipeline when dealing with regular diffuse objects.Diffusion Model for Depth Estimation. Recently, researchers have employed diffusion models to estimate optical flow [22] and predict depths with a single RGB image input [23] [24] [25] [26] . Such monocular methods can estimate the depth map up to an unknown absolute scale. Bhat et al. [27] proposed to train an extraneous network to predict the scale and achieve decent accuracy. However, the monocular methods either lack the absolute scale or have inferior accuracy for robotic manipulation tasks. Another line of work has pioneered adjusting the diffusion model to stereo settings. Nam et al. [28] proposed to learn matching based on cost volume in a diffusion manner, which could not handle matching ambiguities. Shao et al. [29] proposed to refine raw map for high quality human reconstruction. The authors designed a novel linear scheduler and condition on all the stereo-related information. Nonetheless, we found that using the default DDPM [30] scheduling works well and only needs to condition on stereo images and raw disparity if necessary. Additionally, we combined the stereo matching loss gradient with the learned gradient by the diffusion model. The stereo matching loss is obtained by checking the left-right image photometric consistency in an unsupervised manner [31] [32] . Such guided diffusion model [33] achieves the best results in our experiments.3D Vision-Based Robotic Manipulation. 3D vision is becoming increasingly critical for robotic manipulation [34] . Most basically, depth perception enables robots to comprehend the size, shape, and position of objects within three-dimensional space, thereby facilitating more sophisticated and reliable interactions [35, 36] . Moreover, numerous works [37, 38, 39, 40] utilize RGB-D point clouds as input. However, the substantial domain gap between simulated and real RGB-D images can result in a significant sim-to-real gap [41] . Additionally, transparent and specular objects exacerbate this issue, leading to poor depth-sensing performance. Policies trained in simulators often struggle to transfer effectively to real-world scenarios, particularly in the context of mobile manipulation. Our proposed high-performance depth estimation network is a promising direction to improve existing 3D vision-based tasks. 
method: 
 In this section, we introduce D 3 RoMa, a disparity diffusion-based depth sensing framework for material-agnostic robotic manipulation. Our framework focuses on improving the accuracy of disparity map in depth estimation, especially for transparent and specular objects which are ubiquitous yet challenging in robotic manipulation tasks. Given an observation of the scene, our framework takes the raw disparity map D and the left-right stereo image pair I l , I r from the depth sensor as input, and outputs a restored disparity map x 0 , which will be converted into the restored depth map. With the geometry prior from the stereo matching between I l and I r as guidance for the reverse sampling, our diffusion model can gradually perform the denoising process conditioned on D to predict the restored disparity map x 0 .Stereo Vision and Depth Estimation. Once the disparity map x for the observed points between a pair of stereo cameras is known, the depth map d for the points can then be calculated using the camera intrinsic parameters through d = (f • b)/x, where f and b are the camera focal length and the stereo baseline, respectively. The estimation of the disparity map x is traditionally modeled as a dense matching problem, which can be solved within the image domain. Thus, stereo depth estimation can be studied independently from different camera devices.Denoising Diffusion Probabilistic Model. Diffusion models [30] [42] are special latent variable models that reverse the diffusion (forward) process which gradually diffuses the original data x 0 through a Markov process. When the network s θ is trained converged, the gradient of the noise distribution also called the score function [43] isEQUATIONDuring inference, data samples can be generated through ancestral sampling which resembles the stochastic gradient Langevin dynamics (SGLD) [44] xEQUATIONIn this work, we formulate the stereo depth estimation problem as an image-to-image translation problem in the diffusion model. One important design choice is what to condition on. The model is usually formulated to condition on the stereo image pairs I l , I r for stereo depth estimation. Our experiments found that conditioning additionally on raw disparity D makes the network converge faster during training and generalizes more robustly in out-of-distribution scenarios. The raw disparity can be easily obtained either from a traditional stereo matching algorithm SGM [5] or from the real camera sensor outputs. For real active stereo depth sensors like RealSense, the left and right images are captured by infrared (IR) cameras with special shadow patterns projected by an IR projector. As a result, conditioning on the left and right images and the raw disparity map D, we train a conditional diffusion model to learn the distribution of the disparity mapEQUATIONwhere y = {I l , I r , D}. Empirically, this conditional denoising network has been shown to be successful [45] [46] . Batzolis et al. [47] further provided proof (see Theorem 1) that the conditional score ∇ xt p(x t |y) can be learned through the same training objectives as in the unconditional case even though the condition y does not appear in the training objectives. After the network is trained, the estimated disparity can then be sampled throughEQUATIONInspired by the classier guidance to image generation tasks [48] [49], we propose to guide the disparity diffusion process with the model-based geometry gradient. The guided reverse process is illustrated in Figure 2 . Specifically, the conditional score function is perturbed with gradient computed from stereo-matchingEQUATIONwhere L sm is the similarity loss function which compares the left image with the warped left image.The warped left image is obtained by warping the right image with the estimated disparity. s ∈ R + controls the geometric guidance strength and it balances the learned gradient from diffusion model and geometric gradient from stereo models. A detailed derivation of Equation 5 is provided in Appendix B. To mitigate the gradient locality in stereo matching, we downsample the stereo images into multiple different lower resolutions when computing the gradient of stereo matching. More specifically, we haveEQUATIONwhere k is the index of the layer for different resolutions and γ ∈ R + is a weighting constant balances the two photometric and smoothness losses. The L ssim is the structural similarity index (SSIM) [50] which computes the photometric loss between the left image I l and warped image Ĩleft :EQUATIONwhere u, v are the pixel coordinates in the image plane and ⟨ ⟩ is linear sampling operation. L smooth is an edge-aware smoothness loss [31] [32] defined asEQUATIONwhich regularizes the disparity by penalizing large discontinuity in non-edge areas. Here ∂ u means partial derivative in u (horizontal) direction in the image plane. Then we predict the disparity map x 0 with the perturbed gradient from Equation 5following the sampling process introduced in Equation 2. Finally, We can convert the disparity to depth once we know the camera parameters.We create our synthetic dataset HISS based on Habitat Synthetic Scenes Dataset (HSSD) [10] . We leverage the 168 high-quality indoor scenes from HSSD to increase scene diversity. For objects, we include a total of over 350 object models from DREDS [13] and GraspNet [34] . The scene and randomly selected object CAD models are rendered in Isaac Sim [51] . During rendering, object materials and scene lighting are specifically randomized in simulation to mimic the transparent or specular physical properties of objects (cups, glasses, bottles, etc.) in the real world. To obtain the correct depth values for transparent surfaces, we adopt a two-pass methodology. We first render RGB images and depth maps of the scenes with object materials set to diffuse. The lightings are all turned to enable photorealistic rendering. In the second pass, we turn the normal lighting off and project a similar shadow pattern on the scenes to mimic the RealSense D415 infrared stereo images. Using the intrinsics of the RealSense D415 depth camera, we render over 10,000 photo-realistic stereo images with simulated shadow patterns. Experiments demonstrate that our dataset is the key enabler of our method's excellent generalizability in the real world. 
depth estimation in robotic scenarios: 
 DREDS. We evaluate our method on DREDS [13] , a tabletop-level depth dataset with both synthetic and annotated real data for specular and transparent objects. We compare our method with several state-of-the-art baselines: 1) NLSPN, 2) LIDF, 3) SwinDR, and 4) ASGrasp, which are four methods that have been shown effective on depth estimation for transparent or specular objects. We also compare another 3 stereo depth SoTA methods: RAFT-Stereo, IGEV-Stereo, and CroCo-Stereo. We use mean absolute error (MAE), relative depth error (REL), root mean square error (RMSE), along with 3 other metrics related to depth accuracy as metrics for evaluation. We include detailed descriptions for the baselines and the metrics in the Appendix C.As shown in Table 1 , on the DREDS-CatKnown data split (synthetic data), all variants of our method surpass all baselines on all metrics. Further, the results of our ablations show that the performance of our method can be steadily improved with more information provided, especially the integration of the raw disparity. More ablation studies regarding the geometry-based guidance, network hyper-parameters and architectures are provided in Appendix F.SynTODD. SynTODD [15] is another synthetic dataset for transparent objects by using Blender. It contains 87512 train images and 5263 test images. The authors proposed a novel multi-view method (MVTrans) to estimate object depths, poses, and segmentations. Because the dataset provides neither simulated raw disparity nor correct camera intrinsic for stereo images, We compare with MVTrans [15] using our monocular variant, ie., we modify the network to condition only on RGB images. We do scale alignment after the prediction of the scale-invariant depth following Mi-Das [53] . As shown in Table 2 , Our method achieves better performance than all the variants of MVTrans including 2-views, 3-views, and 5-views.ClearPose ClearPose [16] is a large-scale real-world RGB-D benchmark for transparent and translucent objects. The dataset contains 350,000 real images captured by the RealSense L515 depth camera. The authors collected a set of very challenging scenes including different backgrounds, heavy occlusions, objects in translucent and opaque covers, on non-planar surfaces, and even filled with liquid. We evaluate our method on ClearPose in all settings. Our method D 3 RoMa outperforms two previous SoTA ImplicitDepth [54] and TransCG [55] by a large margin as shown in We further demonstrate the effectiveness of our method for stereo matching in general scenarios. We compare our method with state-of-the-art stereo-matching baselines on SceneFlow [57] , a synthetic dataset containing more than 39,000 stereo frames in 960×540 pixel resolution. The dataset contains three challenging scenes, FlyingThings3D, Driving, and Monkaa, which makes it a high-quality dataset for pre-training [7, 18, 20] . We train our model from scratch using 35,454 stereo pairs for training and leave the rest as testing split. We also resize the images in the dataset into 480×270 to be consistent with our robotic perception settings. Following the previous work [18] , the ground truth disparity is normalized using the maximum disparity value 192, which is also used to crop the test data. More implementation details are provided in Appendix E. As shown in Table 4 , we achieve the best results compared to existing state-of-the-art methods. In the grasping experiments, we first acquire the depth map by D = (f • b)/X. Then back project the depth into point cloud P = DK -1 P , where K ∈ R 3×3 is the camera intrinsics and P are the homogeneous points in the image plane corresponding to each pixel. With the restored point cloud, we leverage GSNet [62] to predict 6 DoF grasping poses. To increase the grasping success rate for all baselines, we filter the grasping pose which has the angle between the grasping pose and the z (up) direction less than 30 degrees. We always select the grasping pose with the highest core and transform it into the robot base frame.Environment Setup. We set up a tabletop grasping, articulated object manipulation and a mobile grasping environment in the real world, as shown in Figure 4 . In the tabletop grasping experiments, we use a Franka 7-DoF robotic arm. We place STD objects on surfaces with bumps and pits, which are challenging setups for both depth sensing and robot grasping. The objects have non-diffusive surface materials such as glass, porcelain, glass, etc.In the mobile grasping experiments, we use a customized wheeled robot with 7 DoF arms as shown in Appendix E.3. The robot is arbitrarily placed near the target objects omitting the navigation phase which is beyond our scope. We place the objects in clusters at different tables and furniture with both flat and non-flat surfaces and different backgrounds, bringing challenges to depth sensing with the complex scene environments. More details about the articulated object manipulation experiments can be found in Appendix J.Results and Analysis. We compare our method with two other baselines. All baselines use the same motion planner CuRobo [63] but different depth sensing. We also compare with ASGrasp [14] which was mainly designed for table-top grasping of STD objects. We report the results for different objects (STD) respectively and a overall success rate in Table 6 . The quantitative results for three different scenes of mobile manipulation are provided in Table 5 . While ASGrasp and D 3 RoMa both improved over the raw sensor outputs, our method outperforms ASGrasp with a large margin.  
conclusion: 
 In this work, we proposed a novel geometry gradient guidance to diffusion model in disparity space to predict depth for stereo images. Conditioning on stereo image pair and a raw disparity map, our network achieves SOTA performance on existing benchmarks. Both disparity evaluation on pure synthetic datasets and depth evaluation on depth datasets demonstrate the efficiency of our method.Our key observations include data diversity can make a big impact on generalization in the wild and guidance helps in more challenging real-world scenes. Current 3D vision-based robotics manipulation pipelines including grasping and part manipulation can be significantly improved simply by improved depth perception. Especially, we found that depth estimation for challenging transparent objects could be better dealt with generative models than traditional stereo methods.Limitations and Future Work One limitation inherited from the diffusion model is the iterative inference with many denoising steps. This issue can be mitigated by incorporating diffusion sampling acceleration techniques studied in a variety of existing works. Another limitation is that the EPE is linear to the resolution of the input image. Future research could use coarse-to-fine or simple sliced techniques to split high resolution images into smaller patches to resolve this issue.where we apply Bayes' theorem again in the third equation. Substitute Equation 13 back to Equation 12, we haveEQUATIONThe first term is learned by the denoising network and the second term is the geometric guidance which can be calculated by stereo matching. In the experiments, we leverage more available data such as I r and D in addition to I l into the network during training:EQUATIONHere we empirically scale the geometry gradient with s ∈ R + and set it to 1 in the experiments. 
 
 Appendix: In addition to the left and right IR images, active stereo cameras provide another color image I c captured from a third color camera. While the above derivation directly applies to active stereo cameras if we ignore the color image, we found that further feeding the color image into the network slightly improves the performance in DREDS [13] . However, most stereo datasets are passive and do not have additional color images. Therefore, during mixed dataset training, this additional color image is dropped. Here, we provide an active stereo version of derivation analogous to Equation 13:EQUATIONwhere the third equation assumes p(I l |I c , x t ) = 1. The I c and I l are already aligned and the only difference is the shadow pattern projected from the camera IR projector. The shadow pattern is irrelevant to the depth. Therefore, I c is approximately the sufficient statistic of I l . For the same argument, we have log p(I r |I l , x t ) = p(I r |I l , I c , x t ). Likewise, the guidance for the active stereo camera can then be obtained by substituting Equation 16into Equation 12:EQUATIONIn active stereo vision scenarios, we further train the network by conditioning it also on other available images. We set y = {I l , I r , I c , D}.Baselines. NLSPN [21] is a depth completion work that uses an end-to-end non-local spatial propagation network to predict dense depth given sparse inputs. LIDF [12] proposes to learn an implicit density field that can recover missing depth given noisy RGB-D input. SwinDR [13] proposes a depth restoration framework based on SWIN transformer and is trained on a proposed table-top dataset with STD objects (DREDS). ASGrasp [14] proposes a stereo-depth estimation method based on Raft-Stereo to predict two-layer depths for tabletop grasping. Raft-Stereo [7] is the seminal deep stereo network. To this day, it is still the most adopted architecture in stereo vision.Disparity Metric. End-Point Error (EPE) = 1 H×W |X -X| is the mean absolute difference for all pixels between the ground truth and estimated disparity map.Depth Metrics. We use the following depth metrics: 1) RMSE = Because our method is diffusion model based, we inherited the stochasticity in the reverse sampling process. To visualize the stochasticity, we run the same input 10 times. The uncertainty is obtained as the variance of the output disparity map. We conduct the experiments on DREDS and show the results in Figure 5 . We observed that high uncertainty area corresponds to object edges where depth dramatically changes between foreground and background. Flat surfaces have lower uncertainty as the geometry is simpler. Such uncertainty could be used to filter outliers. While our method works only in stereo cases, there are seminar works predicting depth given single RBG images. The attractive part of monocular depth estimation (MDE) is that more data is available for training. Therefore, these methods can generalized well in the wild. While some monocular methods like ZeoDepth [27] propose to recover metric depth after a special training procedure, most monocular methods predict relative depth. The relative depth can be recovered with an absolute scale which can be obtained via other sensors like lidar or prior knowledge. However, our experiments (Figure 6 ) found that most monocular methods produce inferior quality depth even without considering the absolute scale.We used libSGM [64] , a CUDA-accelerated, widely adopted implementation of the Semi-Global Matching (SGM) [5] algorithm. To seamlessly integrate libSGM into our pipeline, we utilized py-bind11 to encapsulate the original codebase within our Python-based framework. This integration allows the adapted version of libSGM to achieve a performance of approximately 55 frames per second (FPS) at an input resolution of 960 × 540, with around 380MB of memory allocated on an NVIDIA RTX 4090 GPU. We implement our network using Hugging Face Diffusers [65] and pre-compute raw disparity maps using libSGM [64] . The network is trained 600 epochs with the batch size 6×8 and a constant learning rate 0.0001. All the images are randomly cropped to 320×240 and no other data augmentation is used during training. We use cosine scheduler [66] with 128 denoising time steps for β t starting at 0.0001 and ending at 0.02. We use UNet as our denoising network. In the DREDS experiments, we have 6 downsampling ResNet blocks each layer has 128, 128, 256, 256, 512, and 512 channels. The second-to-last channel is a downsampling block with spatial attention. We use MSE as our loss function. For the SceneFlow experiment, we scale down the original image resolutions from 960×640 into 480×270. We use a multi-resolution pyramid noise strategy as in [26] . We further use pretrained StableDiffusion v2 [67] in the grasping experiments and adapt the input Conv block accordingly to the conditioning inputs [26] . We also train the mixed datasets including DREDS, HISS, and SceneFlow at the batch level.In the grasping experiments, we mount the RealSense D415 on the wrist of the arm. After the camera captures a frame, we first acquire the depth map by D = (f • b)/X. Then back project the depth into point cloud P = DK -1 P , where K ∈ R 3×3 is the camera intrinsics and P are the homogeneous points in the image plane corresponding to each pixel. With the restored point cloud, we leverage GSNet [62] to predict 6 DoF grasping poses. To increase the grasping success rate for all baselines, we filter the grasping pose which has the angle between the grasping pose and the z (up) direction less than 30 degrees. We always select the grasping pose with the highest core and transform it into the robot base frame. Then we grasp the object with a motion planner like CuRobo [63] . We did not perform workspace point cloud cropping operation as in the baseline ASGrasp [14] hence leading to an overall success rate drop in the main text compared with the numbers reported in ASGrasp.We use a wheeled mobile base mounted with two 7 DoF customized arms in the real mobile grasping experiments. Each arm attaches a parallel gripper. We only use the left arm in the experiments. Figure 7 displays the robot and the workplace. Since DREDS does not provide IR images for the STD-CatKnown and STD-CatNovel data split (real data), we train the variant of our framework which only conditions on RGB image and raw disparity to compare with SwinDR.We further perform ablation studies for the geometry-based guidance on the STD-CatKnown and STD-CatNovel data split to validate its effectiveness for diffusion-based depth estimation in realworld scenarios. As Table 7 shows, our geometry-based guidance can significantly boost performance, especially for out-of-distribution scenarios. More ablation studies regarding network hyperparameters and architectures are provided in Appendix F. We provide ablation studies on the DREDS dataset in Table 8 . The baseline is conditioned on the left, and right image and raw disparity. Its hyperparameters and network architecture are described in Appendix E.2. We also trained variants with different network architectures, loss functions, and noise strategies. We reduce the channels from 512 to 256 of the last two layers denoted as reduced channels. We also changed the loss function from MSE to L1 and used the default standard Gaussian noise. The main factors of run time are the input image resolution and the number of denoising steps. To better understand the computation-accuracy tradeoff, we list the runtime of our method and other SoTAs in Table 9 . We also evaluate the effects of different samplers in the real experiments where we used pretrained StableDiffusion [67] . We report the inference time of our method in Table 10 .The network has about 865M parameters in total for this variant. We fixed the number of time steps during training to 1000 and used the same standard cosine scheduler [66] , and all the samplers take 10 denoising steps during inference. We perform reverse sampling using different schedulers implemented by Diffusers [65] . All the samplers achieve similar qualitative results except Euler Ancestral. The results are shown in Figure 8 . Empirically, we select DDPM with 10 denoising steps and a resolution of 640×360 in our real experiments. This section will study alternative guidance to the diffusion model during the reverse sampling processes. In the stereo vision case, the gradient of the photometric loss is obtained by checking the consistency of the left and right images. Mathematically, the gradient should also have the same direction with x 0 -x t where x 0 is the ground truth disparity. In test time, x 0 is unknown but can be approximated by an external less noisy measurement source such as a Lidar. The external depth measurement can be converted to the disparity space x0 and is multiplied with a mask if it is sparse:EQUATIONWe here experiment with raw depth guidance. The guidance x0 is approximated by camera raw sensor depth. Therefore the sign(x 0 -x t ) is the approximate gradient. We set mask ω(u, v) = (x 0 > 0) and α is a constant controls the guidance strength. We qualitatively study the guidance of the approximate gradient in Figure 9 . The benefits of guidance by the approximate gradient are limited when raw depth is highly noisy. We provide the results of our method on ClearPose compared with two other baselines as shown in Table 11 . There are a total of 6 different scenes, including different backgrounds, heavy occlusions, objects in translucent and opaque covers, on non-planar surfaces, and even filled with liquid. In this section, we train other SOTA stereo methods from scratch and compare with our method on the HISS dataset. We further rendered 300 images in 5 new scenes different from our training dataset as the test set. The results are given in Table 12 . We also show more real depth estimation results in Figure 12 and more comparisons in Figure 13 , which we consider also attributed to the joint training on our dataset. One of the motivations for using the diffusion model to predict depth is the multi-step reverse sampling process. It resembles the iterative solver which has been proven successful in RAFT [6] and its successors. In figure 11 we show an example of the denoising process trained on our HISS dataset. The total denoising steps is set to 128 and we visualize every 32 timesteps.HISS. We further evaluate the effectiveness of our dataset for transparent and specular object depth estimation. We compare our method with the previous state-of-the-art methods. As shown in Figure 13 , compared with RAFT-Stereo [7] , which is trained on large-scale datasets for stereomatching, our method can predict better depth, especially on transparent bottles. To ensure fair comparisons, we further fine-tune RAFT-Stereo on HISS for 400,000 epochs. Compared to the original model, the fine-tuned RAFT-Stereo can recover the missing depth of transparent objects better but the object shapes are still inaccurate. We also compare our method with ASGrasp [14] which is specially designed to detect and grasp transparent objects based on depth estimation. It has a similar performance to the fine-tuned RAFT-Stereo but has blurred object boundaries. Our methods can provide the best depth for all STD objects, with significantly clearer object boundaries  


Decision: Summary of Strengths:
* Important problem of estimating depth for translucent objects with an interesting approach
* A new dataset
* Experiment with a real robot
*  Clear improvement over SOTA

Summary of Weaknesses:
* Comparison with other SOTA - addressed  and to be added to the paper
* Needs to clarify novelty and parts of the method -  addressed and to be added to the paper
* Missing explanation about why it works on translucent objects -  addressed and to be added to the paper

Reviews: review: title: title: Review for paper titled D3RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation
paper_summary: paper_summary: This paper proposes a denoising diffusion probabilistic model method to estimate accurate depth, even for translucent objects from stereo images for indoor scenes by casting the image restoration and depth estimation problem into an image-to-image translation problem. They collected a synthetic dataset with specular objects to train and better evaluate the proposed method. Various experiments are performed in both simulation and the real world to prove the effectiveness of the proposed method.
main_review: main_review: **Strengths:**
1. This paper handles an important problem of estimating depth for translucent objects with an interesting approach.
2. The evaluation of the proposed method is done on various tasks and configurations and provides a clear improvement over the state of the art. 
3. Good ablation studies are performed to show the effectiveness of various configurations and the proposed design choices. 

**Concerns/weaknesses:**
1. The accurate depth results for translucent objects currently seem like a happy result of the proposed black-box diffusion model. I request the authors to comment on how the diffusion models can solve this problem compared to the cost-volume-based approaches.

2. I can only see run time evaluations of the proposed method but, it would be nice to have the comparison with the other methods from Table 1 to understand the trade-offs between accuracy, speed, and memory.
3. Please explain why adding disparity resulted in better convergence and results. Also, please comment on how this improved the depth estimation of translucent objects. 
4. Please comment on what happens when the pre-processed disparity is bad. Did you perform any evaluation on progressively destroying the disparity by adding noise or masking?
5. I see the quantitative results for RAFT-Stereo in the sup-material. Please try to add them to the main paper for better understanding and readability. 

Please address my concerns in the rebuttal response.
questions: questions: Please see the concerns/weaknesses above.
limitations: limitations: 2
review_summary: review_summary: Overall this is a well written paper with good evaluation and ablation studies. However, I have some concerns and questions. I would like to see the response from the authors during the rebuttal process.
title: title: Impressive performance with diffusion and new dataset
paper_summary: paper_summary: This paper addresses the problem of depth perception of transparent objects with a stereo camera. This is an understudied topic, usually monocular or depth completion is the focus. The contributions are: (1) a diffusion-based stereo depth estimation frame-work specifically designed for transparent and reflective objects. On the method side, a geometry-guided diffusion, a new synthetic dataset, evaluation on robot tasks.
main_review: main_review: This work addresses the important but understudied problem of depth perception of transparent and reflective objects using a stereo camera. This is a task relevant to robotics as depth cameras are widely available and tend to provide more robust scale than monocular approaches. The contribution consists of a new dataset, which builds on DREDS to get simulated RGBD even for transparent and reflective objects. The authors improve the dataset by adding more generic scenes, as well as the transparent and reflective assets from DREDS.

On the method side, the authors introduce a geometry-aware loss to guide the diffusion and achieve better results - although I have not seen this ablated. The simulation results show D3RoMa is competitive against Raft-Stereo when exposed to the same data, although it would have been better to evaluate more SoA stereo methods in the same way. A limiting factor in evaluation is the lack of other stereo datasets for transparent objects. 

The robot experiments are great, and show the improvement in performance in the real world beyond the training set. Perhaps you could add some of the sensory readings of these experiments of the appendix? This would help explain the challenges in each of the subsets of the tasks. For example, the performance in figure 8 in the appendix is convincing.

Overall the authors have contributed a method for stereo mvs of transparent and reflective objects, a usually understudied subject. They have done so with significant improvements over SoA, a new method and dataset.
questions: questions: See comments above, consider adding ablations and more information/data on the real-world experiments. Also consider adding mode limitations and failure cases in the limitations section.
limitations: limitations: 3
review_summary: review_summary: Sufficient novelty and improvement over SoA.
title: title: review for D3RoMa
paper_summary: paper_summary: Depth estimation is a crucial skill for robot control with vision feedback. However, existing depth sensor often reports incorrect raw depth, which limits robot performance. This paper proposes a new method for depth estimation using stereo image pair and raw disparity map either from sensor or traditional model peridtion. It uses a diffusion based approach to iteratively improve the disparity map conditioned on the stereo image pair. The performance of the proposed method is been validated on both real-world and simulated datasets. And it also perform well on real robot for vision feedback based manipulation task.  On top of that, experiment results show that the model is effective for challenging cases where objects are transparent or highly reflective. And a new synthetic dataset, HISS is created for training and testing the proposed method.
main_review: main_review: Depth estimation is a crucial skill for robot control with vision feedback. However, existing depth sensor often reports incorrect raw depth, which limits robot performance. This paper proposes a new method for depth estimation using stereo image pair and raw disparity map either from sensor or traditional model peridtion. It uses a diffusion based approach to iteratively improve the disparity map conditioned on the stereo image pair. The performance of the proposed method is been validated on both real-world and simulated datasets. And it also perform well on real robot for vision feedback based manipulation task.  On top of that, experiment results show that the model is effective for challenging cases where objects are transparent or highly reflective. And a new synthetic dataset, HISS is created for training and testing the proposed method.

However, there are still several questions about the method and dataset introduced in this paper. Firstly one of the novelty claimed in this paper is using diffusion model for disparity prediction. And the author considers this method to be an image-to-image translation instead of disparity prediction and claims this to be another novelty point. However, using end to end model to predict whole disparity map is not a new idea and can be found in many publications. claiming that to be a novel idea is questionable. Furthermore, using diffusion model for disparity/depth prediction is also not new, papers like DiffusionDepth and DDP already shows diffusion model can be used for this task. 

Another question is related to the novelty of HISS dataset, compared to existing transparent object datasets, both real and rendered, it is lacking in terms of number of objects and scenes. What is the necessity to create a new dataset when existing dataset like ClearPose and SynTODD are already there.
questions: questions: 1. justify why using diffusion for disparity prediction is novel when existing works already did depth prediction using diffusion.
2. what is the impact of the original disparity map on the final prediction, especially when raw map is wrong or incomplete for transparent and specular objects. what is the difference between disparity from sensor and from other model prediction?
3. For rendering synthetic dataset, what is the method used to mimic the raw disparity? how do you make sure the simulated raw disparity is a good estimation of real-world, especially for transparent and specular objects.
4. justify why not using existing transparent objects datasets instead of creating a new one, especially when existing datasets are better.
5. Provide your model performance on existing transparent object datasets like ClearPose and SynTODD. 
6. For the claimed sim2real performance, report model eval result on ClearPose
limitations: limitations: 3
review_summary: review_summary: Depth estimation is a crucial skill for robot control with vision feedback. However, existing depth sensor often reports incorrect raw depth, which limits robot performance. This paper proposes a new method for depth estimation using stereo image pair and raw disparity map either from sensor or traditional model peridtion. It uses a diffusion based approach to iteratively improve the disparity map conditioned on the stereo image pair. The performance of the proposed method is been validated on both real-world and simulated datasets. And it also perform well on real robot for vision feedback based manipulation task.  On top of that, experiment results show that the model is effective for challenging cases where objects are transparent or highly reflective. And a new synthetic dataset, HISS is created for training and testing the proposed method.  However, there are still several questions about the method and dataset introduced in this paper. Firstly one of the novelty claimed in this paper is using diffusion model for disparity prediction. And the author considers this method to be an image-to-image translation instead of disparity prediction and claims this to be another novelty point. However, using end to end model to predict whole disparity map is not a new idea and can be found in many publications. claiming that to be a novel idea is questionable. Furthermore, using diffusion model for disparity/depth prediction is also not new, papers like DiffusionDepth and DDP already shows diffusion model can be used for this task.   Another question is related to the novelty of HISS dataset, compared to existing transparent object datasets, both real and rendered, it is lacking in terms of number of objects and scenes. What is the necessity to create a new dataset when existing dataset like ClearPose and SynTODD are already there.