Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
Authors: Jeffrey Ichnowski
Yahav Avigal
Justin Kerr
Ken Goldberg
Venue: CoRL2021 Poster (2021)
Keywords: neural radiance fields
grasp planning
transparent objects

Abstract: The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's view-independent learned density, place lights to increase specular reflections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner. We show how additional lights create specular reflections that improve the quality of the depth map, and test a setup for a robot workcell equipped with an array of cameras to perform transparent object manipulation. We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher. In each setting we show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90% and 100% grasp-success rates in physical experiments on an ABB YuMi, on objects where baseline methods fail.


Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
Jeffrey Ichnowski
Yahav Avigal
The AUTOLAB University of California Berkeley, USA jeffi@berkeley. edu edu
The AUTOLAB
University of California, Berkeley, USA yahav.avigallberkeleg. edu
Justin Kerr
Ken Goldberg
The AUTOLAB
The AUTOLAB
University of California, Berkeley, USA justin kertiinkerrrbrreeeyy edu
University of California, Berkeley, USA
goldbergåberkeley. edu
The ability to grasp and manipulate transparent objects is major chal- lenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's s view- independent learned density, place lights to increase specular reflections, and per- form a transparency-ave planner. We show how additional lights create specular reflections that improve the quality of the depth map, and test setup for robot workcell equipped with an array of cameras to perform transparent object manipulation. We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher. In each set- ting we show that NeRF and Dex-Net are able reliably compute robust grasps on transparent objects, achieving 90 % and 100 % grasp-success rates in physi- cal experiments on an ABB YuMi, on objects where baseline methods fail. See https://sites: google. com/ cwm/ciew/ /ie-/dex-weeeeeeeeeeeeeeeeeeee dex-nerf for code, video, and datasets.
Abstract:
denth-rendering that we feed into the Dex-Net grasp
Introduction
Transparent objects are common in homes, restaurants, retail packaging, labs, gift shops, hospitals, and industrial warchouses. Effectively automating robotic manipulation of transparent objects could have a broad impact, from helping in everyday tasks and performing tasks in hazardous environ- ments. Existing depth cameras assume that surfaces of observed objects reflect light uniformly in all directions, but this assumption does not hold for transparent objects as their appearance varies significantly under different view directions and illumination conditions due to reflection and fraction properties of transparent materials. In this paper, we propose and demonstrate Dex-NeRF, a new method to sense the geometry of transparent objects and allow for robots to interact with them.
Dex-NeRF uses a Neural Radiance Fields (NeRF) as part of pipeline (Fig. 1, right) to compute and execute robot grasps transparent objects. While NeRF originally proposed native for explicit volumetric representations and shown to render novel views of complex scenes realistically [1]. it can also reconstruct the scene geometry. In particular, due to the view-dependent nature of the NeRF model, can learn to represent the geometry associated with transparency ac- curately. The only input requirement to train NeRF model is set of images taken from camera with known intrinsics (e.g., focal length, distortion) and extrinsics (position and orientation in the world). While the intrinsies be determined from calibration techniques from the camera itself, determining the extrinsics is often a challenge [2, 3]. However, robots operating in fixed workcell with camera mounted the manipulator arm, can readily determine camera intrinsics. This makes NeRF a particularly good match for robot manipulators.
alter-
"quual contribution.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Dataset
Generation
NeRF Training
NeRF
Depthmap Rendering
3
ABR
AUTO
Grasp Planning
4
Dex-Net
Grasp
5
Exccution
Using NeRF to grasp transparent objects Given a scene with transparent objects (left column), we the pipeline the right to compute grasps (middle column). The top row shows Dex-NeRF working in simulated scene while the bottom row shows it working in physical scene.
Figure 1:
experiments, we show qualitatively and quantitatively that Dex-NeRF can compute high accuracy depth images from photo-realistic synthetic and real scenes, and achieve 90 % or better grasp-success rates on real objects.
The contributions of this paper are: (1) integration of NeRF with robot grasp planning, (2) trans- parency-aware depth rendering method for NeRF, (3) experiments on synthetic and real images showing NeRF with Dex-Net generates high-quality grasps, (4) synthetic and real image datasets with transparent objects for training NeRF models.
2 Related Work
Detecting Transparent Objects For robots to interact with transparent objects, they must first be able to detect them. The most recent approaches detecting and recognizing transparent objects are data-driven. Lai et al. [4] and Khaing et al. [5] propose using a Convolutional Neural Network (CNN) to detect transparent objects in RGB images. Recently, Xie al. [6] developed a transformer- based pipeline [7] for transparent object segmentation. Other methods rely on deep-learning models predict the object pose. Phillips et al. [8] trained random forest to detect the contours of trans- parent objects for pose estimation and shape recovery. Xu et al. [9] proposed two-stage method for estimating the 6-degrees-of-free of-frredom (DOF) pose of transparent object from a single RGBD image by replacing the noisy depth values with estimated values and training a DenseFusion like network structure [10]. Sajjan et al. [11] extend this and incorporate a neural network trained for 3D pose estimation of transparent objects in robotic picking pipeline. Zhou et al. [12, 13] train a grasp planner directly on raw images from light-field camera. Zhu al. [14] used an implicit function to complete missing depth given noisy RGBD observation of transparent objects. However, these data-driven methods rely on large annotated datasets that are hard to curate, whereas Dex-NeRF does not require any prior dataset.
Neural Radiance Fields Recently, implicit neural representations have led significant progress in 3D object shape representation [15, 16, 17] and encoding the geometry and appearance of 3D) scenes [18. 1]. Mildenhall et al. [1] presented Neural Radiance Fields (NeRF), a neural network whose input is a 3D coordinate with an associated view direction, and output is the volume density and view-dependent emitted radiance. Due its view-dependent prediction, NeRF can represent non-Lambertian effects such as specularities and reflections, and therefore capture the geometry of transparent objects. However, NeRF is slow to train and has low data efficiency. Yu al. [19] proposed Plenoctrees, mapping coordinates to spherical harmonic coefficients, shifting the view- dependency from the input the output. addition, Plenoctrees pre-samples sparse octree structure, achieving a significant speedup in training over NeRF Deng et al. [20] proposed JaxNeRF, an efficient JAX implementation of NeRF reduces the training time of a NeRF model from over day to several hours. Deng et al. [21] add depth supervision to train NeRF 2 to
model into

---

6x faster given fewer training views. Adamkiewicz et al. [22] proposed an algorithm that uses NeRF model for robot navigation. this work, we propose to use NeRF to recover the geometry of transparent objects for the purpose of robotic manipulation.
Robotic Grasping Traditional robot grasping methods analyze the object shape identify suc- cessful grasp poses [23, 24, 25]. Data-driven approaches learn a prior using labeled data [26, 27] or through self- sufs-suueviiion over many trials in a simulated or physical environment [28. 29] and gen- eralize to grasping novel objects with unknown geometry. These approaches rely on RGB and depth sensors to generate an accurate observation of the target object. Additionally, different methods use different inputs, such as depth maps [30, 31, 32], point clouds [33, 34, 35, 9], octrees [36], truncated signed distance function (TSDF) [37, 38). In contrast, in this paper we propose a method to render high-quality depth map from NeRF model to then pass to Dex-Net [30] to compute grasp. While standard depth cameras have gaps in their depth information that needs to processed out with hole-filling techniques, the depth map rendering from NeRF is directly usable. It is possible that other grasp- plaaniin techniques may be able to plan grasps from NeRF models.
Problem Statement
We assume an environment with an array cameras at known fixed locations or that the robot manipulate camera (e.g., wrist-mounted) capture multiple images of the scene. Given the environment with rigid transparent objects, Dex-NeRF computes a frame for robot gripper that will result in stable grasp of a transparent object.
4 Method
This section provides brief background NeRF, then describes recovering geometry of transpar- ent objects, integrating with grasp analysis, and improving performance with additional lights.
4.1 Preliminary: Training NeRF
NeRF [1] learns a neural scene representation that maps 5D coordinate containing a spatial location (ar y: 2) and viewing direction (9, o) to the volume density and RGB color c. Training NeRF's multilayer perceptron (MLP) requires multiple RGB images of a static scene with their correspond- ing camera poses and intrinsic parameters. The expected color C(r) of the camera ray r + /d between near and far scene bounds L, and by is:
C(r) [ (4 T(t)(((())c((((t)))) d)dtt
where (tt) exp (-f", (r((s)) is the probability that the camera ray travels from near bound tn to point without hitting any surface. NeRF approximates the expected color C(r) as:
((r) = Σ Ti(1 exp(-otd:))c;;
==1
where T; = exp (- (-Ej- o5 and 6, 4,- 4 is the distance between consecutive samples on the ray The training process minimizes the error between rendered and ground-truth colors.
4.2 Recovering Geometry of Transparent Objects
We observe that NeRF does not directly support transparent object effects-it casts single ray per source image pixel without reflection, splitting, or bouncing. NeRF recovers non-Lambertian effects such as reflections from specular surface by regressing on view direction and supervising with view-dependent emitted radiance. However, while RGB color c is view-dependent, the volume density is met-menning NeRF has learn non-zero to represent any color at that spatial location. The usual result that the transparent object shows up as a "ghostly" or "blurry" version of the original object in rendered RGB images.
When training, NeRF model learns a density a of each spatial location. This density corresponds to the transparency of the point, and serves to help learn how much a spatial location contributes to

---

Real Image
RealSense Depth
Depth (Dex-NoRF)
Figure 2: Comparison RealSense Depth Camera. We compare the results of the proposed pipeline in a real-world setting against depth map produced by an Intel RealSense camera. the left image is real-world scene, the middle shows the depth image from the RealSense, and the right shows the result of our pipeline. The color scheme in the RealSense image is provided by the RealSense SDK. while the color scheme the right column is from MatPlotLib. We observe that the RealSense depth camera is unable to recover depth from large portion the scene, shown black. On the other hand, the proposed pipeline, while having few holes, can recover depth for most of the seene.
Depth Map
(C) Depth Map Dex-NeRF
Depth Difference
Dex-Net Grasp
Dex-Net Grasp
(A) Real Image
(D)
(E) Vanilla NeRF
(F) Dex-NeRF
Vanilla NeRF
Figure 3: Using NeRF to render depth for grasping transparent objects. Dex-NeRF uses a transparency- aware depth rendering render depth maps that can be used for grasp planning. In contrast, Vanilla-NeRF's depth maps are filled with holes and result poor grasp predictions.
the color of ray cast through it. Although NeRF converts each o, to occupancy probability a; cxp(-cx (0,,,;),, where 5, is the distance between integration times along the ray, thus implicitly giving a; upper bound of 1, it does not place bound the value. Dex-NeRF uses the raw value of a to determine if point in space is occupied.
4.3 Rendering Depth for Grasp Analysis
To compute a grasp from a trained NeRF model, we propose to render a depth image and have Dex-Net use it to plan the grasp. To generate depth image, consider two candidate recon- structions depth. First, we use the same depth rendering that NeRF uses. This Vanilla NeRF reconstruction first converts di to occupancy probability ag It then applies the transformation w; = a; |1 (1 (1. To render depth pixel coordinate [u, v]. computes the sum of sample distances from the camera weighted by the termination probability D(u, v] Σ w;d; When applied on transparent objects, however, this results in noisy depth maps, as shown in Fig. 3.
Instead, consider second, transparency-awas the ray for which a, > m, where m a fixed threshold. The depth is then set to the distance that sample d,- We explore different values for m, and observe that low values result in noisy depth map while high values create holes in the depth map. In our experiments we set m = 15 (see Fig. 8).
method that searches for the first sample along
4.4 Improving Reconstruction with Light Placement
For NeRF to learn the geometry of transparent object, it must be able to "see" from multiple camera views. If the transparent object is not visible from any views, then it will have effect on the loss function used training, and thus not be learned. We thus look for way improve visibility of transparent objects to NeRF.

---

One property that transparent objects share (e.g., glass, clear plastic) is that they are glossy and thus produce specular reflections when the camera view direction is opposite the surface normal of the incident direction of light. To the NeRF model, specular reflection viewed from multiple angles will appear as bright point on a solid surface- c 11 and a > O, while from other angles it will appear as < 0. As a is view-independent, NeRF learns a a between fully opaque and fully transparent for such points.
By placing additional lights in the scene, we create more angles from which cameras will see specu- lar reflections from transparent objects- this results in NeRF learning a model that fills holes in the scene. While the number and placement of lights for optimal training is dependent on both the ex- pected object distribution and camera placement, in experiments (Sec. 5.5) we show that increasing from light to a 5x5 array of lights improves the quality of the learned geometry.
Experiments
We experiment in both simulation and a physical ABB YuMi robot. We generate multiple datasets, where each dataset consists of images and associated camera transforms of one static scene containing one more transparent objects. We train NeRF models using modified JaxNeRF [20] implementation on Nvidia V100 GPUs. We use an existing pre-trained Dex-Net model for grasp planning without modification or fine-tuning. We can do this since NeRF models can be rendered to depth maps from arbitrary camera intrinsics and extrinsics, thus we match our NeRF rendering to the Dex-Net model instead of training a new one.
5.1 Datasets
As existing NeRF datasets do not include transparent objects, and existing transparent-objee grasping datasets do not include multiple camera angles, we generate new datasets using 3 different methods: synthetic, Cannon EOS 60D camera with Tamron Di II lens with locked focal length, and an Intel RealSense.
For synthetic datasets, we use Blender 2.92's physically-based Cycles renderer with path tracing set to 10240 samples per pixel, and max light path bounces set to 1024. We chose theses settings by increasing them until renderings were indistinguishable from the previous setting-findingg lower settings lead to dark regions and smaller specular reflections. For glass materials, we set the index of refraction to 1 1.45 to match physical glass. We include 8 synthetic datasets of transparent objects: 2 scenes with clutter: light array and single light; 4 singulated objects from Dex-Net: Pipe Connector, Pawn, Turbine Housing, Mount; and 2 household objects: Wineglass upright and Wineglass on side. As these computationally demanding to render due to the high quality settings, distribute these as part of the contribution.
that
For the Cannon EOS and RealSense real-world datasets, we place ArUco markers in the scene to aid in camera pose recovery and take photos around the objects using fixed ISO, f-stop, and focal length. We use bundle adjustment from COLMAP [2, 3] to refine the camera poses and intrinsics to high accuracy. We include 8 physical datasets of transparent objects with variety of camera poses: table with clutter, Dishwasher, Tape Dispenser, Wineglass on side, Flask, Safety Glasses, Bottle upright, Lion Figurine in clutter. The main difficulty in generating these datasets is calibration and computing high-precision camera poses.
The datasets (at https: //s:://sitteee google. wwwgll..co/vviii. focus on scenes with transparent objects in a graspable setting, with over camera poses each.
differ from prior work in their
5.2 Synthetic Grasping Experiments
We test the ability of Dex-NeRF to generate grasps on the synthetic singulated transparent Dex-Net object datasets. For each dataset, we evaluate the grasp in simulation using a wrench resistance metric measuring the ability of the grasp to resist gravity [39]. Fig. shows images of the synthetic objects, Mex-NNRFF-generad the effect of training time on grasp-success rate, we simulate and record grasps over the course of training. In Fig. 5, we observe that grcspssuccess rate improves with training time, but plateaus between 80% % and 98 % success rate at around 50k to 60k iterations. This suggests that there may be a practical fixed iteration limit to obtain high grasp success rates.
depth map, and an example sampled grasp for each. To measure
5

---

Grasp Success Rate (%)
100
80
60
Pipe Connector Pawn
Turbine Housing Mount
90
10
20 30
40
50
60
70
80 50 100
Training epochs (x 1000)
Pipe
Housing
Grasp-success rate training Figure 4: Synthetic singulated objects used in simulation ex- epochs. As opposed to view-synthesis, which periments. Top row: image of the object in the training data. requires over 200k epochs, we observe high Bottom row: computed depth map and candidate grasp.
Figure S:
grasp success rates after 50k to 60k epochs.
Object
PhoXi
Vanilla NeRF
Dex-NeRF
Tape Dispenser
0/10
0/10
10/10
Wineglass
0/10
0/10
9/10
Flask
0/10
1/10
9/10
Dispenser
Wineglass
Flack
Safety Glasses
0/10
0/10
10/10
Bottle
0/10
10/10
10/10
Lion Figurine
0/10
3/10
10/10
Table 1:
rate. For each object, we compute depth map using a PhoXi camera, unmodified Vanilla NeRF, and Dex-NeRF for grasping transparent objects. From the depth map, Dex-Net computes 10 different grasps, and ABB YuMi attempts the grasp. Successful grasps lift the object.
Physical grasp
Safety Glasses
Bottle
Lion Figurine
Figure 6: Physical grasps objects. In the back- ground the base the YuMi robot.
We test Dex-NeRF scene of tabletop cluttered with transparent objects. In this experiment, the goal is to grasp a transparent object placed in stable pose in close proximity to other transparent objects. The challenge twofold: the depth rendering quality should be sufficient for both grasp planning and collision avoidance. Fig. shows the robot and scene in the upper left, and the overhead image, depth, and computed grasp inline in the pipeline, and the final computed grasp with simulated execution is in the upper middle image. The final grasp contact point was accurate to 2 mm tolerance, suggesting that Dex-NeRF with sufficient images taken from precisely-knows camera locations may be practical in highly cluttered environments.
5.3 Physical Grasping Experiments
To test the Dex-NeRF in a physical setup, we place transparent singulated objects front of an ABB YuMi robot, and have the robot perform the computed grasps. We compare to 2 baselines: (1) PhoXi, in which a PhoXi camera provides the depth map; and (2) Vanilla NeRF, in which we the original depth rendering from NeRF. The PhoXi camera is normally able generate high- precision depth maps for non-transparent objects. All methods use the same pre-trained Dex-Net model, and both Vanilla NeRF and Dex-NeRF the depth rendering. We test with objects (Fig. 6), and compute and execute 10 different grasps for each and record the success rate. grasp successful if the robot lifts the object. In Table 1, we see that Dex-NeRF gets 90% % and 100 % success rates for all objects, while the baselines get few successful grasps. The PhoXi camera unable recover any meaningful geometry which causes Dex-Net predictions to fail. The Vanilla NeRF depth maps often have unpredictable protrusions that result Dex-Net generating unreliable grasps.
the same NeRF model-the only difference is
5.4 Comparison to RealSense Depth
We qualitatively compare the rendered depth map of the proposed pipeline against readily-available depth sensor on scenes with transparent objects real-world settings (Fig. 2). We select the Intel RealSense as is common to robotics applications, readily available, and high-performance. The

---

(a) RGB Single Light Source
(b) Depth Rendering Single Light Source
Scene
Depth Remdering Multiple Light Sources
Multiple Light Sources
Figure 7: More lights mean more specular reflections, and result in better NeRF depth estimation of transparent surfaces. In (a) and (h), we show scene lit by a single overhead high-intensity light. In (c) and (d) we show the same scene lit by an overhead 5x5 array of lights. The combined light wattage equal in both scenes. Images (a) and (e) are views of the scene, and (b) and (d) are the corresponding depth images obtained from the pipeline. Two glasses on their sides are missing top surfaces (outlined in dashed red) in (b), while the effect is reduced in (d) due to the additional light sources.
0=1
0=5
0= 15
0 == 150
0= 500
Figure 8: depth rendering using NeRF with different thresholds Here we show the effect of the threshold value on the depth rendering of an isolated deer figurine. Values too low result in excess noise, while values too high cause parts of the scene to disappear.
RealSense, like most stereo depth cameras, struggles with transparent objects as they are unable to compute a stereo disparity between pixels from different cameras when the pixels are specular reflections or the color of the object behind the transparent object. The RealSense optionally projects structured light pattern the scene aid in computing depth from textureless surfaces; however, in experiments, we observed no qualitative difference with and without the light pattern emitter enabled. We use Canon EOS for NeRF, and use RealSense for depth image. In this experiment, we observe that the RealSense cannot compute the depth of most transparent objects and often produces regions of unknown depth (shown in black) where transparent objects are. On the other hand, the proposed pipeline produces high-quality depth maps with only a few noisy areas.
5.5
One vs Many Lights
We experiment with different light setups to test the effect of specular reflections on the ability of NeRF recover the geometry of transparent objects. We create two scenes (Fig. 7), one with single bright light source directly above the work surface, and another with an array of 5x5 (25) lights above the work surface. We set the total wattage of the lights in each scene to the same. Since most lights in the multiple light scene are further away from the work surface than the single light source, the scene appears darker, though more evenly illuminated. The effect of the specular reflections is prominent on the lightbulb in the lower part of the image. In the single light source, there is a single specular reflection, while in the multiple light scene, the reflection of the array of lights is visible.
With the same camera setup for both scenes, we train NeRF models with the same number of itera- tions. We show the depth rendering in Fig. 7 and circle a glass and a wineglass on their side. In the single-light source image, the closer surfaces of the glasses are missing, while in the multiple-light source depth image, the glasses are nearly fully recovered. This suggests that additional lights in the scene can help NeRF recover the geometry of transparent objects better.
5.6 Workcell Setup
We experiment with a potential setup for a robot workcell in which a grid of overhead cameras captures views of the cluttered scene so that a robot manipulator arm can then perform tasks with
7

---

9 Cameras
Cameras 25 Cameras
36 Cameras
49 Cameras
Figure 9: Depth rendering using grid of overhead cameras. Using increasing amounts of overhead cameras improves the quality of the depth map and its utility in grasping, however, beyond certain number of cameras there is diminishing return.
transparent objects in the workcell. We propose that a grid of overhead cameras would be practical to setup and would not obstruct manipulator tasks nor operator interventions. The objective is to de- termine how many overhead cameras would be needed to recover a depth map of sufficient accuracy perform manipulation tasks.
We place a m by m grid of cameras 1 m above the work surface, and have them all point at the center of the work surface. Each camera has the same intrinsics, and are evenly spaced along the grid. We experiment with grids having 4, 9, 16. 25, 36, and 49 cameras. The environment has the same 5x5 grid of lights as before. For each camera grid, train JaxNeRF for 50k iterations and compare performance.
After training, we observe increasing peak signal to noise ratios (PSNR) and structural similarity (SSIM) scores with increasing number of cameras. The 2x2 grid of cameras produces a high train-to- test ratio for PSNR, likely indicating overfitting to training data, and results in depth map without apparent geometry. This ratio decreases with additional cameras. The minimum number of cameras for this proposed setup appears to be around 9 (3x3) as its depth map is usable for grasp planning, while the 5x5 grid shows better PSNR and SSIM and ratio between train and test PSNR, and the 7x7 grid is the best. See Fig. 9 for a visual comparison. Additionally, grids, observing no statistically significant improvement beyond the 7x7 grid.
trained 9x9, 1 1x11, and 13x13
6 Conclusion
In this work, we showed that NeRF recover the geometry of transparent objects with sufficient accuracy to compute grasps for robot manipulation tasks. NeRF learns the density of all points in space, which corresponds to how much the view-dependent color of each point contributes to rays passing through it. With the key observation that specular reflections on transparent objects cause NeRF learn non-zero density, we have Dex-NeRF recover the geometry of transparent objects through a combination of additional lights to create specular reflections and thresholding to find transparent points that are visible from some view directions. With the geometry recovered, we pass it to a grasp planner, and show that the recovered geometry is sufficient to compute a grasp, and accurate enough to achieve 90 % and 100 % grasp success rates in physical experiments on an ABB YuMi robot. We created synthetic and real datasets for experiments in transparent geometry recovery, but we believe these datasets may of interest to researchers interested in extending NeRF capabilities in other ways and thus contribute them as well. Finally, to test if NeRF could be used in a robot workcell, we experimented with grids cameras facing a worksurface and their ability to recover geometry in potential setup, and showed the increased capabilities and point of diminishing return for additional cameras.
In future work, we hope to address one of the main drawbacks of NeRF the long training time re- quired obtain a NeRF model. Many research groups have sped up training time through improved implementations, new algorithms, new network architectures, pre-conditioned network weights, fo- cused sampling, and more. While these approaches apply to general NeRF training, we plan to exploit features specific to robot scenerios to speed up training, including using depth camera data cameras to inspect regions of interest, and
as additional training data, manipulator-amm-mouun visio-spatial foresight to adapt to changes in the environment.
8

---

Acknowledgments
This research was performed at the AUTOLAB at UC Berkeley affiliation with the Berkeley Al Research (BAIR) Lab, Berkeley Deep Drive (BDD), the Real-Time Intelligent Secure Execution (RISE) Lab, and the CITRIS "People and Robots" (CPAR) Initiative. We thank our colleagues who provided helpful feedback and suggestions, in particular Matthew Tancik. This article solely reflects the opinions and conclusions its authors and do not reflect the views of the sponsors or their associated entities.
References
[1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[2] L. Schönberger and J.-M. Frahm. Structure- Structureffrom-motiee froimmotton revisited. Conference on Com- puter Vision and Patlern Recognition (CVPR), 2016.
[3] L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured mueti-view stereo. In European Conference on Computer Vision (ECCV), 2016.
[4] P.J. Lai and C.-S. Fuh. Transparent object detection using regions with convolutional neural network. In IPPR Conference on Computer Vision, Graphics and Image Processing, volume 2, 2015.
[5] M. P. Khaing and M. Masayuki. Transparent object detection using convolutional neural net- work. In International Conference Big Data Analysis and Deep Learning Applications, pages 86-93. Springer, 2018
[6] E. Xie, W. Wang, W. Wang, P. Sun, H. Xu, D. Liang, and P. Luo. Segmenting transparent object in the wild with transformer. arXiv preprint arXiv:2101.08.0 2021.
[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. arXiv preprint arXiv: 1706 03762, 2017.
[8] C. J. Phillips, M. Lecce, and K. Daniilidis. Seeing glassware: from edge detection to pose estimation and shape recovery. In Robotics: Science and Systems, volume 3, 2016.
[9] C. Xu, J. Chen, M. Yao, J. Zhou, L. Zhang, and Y. Liu, 6DoF pose estimation of transparent object from a single RGB-D image. Sensors, 20(23):6790, 2020.
[10] C. Wang. D. Xu, Y. Zhu, R. Martin-Martin, C. Lu, I.. Fei-Fei, and S. Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3343-3352, 2019.
[11] S. Sajjan, M. Moore, M. Pan, G. Nagaraja, Lee, A. Zeng, and S. Song. Clear grasp: 3d shape estimation of transparent objects for manipulation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 3634-3642. IEEE, 2020.
[12) 7. Zhou, T. Pan, S. Wu, H. Chang, and O. C. Jenkins. Glassloc: Plenoptic grasp pose detection transparent clutter. arXiv preprint w:xiv::909.042200 2019.
[13] Z. Zhou, X. Chen, and O. C. Jenkins. Lit: Light-field inference of transparency for refractive object localization. IEEE Robotics and Automation Letters, 5(3):4548-45555 2020.
[14] L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. van Eenbergen, Debnath, and D. Fox. RGB-D D local implicit function for depth completion of transparent objects. arXiv preprint arXiv:2104.00622, 2021.
[15] I. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning reconstruction in function space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.

---

[16] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 165-174, 2019.
[17] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5535 5948, 2019.
[18] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous di-sstucture-awaan neural scene representations. arXiv preprint arXiv: (90:00996.01688, 2019.
[19] A. Yu, R. Li, M. Tancik, H. Li, R. Ng. and A. Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. arXiv preprint arXiv:2 /402,
[20] B. Deng. J. T. Barron, and P. P. Srinivasan. JaxNeRF: an efficient JAX implementation of NeRF, 2020. URL http://gitthl/ master/ jasner.
[21] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. arXiv e-prints, pages arXiv-2107, 2021.
[22] M. Adamkiewicz, T. Chen, A, Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwa- ger. Vision-only robot navigation in neural radiance world. arXiv preprint arXiv:2 10.00168, 2021.
[23] K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber. A survey on learning- robotic grasping. Current Robotics Reports, pages 1-11, 2020.
[24] A. Bicchi and V. Kumar. Robotic grasping and contact: A review. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 348-353. IEEE, 2000.
[25] R. M. Murray, Z. Li, and S. S. Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017.
[26] D. Kappler, Bohg, and Schaal. Leveraging big data for grasp planning. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4304 4311 IEEE, 2015.
[27] D. Prattichizzo. J. C. Trinkle, B. Siciliano, and O. Khatib. Springer handbook of robotics. Grasping; Springer: Berlin/HHeidelberg, Germany, pages 671-700, 2008
[28] E. Jang, C. Devin, V. Vanhoucke, and Levine. Grasp2vec: Learning object representations from self-supervised grasping. arXiv preprint 1rXiv:/811.066 2018.
[29] G. Peng. 7. Ren, H. Wang, and X. Li. A self-supervised learning baased 6-DOF grasp planning method for manipulator. arXiv preprint arXiv:2102.00000 00200, 2021.
[30] Mahler, J. Liang, S. Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and K. Goldberg. Dex- Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. arXiv preprint arXivv/703.0000 703.09932, 2017.
[31] I. Lenz, H. Lee, and Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Reseurch, 34(4-5):705-724 2015.
[32] J. Redmon and A. Angelova. Real-time grasp detection using convolutional neural networks. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1316-1322. IEEE, 2015.
[33] A. Mousavian, C. Eppner, and D. Fox. 6-DOF GraspNet: Variational grasp generation for object manipulation. Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2901-2910, 2019.
[34] Y. Qin, R. Chen, H. Zhu, M. Song, J. Xu, and H. Su. S4G: Amodal single-view single-shot SE(3) grasp detection in cluttered scenes. In Conference on robot learning, pages 53-65. PMLR, 2020.
10

---

[35] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-GraspNe Efficient 6-DoF grasp generation in cluttered scenes. arXiv preprint arXiv:2103.14127 2021.
[36] Y. Avigal, Satish, Tam, Huang, H. Zhang, M. Danielczuk, Ichnowski, and K. Gold- berg. AVPLUG: Approach vector planning for unicontact grasping amid clutter. In 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE), pages 1140-1147. IEEE, 2021.
[37] M. Breyer, J. J. Chung, L. Ott, R. Siegwart, and J. Nieto. Volumetric grasping network: Real- time 6 DOF grasp detection in clutter. arXiv preprint arXi::2101.01132 2021.
[38] S. Song, A. Zeng. J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6 DOF closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3): 4978 4985, 2020.
[39] J. Mahler, M. Mail, Liu, A. Li, D. Gealy, and K. Goldberg. Dex-Net 3.0: Computing robust vacuum suction grasp targets point clouds using a new analytic model and deep learning. In 2018 IEEE International Conference robotics and automation (ICRA), pages 5620-5627.
IEEE, 2018


Reviews:
[{"review_id": "mYzmzq6c8-o", "reviewer": "Reviewer_nhV7", "summary": "The paper presented a method on recovering depth of transparent objects with \"Neural Radiance Field\", and the potential application in grasping. While the recovering depth part showed meaningful results, the paper does not provide much data point on the application of grasping, thus the main focus is on depth recovering.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: adopted NRF to recover depth of transparent objects, which is usually hard for conventional approaches.\n\nWeaknesses: Does mention grasping in the title but not providing sufficient amount of data in experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Unless provide more data on grasping or switch to focus purely on depth recovering, I would suggest on weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1 grasping should include more details, e.g. examples, experiments, metrics, so that reader could see the difference between having the proposed approach and baseline.\n2 more details on experimental setup, e.g. how the cameras are configured, what metrics has been used. more quantitive results on a set of defined metrics."}}, {"review_id": "mVWAZY_z9a", "reviewer": "Reviewer_21Zr", "summary": "This paper proposes to render depth maps of transparent objects with NeRF. The rendered depth maps can be used for grasping these transparent objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- NeRF can model transparent objects accurately. Based on the learned NeRF representation, depth maps can be rendered.\n- This paper shows that using a threshold on density to locate boundary of the object and placing additional lights in the scene can improve the quality of the learned geometry.\n\nWeakness:\n- No quantitative comparisons. For example, the quantitative comparison between the depth maps rendered by vanilla NeRF and the proposed method (in simulation), and the grasp success rate given depth maps from RealSense, vanilla NeRF, and the proposed method (in simulation and real-world).\n- There's not much novelty in the method part. It's just utilizing an existing method (NeRF) to a relatively new problem (render depth map of transparent objects).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It's an interesting idea to render depth maps of transparent objects with NeRF. However, the method is only slightly different from NeRF in depth rendering part. Most importantly, no quantitative experiments are conducted to prove the validity of the method. Only qualitative results are shown. It's hard to judge the quality of the obtained results.\n\nGiven the authors' concrete response, I would like to change to weak accept.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "As stated in weakness. \n\nAlso, the proposed method only uses reflection to locate the surface of the object. One important property of transparent objects, the distortion is not handled. If the method can utilize the clues of distortion, the few light sources would be enough to mode the geometry. That would be very interesting."}}, {"review_id": "LDjYK-PvUBa", "reviewer": "Reviewer_hn8F", "summary": "This paper tackles the problem of grasping transparent objects by using combination of NeRF and dexnet. It takes in multiple view of the scene and trains a nerf model that underneath predicts occupancy probability of each point in the workspace volume. They also add some small modifications to nerf where it reasons about view independent occupancy and also instead of aggregating the occupancy probabilities along the ray it thresholds the occupancies and terminates the depths at the first location that has a probability higher than a threshold. From the predicted depth, they use dexnet to generate grasps and pick up the objects.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This is a systems paper on how to integrate different modules properly to solve the task. I like the idea of using nerf for depth prediction and using dexnet\n- The idea of using more lights to improve the depth quality also makes sense.\n\nWeaknesses:\n- What is the success rate on the whole system in grasping real transparent objects?\n- How long does the training of nerf takes?\n- What are the failure cases look for depth prediction and (depth prediction and grasp generation) look like?\n- As far as I understand NeRF, \\sigma is a probability between 0 and 1 for likelihood of a point being occupied or not. However, the thresholds are all more than 1. I would like to get clarification on why all these numbers are greater than 1.\n- video of real robot experiments missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the general idea of the paper but I think the paper focuses too much on just depth prediction and I'd like to see how authors responds to the points I raised above.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses above.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "B8vjPy3z6g", "reviewer": "Reviewer_qjfP", "summary": "This paper applies the wildly popular and promising NeRF -- wherein we learn a radiance field network that is integrated along a ray, resulting in estimates of the RGB and depth -- to the setting of transparent object detection for robotic grasping. For the latter, the paper makes use of another existing and widely-used method, DexNet. \n\nThe main contribution of the paper is showing that depth maps produced by NeRF integration are of sufficient high-quality for robotic grasping. This is important because typical 3D perception sensors relying on infrared imaging do a very poor job at handling the specularities and transparent nature of glass, the depth maps they output are almost unusable for manipulation. A secondary contribution is the creation of datasets of transparent objects, both synthetic and real.         ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nS1. The paper is well written and it addresses a very important and challenging problem in a practical way.\n\nS2. It confirms that the NeRF depth map is practical for depth-based grasping and manipulation, which is very important.\n\nS3. It contributes to addressing the issue of data collection for pose detection and grasping of transparent objects, without a need for object models and without a lot of need for manual handling of the objects in the scene. I think this is also very important.\n\nWeaknesses\n\nW1. Technically speaking, the proposed method is not novel. It \"just\" combines two existing methods. In fact, the original NeRF paper showcased a few examples of how it worked with transparent objects. That said, I think the whole is more than the sum of its parts here, and the paper offers many insights and experimental results about the performance of NeRF in a wide variety of transparent objects. I think this worthwhile enough for me to completely ignore the issue of novelty.\n\nW2. I'd have appreciated some more details about the size of the 3 contributed datasets and how they differ from other papers proposing similar datasets. \n\nW3. I'd have appreciated some discussion on the relationship between NeRF and ray-tracing methods. Can NeRF handle second-order light effects from ray bounces and shadows? What happens when you have transparent objects occluding other transparent objects?\n\nW4. Section 4.2 was quite unclear to me. After reading it, it was still ambiguous whether the proposed method uses view-independent or view-dependent volume density sigma?\n\nW5. It's not very clear how or whether the ground truth object pose was collected in the Cannon EOS datasets.  \n\nW6. The future work directions are mostly about speeding up the training NeRF, and not necessarily about increasing its accuracy. I'd have appreciated some discussion in that direction. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending that the paper be accepted. It addresses a very important and challenging problem in a way that alleviates the need for calibrated data collection, which is a very tedious process. While I would have liked some more extensive discussion on W3, W4, and W6, I find the paper to be useful enough to push this sub-area of robot perception forward.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "It would be great if weaknesses W2-W6 could be addressed by the rebuttal deadline. They'd place the paper in much better context.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper's contributions in regards to depth estimation have been well received, both in terms of its methodology and on the contribution of datasets. The writing is clear and the reporting of the investigations is broad and insightful. The methodology is sensible and original while its motivation is easy to understand.  

However, it seems there is confusion when it comes to robotics evaluation. One interpretation is that there was no real grasping experiment at all, although Figure 1 implies that Dex-Net was used for the Fetch robot (if Fig. 1 is a simple illustrative figure this should be made clear). Assuming that no real-robotics experiments were done then sentences such as "Experiments on synthetic and real images showing NeRF with Dex-Net generates high-quality grasps." and "Grasping real-world Objects" are not supported with experimental, quantitative evidence (e.g. success rates). On the other hand, if real grasping experiments were done, then the paper is missing deeper evaluations which should be added or at least clarified during the rebuttal.

Post-rebuttal ====================

The authors clarified the main concerns and weaknesses during the meta-review. As a result, the reviewers have been positive in raising their scores. The authors now provide strong experimental evidence on the efficacy of their method.
Decision: Accept (Poster)