SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo
Authors: Mike Laskey
Brijen Thananjeyan
Kevin Stone
Thomas Kollar
Mark Tjersland
Venue: CoRL2021 Poster (2021)
Keywords: Sim-to-Real
Computer Vision
Manipulation

Abstract: Robot manipulation of unknown objects in unstructured environments is a challenging problem due to the variety of shapes, materials, arrangements and lighting conditions.  Even with large-scale real-world data collection, robust perception and manipulation of transparent and reflective objects across various lighting conditions remains challenging.  To address these challenges we propose an approach to performing sim-to-real transfer of robotic perception.  The underlying model, SimNet, is trained as a single multi-headed neural network using simulated stereo data as input and simulated object segmentation masks, 3D oriented bounding boxes (OBBs), object keypoints and disparity as output.  A key component of SimNet is the incorporation of a learned stereo sub-network that predicts disparity.  SimNet is evaluated on unknown object detection and deformable object keypoint detection and significantly outperforms a baseline that uses a structured light RGB-D sensor. By inferring grasp positions using the OBB and keypoint predictions, SimNet can be used to perform end-to-end manipulation of unknown objects across our fleet of Toyota HSR robots. In object grasping experiments, SimNet significantly outperforms the RBG-D baseline on optically challenging objects, suggesting that SimNet can enable robust manipulation of unknown objects, including transparent objects, in novel environments.

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo
Thomas Kollar" 1 Michael Laskey" Kevin Stone".1 Brijen Thananjeyan 1.2 Mark Tjersland". .1
Robot manipulation of unknown objects in unstructured environments is a challenging problem due to the variety of shapes, materials, arrangements and lighting conditions. Even with large-scale real-world data collection, robust perception and manipulation of transparent and reflective objects across various lighting conditions remains challenging. To address these challenges we propose an approach to performing sim-to-real transfer of robotic perception. The underlying model, SimNet, is trained as single multi-headed neural network using simulated stereo data as input and simulated object segmentation masks, 3D oriented bound- ing boxes (OBBs), object keypoints and disparity as output. A key component of SimNet is the incorporation of learned stereo sub-network that predicts dis- parity, SimNet is evaluated on unknown object detection and deformable object keypoint detection and significantly outperforms baseline that uses structured light RGB-D sensor. By inferring grasp positions using the OBB and keypoint predictions, SimNet can be used to perform end-to-end manipulation of unknown objects across our fleet of Toyota HSR robots. In object grasping experiments, SimNet significantly outperforms the RBG-D baseline optically challenging ob- jects, suggesting that SimNet can enable robust manipulation of unknown objects, including transparent objects, in novel environments. Additional visualizations and materials are located at tttpp:///iiyurr/ com/simmeeeeeeeeee sttsstimntttttcc corl.
Abstract:
Keywords: Sim-to-Rest Computer Vision, Manipulation
Introduction
To deploy robots into diverse, unstructured environments, robots must have robust and general behaviors. Enabling general behaviors in complex environments, such the home. requires robots to be able to perceive and manipulate previously unseen objects, such as new glass cups or t-shirts, even in the presence of variations in lighting, furniture, and objects. A promising approach to enable robust, generalized behaviors is to procedurally generate and automatically label large-scale datasets in simulation and use these datasets train perception models [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. These models can extract the necessary representations for a wide variety of manipulation behaviors and can enable the manipulation policy to be implemented
classical planner [2, 8, 11, 12, 13].
However, perception models trained purely on simulated RGB data can overfit to simulation artifacts such as texture and lighting. In order to explicitly force models to focus on geometric features instead, models are often trained on active depth information [1, 3, 4, 6]. However, active depth sensors use structured light, which struggles in environments where reflective and transparent objects are present [3]. Natural home environments, which often have harsh lighting conditions and reflective or transparent objects such as glassware, motivates designing a method that is robust to these variations and can leverage geometric features without using depth sensors.
An alternative to active depth sensing is passive stereo matching, which captures images from two cameras and matches pixels in each image to single point in 3D space. The disparity. or horizontal difference in the pixel coordinates, of the point can be directly mapped to depth [14]. Recent work from the stereo vision community shows how to approximately perform stereo matching to predict depth images using a differentiable cost volume neural network that matches features in stereo images [15, 16]. Unlike prior work, which typically focuses on accurate reconstruction of depth images, this work the "low-level" features from approximate stereo matching [5, 17] are used as an intermediate representation for "high-level" vision tasks.
Toyota Research Institute. Uniierriity of Califomia, Berkeley. Denotes equal contribution, - B ordering.
Contact: {first name. last name) 0 tri tri.global
5th Conference on Robot Learning (CoRL 2021), London, UK.

---

CARS
SMALL OBJECTS
T-SHIRTS
Simulated data was generated for three domains: cars, small objects and t-shirts using non- non-photorealistic simulator with fomain--andomizztioon machines and can be generated in an hour for S60 (USD) cloud compute cost. By forcing the network to learn geometrie features, sim-to-real transfer can be performed using only very low-quality scenes.
Figure I:
Simulation images:
Dataset generation is parallelized across
To make perception robust in challenging home environments, we introduce SimNet, a lightweight neural network which leverages "low-level" vision features from learned stereo network for "high- level" vision tasks (Fig. 2). SimNet is able to be trained entirely on simulated data. By forcing the network focus on geometric features using Nomain-randonee a learned stereo network that is robust to diverse environments, SimNet learns to robustly predict representations that can be used for manipulation of unknown objects in novel scenes. SimNet is able to predict a variety of "high-level" outputs, including segmentation masks, 3D oriented bounding boxes and keypoints. In contrast previous work on manipulating unknown objects in novel environments, SimNet does not need large-scale real data collection [3, 18, 19, 20], active depth sensing [1. 3, 4], or photorealistic simulation [3. 21].
data (Fig. 1) and by using
SimNet has been evaluated on three real-world domains: unknown object grasping, t-shirt folding, and 2D car detection. For unknown object grasping, SimNet shows accuracy gains over previous depth-based approaches and is able to grasp optically challenging transparent objects even where other depth-based approaches fail (Fig. 3). The demonstration of t-shirt folding shows that sim-to-real transfer is possible with SimNet even the presence of deformable objects and challenging optical conditions. Finally, ablations of the SimNet model components are performed on the KITTI [22] 2D car detection task. These experiments demonstrate the benefits of using a cost-volume for stereo matching to enable sim-to-real transfer.
This work makes three contributions: (i) an efficient neural network for sim-to-real transfer, SimNet, that uses learned stereo matching to enable robust sim-to-real transfer of "high-level" vision tasks, such as keypoints and oriented bounding boxes (OBBs), (ii) the first network enable direct prediction of 3D oriented bounding boxes of unknown objects (iit) and an indoor scenes dataset with 3D oriented bounding box labels of common household objects, corresponding stereo and RGB-D images, and training code for our model.
2
Related Work
The related work is described for perception for manipulation, sim-to-real transfer and learned stereo.
2.1
Parameterized Representations for Manipulation
In contrast to end-to-end robot policy learning [19, 23, 24, 25], popular approach for generating behaviors is to parameterize motions by the outputs of a perception network [1, 2, 8, 11, 26). This decouples perception from planning and control, and enables perception systems to be trained in simulation without the need for accurate physical simulation. One representation commonly used for grasping of rigid objects are oriented bounding boxes (OBBs) [26]. By aligning grasps along the principal components, OBBs can be used to grasp common household objects [26]. Our work contributes method to directly predict OBBs from a neural network. A representation that is used for deformable manipulation are keypoints or learned correspondences, which have seen significant success in tasks such as deformable manipulation [2, 8. 10, 11, 12, 27] and grasping [28, 29]. Ganapathi et al. [8] and Sundaresan al. [2] predict correspondences from depth or monocular RGB images and demonstrate impressive transfer to real fabrics and ropes in constrained lab environments. In contrast, we demonstrate robust transfer on diverse home scenes, and compare to the methods in these papers as baselines.
2.2 Rim-to-Real Transfer
Enabling perception models trained on simulated data to transfer to real-world scenes is an active area of research. One technique to transfer from simulation to reality is to train directly geometric

---

Room Segmenistioo
III
0B8s
SCVN
Law foss Disparity
d
Reypoints
BesitetsFPN Backboone
Bight Image
I|
Fulli Ros Disperity
Output Heads
Figure 2: SimNet: In SimNet, each stereo RGB image is fed into feature extractor before being fed into SCVN. which performs approximate stereo matching. The output of SCVN is resolution disparity image fed with features from the left image to a ResNet-FPN backbone and output prediction heads. The output heads predict room-level segmentation, OBBs, keypoints, and full-resolution disparity images.
representation of the world such as point cloud or depth image [1, 2]. Prior work has used depth- based transfer techniques in variety of scenarios, such as grasping unknown objects [1], tying rope [2], and segmenting object instances in cluttered scenes [4, 6]. Depth- transfer is commonly performed with active sensor using structured light. The applicability of these sensors is limited to regimes where their reflected infrared pattern is consistent enough that can be pattern matched for 3-D reconstruction [30]. However, home settings contain nom-brrbennian objects such glassware and a large amount of natural light, which can create large sensing errors. Existing work fills in missing depth information for transparent objects by combining depth and RGB information from photorealistic simulation data and real data [3]. In contrast to [3]. which uses photorealistic simulation that requires artists to create high quality assets, SimNett uses inexpensive RGB sensors and low quality simulated data for training. SimNet comparable to Xie et al. [4], which separately leverages depth and synthetic RGB inputs compensate for the limitations of depth sensing alone. SimNet is compared to several variants of Xie et al. [4], and is found to outperform them all.
An alternative to depth- based transfer is domain randomization (DR) with RGB images [5, 17,31, 32). Typically. DR randomizes over the lighting and textures in the environment by utilizing low-quality, non- pontooorrraaiiitccccccccc renderer. DR has produced successful results in car detection [31]. ), 6-DOF pose estimation of known objects [33], and camera calibration [34]. By injecting large randomization into texture and lighting, the network is forced to use the geometry of the scene to solve the task [17]. A limitation of this upproach is that geometric reasoning on monocular image requires leveraging global shape priors, lighting effects, and scene understanding [35]. In this work, a sim-to-real transfer technique is proposed that uses learned stereo matching and domain randomization to extract geometric features using only local non-semantic information. Robustness of sim-to-real transfer is demonstrated on the KITTI benchmark. A similar result, [21]. was published concurrently and also demonstrates the ad advantages of stereo for sim-to-real transfer when using photorealistic simulation.
2.3 Learned Stereo Matching
The goal of stereo matching is to explicitly compute the pixel displacement offset between objects from the left and right images stereo pair [14]. By computing a per-pixel displacement offset, or disparity, high-resolution point cloud can be generated of the world. Given that it is challenging to estimate similarity features image patches, recent work has explored using deep learning to advance stereo techniques [15, 16, 36, 37, 38, 39, 40]. In contrast to the learned stereo community that targets achieving perfect depth-sensing when training with real data [15, 16], SimNet uses depth-sensing to enable sim-to- teal transfer for "high-level" vision tasks by regressing coarse depth with an efficient stereo matching network architecture based on [15, 16] and learning features that are robust enough to predict oriented bounding boxes, segmentation masks, and keypoints on real objects for manipulation.
3 SimNet: Enabling Predictions for Manipulation From Synthetic Stereo
In this section, we will discuss our network architecture that leverages approximate stereo matching techniques from [15, 16] and domain randomization [5. 17] predict segmentation masks, OBBs, and keypoints on unseen objects for robot manipulation. Our key insight is that by training approximate stereo matching algorithms on pure synthetic data, robust "low-level" features like disparity can be learned, thereby enabling sim-to-real transfer on "high-level" vision tasks. First, we discuss how to learn robust low-level features, which are then used for "high-level" perception. Then, we discuss
3

---

SimNet
O1Bs
Object Groups
Iub: Lamana 2
Execution
Figure 3: Manipulation overview: A stereo pair is fed as input to SimNet, which produces OBBs. Grasp positions are produced from OBBs and are used by a classical planner to grasp the object [26]. This grasping technique has been deployed on fleet of home robots to perform manipulation in optically challenging scenarios.
how our low-cost synthetic data is generated. For training details see supplemental Sec. B.2; the overall network architecture Fig. 2.
3.1 Stereo Cost Volume Networks (SCVN) For Robust Low-Level Features
The stereo cost volume network (SCVN) for performing robust learned stereo matching is described in this section; ablations performed in Sec. 4.4 show its benefits. Let 3 to denote Hadamard products, and [[t.j-k.:] denote the selection of all elements with index i in the first dimension of tensor I, index in {J....K k- 1} in the second dimension of I, and any index in the third dimension onwards. Let and I, denote the left and right RGB images from the stereo pair. Each image has dimension 3 x Ho x Wo. The left and right images are fed into neural networks Φ and Φ that featurize each image respectively and output feature volumes Φi and Φr. Both Φi and Φ. have dimension Co Hg x Wa. where Co is the number of channels in each feature volume, and H and Wa are their height and width, respectively. We used a lightweight Dilated ResNet-FPN [41] as our feature extractor, to enable large receptive fields with minimal amount of convolutional layers.
The extracted features and o, are fed into stereo cost volume network fenst that consists of an approximate stereo matching module that searches horizontally in the feature volumes for corre- spondences within allowed disparity range. In classical stereo vision literature, correspondences across left and right images can be found by searching along a horizontal line across the images for match, and the disparity is the difference in the coordinates in the match, which is high for closer points in 3D space and low for farther points. The architecture for foost is heavily inspired by the techniques in [15] and approximately performs this search. The first phase of the network p(0)) computes pixelwise dot products between horizontally shifted versions of the feature volumes. The output of this phase has dimension C. H& x W The value 2 (C. - 1) represents the maximum disparity considered by the network, and the minimum disparity considered The i-th H. x W. slice of the output is computed as:
Sele (1h,, de (Badd)))
= (or O 0r Pr fc0l (ch, (r) = 1=0
The first case takes the rightmost He columns of the left feature volume Φi and computes pixelwise dot product with the leftmost H. i columns of Φ.. This operation horizontally searches for matches across the feature volumes at disparity of 2i. The next phase of the network f(3)) feeds the resulting volume into a sequence of ResNet blocks, which outputs a volume of dimension C. x H& Wa before performing soft argmin along the first axis of the volume. The soft argmin operation approximately finds the disparity for each pixel by locating its best match. The final volume is an estimate of a low-resolution disparity image fa, with shape HA x W We denote
Scost = ( ((0))
Disparity Auxiliary Loss In addition to the losses for the high-level perception heads (Sec. 3,2 and 3.3), the weights of 4,, Φ,,, and feost trained by minimizing auxiliary depth reconstruction loss function. In particular, the loss function takes in a target disparity image Itarg,a dimension Ho Wo. downsamples it by factor of Ho/ Ho and then computes the Huber loss [42] fassmall of

---

with the low-resolution depth prediction feost (0t, Ôr). That is, the network weights are trained to minimize Lalsmall (fcost(or Φr), downsample (arrg Ho/Ho)
3.2 Extracting High-Level Predictions for Manipulation Tasks
Given a SCVN to extract geometric features from stereo images, we need to learn high-level predic- tions relevant to manipulation. To design a backbone for robust simulation-trained manipulation, we feed the output of SVCN, La Low into Resnet18- 18FFNN [43] feature backbone fbackbone. Addition- ally, early stage features from the left RGB image, 4, allow high resolution texture information to be considered at inference time, similar to [4]. The features are extracted from the ResNet stem, concatenated with the output of the SCVN and fed into the backbone. The output of the backbone is fed into each of the prediction heads.
The rest of this section describes how SimNet uses the output of the backbone for the core prediction heads (Fig. 2) and the losses used for training the network. The optional auxiliary prediction heads (i.e. segmentation and refined disparity) are described in Sec. 3.3. Each prediction head uses the up-scaling branch defined in [43], which aggregates different resolutions across the feature extractor.
Oriented Bounding Boxes: Detection of an OBB requires determining individual object instances as well as estimating translation, E TR3 scale E R3x33 and rotation, R E R3x3 of the encompassing OBB. These parameters can be recovered by using four different output heads. First, to recover object instances, we regress Wo x Hg image, which is the resolution of the input left image, where for each object in the image Gaussian heatmap is predicted. Instances can then be derived using peak detection as in [44, 45]. We use an L1 loss on this output head and denote the loss as luust-
Given instances of object, the remaining 9-DOF pose parameters can be regressed. To recover scale and translation, we first regress W Ho/8 x 16 output head where each element contains pixel-wise offset from detected peak to the 8 box vertices projected on to the image. Scale and translation of the box can be recovered up to scale ambiguity using EPnP similar to [44]. In contrast to prior work in pose estimation, the predicted box is aligned based on principal axes sized in fixed reference frame. To recover absolute scale and translation, the distance from the camera E R of the box centroid is regressed as a Wo/8 x Ho/8 tensor. The two losses on these tensors are an L1 loss and are denoted L and Leent-
Finally, the rotation of the OBB, R. can be recovered via directly predicting the covariance matrix, E E Rax3 of the ground truth 3D point cloud of the target object, which can be easily generated in simulation. The output tensor of Wa/8 /8/8x Ho/8 x is directly regressed, where each pixel contains both the diagonal and symmetric off diagonal elements of the target covariance matrix. Rotation can then be recovered based on the SVD of Σ. L1 loss on this output head is used and denoted as loov- Note that for the 9-DOF pose loses, the loss is only enforced when the Gaussian heatmaps have score greater than 0.3 to prevent ambiguity in empty space. For more details the OBB prediction head, http:/gtbb.cc//oooaaaeetteeeeeee
the code is located here: https: / /ggtttbt..b.....g//////g...... com www./oooytaaeeearrrroooooooaaaae ch wwstitttet/tttttmmmmmtoinnantttt titee/smmann ssimet..
Keypoints:
Keypoints and learned correspondences are common representation for robot manipula- tion, especially in deformable manipulation [2. 8, 11, 12, 28]. SimNet has an output head that predicts keypoints, such as t-shirt sleeves for t-shirt folding (Sec. 4). The output head predicts heatmaps for each keypoint class, and is trained to match target heatmaps with Gaussian distributions placed at each ground-truth keypoint location using a pixelwise cross-entropy loss 4kp. To extract keypoints from the predicted heatmaps, non-maximum suppression is used to perform peak detection [44].
3.3 High-level Predictions: Optional Auxiliary Prediction Heads
Two optional auxiliary prediction heads can be used to enable better seene understanding of the world. These prediction heads do not affect performance of the other tasks.
Room Level Segmentation: A room-level segmentation head can predict one of three categories: surfaces, objects and background. Cross-entropy loss (seg is used for training, as in prior work [43].
This prediction head enables a mobile robot to detect surfaces and manipulable objects.
Since the SCVN produces a disparity image at quarter resolution, the feature extractor can combine the backbone and the left stereo image to produce full resolution depth image. The same branch architecture as the previous heads is used to aggregate information across different scales. During training we use the same loss the SCVN, but enforeed at full resolution. This output is trained using a Huber loss function and is denoted fa. A full resolution disparity image can be converted into a 3D point cloud for collision avoidance.
Full Resolution Disparity:
5

---

Left Image
RGB D Model
SimMet
OBB regression task with objects of The RGB image from different homes. Middle: Our model prediction trained with RGB-D sim-to-real transfer similar to [4]. The top right corner is the output of the Asus Xtion depth sensor. Bottom: SimNet. In the top right, we show the low-res disparity estimate predicted by the learned stereo net. SimNet consistently enables better sim2real transfer of the predictions for optically challenging scenarios.
Figure 4: Graspable Object Predictions: SimNet is evaluated
varying sizes and shapes on flat surfaces. Top:
3.4 Efficient Synthetic Dataset Generation
Given the complexity of all the output predictions defined in the previous section, it would be impractical label a sufficient amount of real data generalize across scenes. Thus, we are interested in using synthetic data to provide ground truth annotations on a wide variety of scenarios. To force the network to learn geometric features, we randomize over lighting and textures as recommended in Mayer et al. [38]. In contrast to [3, 46], we use OpenGL shaders with PyRender [47] instead of physically based rendering [48] approaches. Low-quality rendering greatly speeds up computation, and allows for dataset generation on the order of an hour. Three datasets are generated: cars, graspable objects, and t-shirts (Fig. 1). Additional details about the datasets are supplemental Sec. D. The training and validation data for the graspable object dataset can be found here: https: //github www.coo////o/// hatm//oootaaaeeradale Insti www.ttttttttttteeeeeeetteee ssimnet..
Experiments
Three real-world computer vision and robotics experiments are performed to evaluate how well SimNet can learn from synthetic stereo data and transfer to diverse, real images in unstructured environments. First, SimNet is compared baseline approaches the small objects dataset and an ablation of multi-task training is performed to understand its benefits. Second, the best model from the first experiment is used to evaluate SimNet's performance two optically challenging manipulation tasks: (1) grasping unknown objects on tabletops and (2) t-shirt folding. Third, sim-to-real transfer for new domains is evaluated using the KITTI 2-D car detection task; additional ablations performed to show the benefit of the SCVN. With the exception of the manipulation experiments on the robot, all of the perception results averaged over trials; little variation between trials is noted.
All physical experiments are conducted on tabletops found across four different, real homes. Manipu- lation is performed with the Toyota HSR robot, which has four DOF arm, mobile base, and pair of parallel jaw grippers [49]. has mounted stereo pair from a Zed camera and Asus Xtion Pro RGB-D DD sensor. Each home has different background objects, furniture, graspable objects, and lighting conditions, which evaluates each network's 's ability robustly generalize diverse scenarios.
4.1 Ablation of Baselines on Validation Data
In this experiment, SimNet is trained on the small objects dataset and evaluated on manually annotated real-world validation set. SimNet is compared to the following baselines: mono, which uses the left stereo image, depth [50], which only uses depth inputs, and RGB-D, which uses both RGB and depth inputs. To make the RGB-D bascline competitive, the best among four different

---

RGR-D Mortel
Simhee
Figure 5: T-shirt Keypoint Predictions: SimNet is evaluated on keypoint regression for shirts in various stages of folding. Three classes of keypoints are predicted: sleeves, neck, and hottom corners. RGB-D performs poorly due strong natural lighting and minimal depth variation. SimNet accurately predicts keypoints the shirts despite these challenges.
approaches to fuse color and depth information was picked (See RGB-D baselines in Appendix, Section B. by evaluating on a held out set of 500 non-optically challenging scenes (i.e. matte objects with minimal natural light). SimNet is compared to a technique inspired by [4], RGB-D Seq. where inference is done a two-stage manner first on depth and then on RGB (see Section B.1). The performance of SimNet after turning off all auxiliary tasks (except auxilliary depth loss) is reported as SimNet- w/o Aux.. 3D mAP@0.22 of annotated bounding boxes each object, a common metric in pose prediction [51], was used as evaluation metric (see Appendix B.3), Results are shown in Table Additional sensing ablations are in the Appendix, Sec. C.
Monocular sensing performs poorly on this task, likely due to the task being 3D in nature, while depth and RGB-D perform much better. Both versions of SimNet (with without auxiliary tasks) perform better than all other sim-to-real techniques, including using RGB-D and the approach inspired by [4] (RGB-D Seq). The similar mAP score between SimNet-w/o Aux. tasks and SimNet shows that the additional output heads are optional and not needed for sim-to-real transfer. These results indicate that stereo matching lead to robust sim-to-real transfer.
Method mAP
Mono [5] Depth [50]
RGB-D
RGB-D Seq [4]
SimNet
SimNet- w/o Aux.
0.164
0.831
0.855
0.774
0.921
0.947
Table I: SimNet Perception Results: An ablation of different sensing modalities the 3D OBB prediction task evaluated using 3D mAP on dataset of real,
images of optically easy objects.
4.2 Unknown Object Grasping
In this manipulation experiment, the robot's task is to grasp objects on a tabletop comprised of two classes of household objects in each home: optically easy, which consists of opaque, non-reflective objects, and optically hard, which contains optically challenging, transparent objects (Fig. 6). For each trial, an object is selected uniformly at random from the dataset and randomly placed on the tabletop with other distractor objects. The task is to grasp the foremost object in the scene using a heuristic grasp planner that takes OBBs as input (Fig. 3). To grasp objects the robot aligns the gripper with the largest principal axis of the OBB. In the event of similar sized principal axes like ball, the robot favors grasping the object on the side closest to the robot. A grasp is successful if the robot able to raise the object off the table and remove it from the scene.
For each of the four homes test five easy objects and five hard objects and compare SimNet against the best RGB-D baseline found in Sec. 4. Quantitative results are reported in Table and qualitative results of the predictions are in Fig. 4. The results show that SimNet outperforms RGB-D, 92.5 % vs. 62.5% in grasp success. On optically easy objects RGBD-D and SimNet perform similarly on average, while SimNet performs much better when grasping challenging objects such as glassware. These results demonstrate that SimNet is able to robustly grasp a variety of unknown challenging objects, even in optically challenging settings.
4.3 Grasp Point Prediction in T-shirt Folding
In this manipulation experiment, the robot evaluated on a t-shirt folding task, where it must execute sequence of four folds on unseen, real t-shirts. This task is challenging perform using depth sensing, because the depth resolution most commercial depth sensors cannot capture the subtle variations in depth due to the thickness of a t-shirt. Keypoints are a popular representation for
7

---

Method (Object Class)
Home
Home
Home
Home 4
Overall
RGB-D (0, Easy)
4/5
5/5
4/5
5/5
18/20
SimNet (O. Easy)
5/5
4/5
5/5
4/5
18/20
RGB-D (O. Hard)
0/5
1/5
1/5
5/5
7/20
SimNet (O. Hard)
5/5
5/5
5/5
4/5
19/20
Table 2: SimNet Grasping Results: Grasp success scores comparing the best RGB-D method and SimNet.
manipulating deformable objects [11, 12]. A shirt folding policy is parameterized using keypoint predictions for the shirt's neck, sleeves, and bottom corners (Sec. B.4.2). To compute quantitative results on sim2real transfer, a validation dataset of 32 real images is collected from 12 t-shirts in homes (Fig. 5). Keypoint prediction mAP is reported (Table 3). Videos of t-shirt folding are presented on the project website: https: / /tiittuill...//.. com/ stt/ssimmntttcoom corl.
The results show that SimNet significantly outperforms RGB-D and depth, and slightly outperforms mono. This is likely due to interference from natural lighting and low depth profile of the shirts. The monocular network transfers well to the real t-shirt images, and SimNet slightly outperforms it. Unlike the models that use active depth sensing, SimNet is robust to lighting variation and the low depth variation of the shirts. Also, these experiments indicate that SimNer can accurately predict object keypoints, thereby enabling manipulation of certain deformable objects.
Method mÁP
Mono
Depth 0.282
RGB-D
SimNet 0.917
0.893
0.631
Table 3: T-Shirt Folding Results: comparison of t-shirt keypoint mAP across different sim-to-real approaches.
4.4 SCVN Ablation on KITTI 2-D Car Detection
To motivate the use of the SCVN for achieving sim-to-real transfer, evaluation on the standard 2-D car detection benchmark on KITTI was performed [22] since it has been used as challenging sim-to-real domain in prior work [31]. The models compared include SimNet (with the SCVN), which uses simulated stereo images, Mono, which uses monocular inputs, Stacked, which uses stacked left and right image inputs, NimNet-No Depth Loss, which uses synthetic stereo images but does not have a depth-auxiliary loss on the output of the SCVN and SimNet-real, which real training data with a standard validation/trainin to use the single-shot box detection output head in CenterNet [45]. m.P@0.5 is reported, similar to [31], for the aggregate of moderate and easy car classes
split used in [45]. To predict 2-D bounding boxes, SimNet is modified
The results can be seen in Table 4. They show that SimNet can significantly outperform monocular images and naive stereo concatenation, which suggests explicit stereo matching leads to more robust transfer. However, if no auxiliary depth loss is enforced, there is a significant decrease in performance, indicating that the SCVN very important for sim-to-real transfer. Furthermore, the gap between real and sim data is only 3.5%, which suggests relatively robust sim-to-real transfer. However, qualitatively SimNet fails to detect cars at distance, since the disparity for their pixels becomes very small as distance from the camera becomes large. Visualizations of the predictions can be seen the Appendix, Fig. 13. Thus, SimNet is best used for applications with limited range from the camera, such as for robot manipulation.
Method
Mono
Stacked
SimNet-No Depth Loss SimNet 0.826
SimNet-real 0.861
mAP
0.565
0.710
0.612
Table 4: KITTI Results: Comparison of different sim-to-real techniques for 2D car detection on KITTI
5
Conclusion
We present SimNet, an efficient, multi-headed prediction network that leverages approximate stereo matching to transfer from simulation to reality. The network is trained entirely simulated data and robustly transfers to real images of unknown shhically-challeeginng objects such as glassware, even in direct sunlight. We show that these predictions sufficient for robot manipulation such t-shirt folding and grasping. In future work, we plan to use SimNet for enabling 3D scene understanding in homes.
8

---

References
[1] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, McKinley, and K. Goldberg. Learning ambidextrous rohot grasping policies. Science Robotics, 4(26), 2019.
[2] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez. and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained synthetic depth data. 2020 IEÈE International Conference on Robotics and Automation (ICRA), pages 941 1-9418.. IEEE, 2020.
[3] S. Sajjan, M. Moore, M. Pan, G. Nagaraja, J. Lee. A. Zeng, and S. Song. Clear grasp: 3d shape estimation of transparent objects for manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3634-3642. IRFF, 2020.
[4] C. Xie, Y. Xiang, Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth unseen object instance segmentation. In Conference on robot learning, pages 337--137. PMLR, 2020.
[5] Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abheel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 TEEE/RSJ international conference on intelligent robots and systems (IROS). pages 23- 23-30. IEEE, 2017.
[6] M. Danielezuk, M. Matl, S. Gupta, A. Li, A. Lec, J. Mahler, and K. Goldberg. Segmenting unknown 3d objects from real depth images using mask r-enn trained on synthetic data. In 2019 International Conference on Robotics and Automation (ICRA). pages 7283-7290. IEEE, 2019.
[7] I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint UrXiv:1910.07111 2019.
[8] A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez. N. Jamali, al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. arXiv preprint arXiv:2003. 12698, 2020.
[9] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Visuospatial foresight for multi-step, multi-task fabric manipulation. arXiv preprint arXiv:2003.09044, 2020.
[10] D. Scita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyam, Ichnowski, N. Jamali, al. Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor. arXiv preprint 1xXiv:1910004854 2019.
[11] J. Grannen, P. Sundaresan, B. Thananjeyan, J. Ichnowski, A. Balakrishna, M. Hwang. V. Viswanath, M. Laskey, J. E. Gonzalez, and K. Goldberg. Untangling dense knots by learning relevatl keypoints. arXiv preprint arXiv:201 2020.
[12] D. Seita, N. Jamali, M. Laskey, A. K. Tanwani. R. Berenstein, P. Baskaran, S. Iba, Canny, and K. Goldberg. Deep transfer learning of pick points on fabric for robot bed-making, arXiv preprint arXiv: 180.0098000000. 2018.
[13] P. Sundaresan, B. Thananjeyan, J. Chiu, D. Fer, and K. Goldberg. Automated extraction of surgical needles from tissue phantoms. In 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), pages 170-177. 177. IEEE, 2019.
[14] D. A. Forsyth and J. Ponce. Computer vision: modern approach. Pearson,, 2012.
[15] N. Mayer, Ilg, P. Hausser, Fischer, D. Cremers, A. Dosovitskiy, and Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1040-4048 2016.
[16] A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE International Conference on Computer Vision, pages 66-75. 2017.
[17] F. Sadeghi and S. Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint 1oXiv:1611.042000 2016.
[18] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. arXiv preprint 1rXiv:1807.07044, 2018

---

[19] D. Kalashnikov. A. Irpan. P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen. E. Holly, M. Kalakrishnan. V. Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint orXiv:/806 10293, 2018
[20] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent ohjects. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11602-11610, 2020.
[21] M. Durner, W. Boerdijk, M. Sundermeyer, W. Friedl, Z.-C. Marton, and R. Triebel. Unknown object 2021.
segmentation from stereo images. arXiv preprint
[22] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference Computer Vision and Pattern Recognition (CVPR), 2012.
[23] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812::::1110000033 2018.
[24] B. Thananjeyan, A. Balakrishna. S. Nair, M. Luo, K. Srinivasan, M. Hwang. J. E. Gonzalez. J. Ibarz, C. Finn, and Goldberg, Recovery rl: Safe reinforeement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-4922, 2021.
[25] S. Levine, C. Finn, T. Darrell, and P. Abbecl. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1))1334-1373, 2016
[26] R. Balasubramanian, L. Xu, P. D. Brook, J. R. Smith, and Y. Matsuoka. Physical human interactive guidance: Identifying grasping principles from human human-plauned grasps. IEEE Transactions on Robotics, 28 (4):899-91) 910, 2012.
[27] J. Maitin- M. Cusumano-Towwe Towner, J. Lei, and P. Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to rohotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pages 2308 2315. IEEE, 2010.
[28] L. Manuelli, Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. arXiv preprint ar 1503.0990006688 2019.
[29] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv:/866.000 888800000666 2018.
[30] A. Grunnet-Jepsen, J. N. Sweetser, P. Winer, A. Takagi, and J. Woodfill. Projectors for intel@) Tealsense depth cameras d4xx. Intel Support, Interl Corporation: Santa Clara, CA, USA, 2018.
[31] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil. T. To. E. Cameracci, S. Boochoon, and S. Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition Workshops, pages 969-977, 2018.
[32] R. Alghonaim and Johns. Benchmarking domain randomisation for visual sim-to-real transfer. arXiv preprint 2r 2020.
[33] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield. Deep object pose estimation for semantic robotic grasping of houschold objects. arXiv preprint 18Xiv: 10790, 2018.
[34] T. E. Lee, J. Tremblay, T. To, J. Cheng, T. Mosier, O. Kroemer, D. Fox, and S. Birchfield. Camera-to- toboot pose estimation from single image. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9426-9432. IEEE, 2020.
[35] 1. P. Howard. Perceiving in depth, volume 1: basic mechanisms. Oxford University Press, 2012.
[36] W. Luo, A. G. Schwing, and R. Urtasun. Efficient deep learning for stereo matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5695 5703, 2016
[37] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017.
[38] N. Mayer, E. 1lg, P. Fischer, C. Hazirbas, D. Cremers, A. Dosovitskiy, and T. Brox. What makes good synthetic training data for learning disparity and optical flow estimation? International Journal of Computer Vision, 126(9):942 260, 2018
10

---

[39] J. Zbontar and Y. LeCun. Computing the stereo matching cost with a convolutional neural network. In Proceedingy of the IEEE conference compuler vision and pattern recognition, pages 1592-1599, 2015.
[40] K. Shankar, M. Tjersland, J. Ma, K. Stone, and M. Bajracharya. A learned stereo depth system for robotic manipulation in homes. arXiv preprint arXiv:2109.1164: 2021.
[41] F. Yu, V. Koltun, and T. Funkhouser. Dilated residual networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472-480, 2017.
[42] P. J. Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492-518. Springer, 1992.
[43] A. Kirillov, R. Girshick, K. Hc, and P. Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6399-649999 6408, 2019.
[44] T. Hou, A. Ahmadyan, L. Zhang, J. Wei, and M. Grundmann. Mobilepose: Real-time pose estimation for unseen objects with weak shape supervision. arXiv preprint 20Xiv:2003.03333 2020.
[45] K. Duan, S. Bai, L. Xie. H. Qi. Q. Huang, and Q. Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 6569-6577 2019.
[46] M. Roberts and N. Paczan, Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. arXiv preprint 1.055, 2020.
[47] M. Matl. Pyrender.
pyrennder, 2013.
[48] M. Pharr, W. Jakoh, and G. Humphreys. Physically based rendering: From theory implementation. Morgan Kaufmann, 2016.
[49] T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Asahara, and K. Murase. Development of human support robot as research platform a domestic mobile manipulator. ROBOMECH journal, 6(1):1-15, 2019.
[50] J. Mahler, Liang S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries, arXiv preprint arXiv: 7703.09322, 2017.
[51] H. Wang, Sridhar, J. Huang, J. Valentin, Song, and I.. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2642-2651, 2019.
[52] A. Valada, R. Mohan, and W. Burgard. Self-supervised model adaptation multimodal semantic segmentation. International Journal of Computer Vision, (28(5):139) 1285, Jul 2019. ISSN 1573-1405. 1ot1000.07/111120 (atpss1223--1111 1119900188--0111 URL http: doi org/ 10. 1077/ 01188-y.
[53] S. loffe and C. Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[54] A. Geiger, P. Lenz, C. Stiller, and Urtasun. Vision meets robotics: The kitti datasct. Journal of Robotics Research (IJRR), 2013.
International
[55] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song,
H. Su, et al. Shapenet: An information-iic 3d model repository. arXiv preprint arXiv:1512 2015.


Reviews:
[{"review_id": "nhqG-gUCpNv", "reviewer": "Reviewer_o8ha", "summary": "This paper presents an architecture to predict object segmentation, oriented bounding boxes, keypoints and disparity from stereo RGB images provided as input. The training data is purely synthetic, generated in simulation via domain randomization. The trained system enables manipulation of objects in the real world, including transparent and reflective objects as well as non-rigid objects such as t-shirts. It can also achieve 2D car detection, as shown in experiments involving the KITTI benchmark.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Surprising performance in manipulation of transparent, reflective and non-rigid objects.\n\nWeaknesses:\n- A few elements can be better explained to increase the clarity of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work achieves remarkable results in manipulating challenging objects (transparent, reflective, non-rigid).\n\nThe release of their dataset is also a significant contribution.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "Figure 2: \"The output of the SCVN is a low resolution disparity image fed in with features from the left image to a ResNet-FPN backbone and output prediction heads\" --> can the authors please better explain what are the features from the left image? How are they obtained?\n\nCan the authors please explain the dimensionalities Wo/8 X Ho/8 X 16, Wo/8 X Ho/8 and Wo/8 X Ho/8 X 6? This is in subsection \"Oriented Boundin Boxes\". It's not clear to this reviewer why the oriented bounding boxes prediction leads to these shapes.\n\nTypos:\n\n- The second component and a core contribution SimNet (Figure 2) --> contribution of\n- However, the robot is only able grasp 35% of the “hard” objects with the RGB-D network --> able to grasp\n- we can learn robust “low-level” features like disparity than enable sim2real transfer of “high-level” vision tasks --> that enable\n- Each prediction head high uses the up-scaling branch defined in [40] --> What is meant here by \"high\"?\n- Since our SCVN, produces a disparity image at quarter resolution --> no comma between \"SCVN\" and \"produces\"\n"}}, {"review_id": "n504HTTuuH", "reviewer": "Reviewer_oTrx", "summary": "The paper presents a multi-task network consisting of ResNet(-FPN) blocks and a stereo matching module that takes in stereo images and outputs different predictions such as segmentation, 3D bounding boxes, keypoints, disparity, etc. The network is trained with data generated from a non-photorealistic simulator together with domain randomization. In the robotic experiments, a T-Shirt folding task based on keypoints and unknown object grasping based on the predicted bounding boxes is tested. Furthermore, 3D/2D object detection is evaluated on human household scenes (including transparent objects) and the KITTI benchmark. They show for their network that stereo inputs outperform mono, depth or RGB-D inputs in all cases and that sim2real transfer works well.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Several relevant robotic tasks are evaluated and the suitability of stereo data for these tasks is demonstrated\n- Sim2Real transfer from non-realistic simulated data is shown\n- Comparisons of input modalities using variations of their own network structure\n\nWeaknesses:\n\n- I did not learn very much. It is not a new finding that stereo can improve results when RGB-D sensors fail. Neither that disparity is a good auxiliary task for semantic predictions improving results over RGB. It is a nicely engineered system from existing components evaluated on robotic tasks. But the evaluation and discussion is lacking depth beyond \"look, our system works\".  \nKeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects, CVPR 2020  \nReal-Time Semantic Stereo Matching, ICRA 2020\n- There is no comparison to existing approaches even when comparing on public benchmarks, e.g. on the 2d car detection of KITTI. It does not seem that results are competitive to state-of-the-art approaches regardless of modalities   \nhttp://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d  \nStereo R-CNN based 3D Object Detection for Autonomous Driving, CVPR 2019  \n- The reader has no idea whether the modality comparisons hold in general or whether other RGB-D methods specifically tailored for fusing modalities would turn results around.\n- The argument for efficient non-realistic data generation becomes quite questionable if hyperparameter tuning takes \"20 single gpu instances for 48 hours\". There are easy ways to generate photorealistic annotated stereo data in much less time, e.g. BlenderProc", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I welcome system papers and I think the approach has potential, but the evaluation is lacking quite a bit. There are no comparisons to existing approaches, no real discussions of results and not much additional findings compared to literature. Therefore, it can merely serve as a proof-of-concept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- It would be nice to see the 3D predictions from a different angle, e.g. the 3d bounding boxes in a point cloud.\n- The supplement is over 50Mb, please reduce the size", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "P3OCUm2d1Xt", "reviewer": "Reviewer_z3Gb", "summary": "This paper proposes a perception model, SimNet, that leverages learned stereo matching to predict keypoints, oriented bounding boxes, and segmentation while only using low-quality simulation data during training. The authors perform extensive experiments in both indoor and outdoor environments and show that the proposed method can achieve better performance compared to monocular, depth-only, or RGB-D input.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n- The motivation is clear and sound.\n- The technical presentation is clear.\n- The experiments on the real world data of different domains show that the proposed method is versatile.\n\n**Weaknesses**: \n- To me, the major contribution of the paper is unclear. \n  - The authors didn’t explain why they had to design Stereo Cost Volume Networks (SCVN, Section 3.1). Is it a better learned stereo matcher compared to existing works listed in Section 2.3?  Can authors replace it with an existing work? If so, does the performance of other heads (e.g., keypoints, segmentation) change?\n  - How important is the disparity auxiliary loss (line 169-174)? The authors didn’t ablate SimNet to show its importance. In the supplementary material, Table 5 seems to indicate that incorporating this loss doesn’t improve other baselines much.\n  - The second contribution indicates “the first network to enable single-shot prediction of 3D oriented bounding boxes of unknown objects”. Are there two-stage approaches? If so, how do SimNet’s accuracy, runtime, and memory consumption compare to them?\n  - Line 71 “(iii) stereo vision experiments on simulated and real home scenes with annotations for oriented bounding boxes, segmentation masks, depth images, and keypoints” -> I don’t think running experiments on these known tasks itself is a contribution.\n\n- Experiments:\n  - Line 72 indicates that one of the contribution is “(iv) experiments on the KITTI stereo vision benchmark dataset that suggest that the network transfers significantly better than standard domain randomization and naive stereo concatenation” -> How does it compare to other state-of-the-arts (SOTAs) for 2D car detection that also report results on the same benchmark? Additionally, reporting results on other tasks will help readers understand the potentials and limitations of SimNet. As SimNet can perform segmentation and disparity prediction too, can authors show results on those tasks and compare to published methods on KITTI?\n  - As SimNet is a multi-task framework, do these tasks help each other? I think the authors can ablate the tasks used to train SimNet to inform readers whether multi-task learning helps or hurts the performance in this case.\n   - Line 108 the authors mentioned that the approach is comparable to Xie et al. [6], can you show the compared performance on segmentation?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend rejecting the paper because 1) the ambiguous contribution of the paper and 2) the lack of experiments (see my comments above). I think authors need to perform a series of ablation studies to justify the design of the pipeline. Additionally, comparing SimNet to published works on famous benchmarks (e.g., KITTI) for tasks such as segmentation and disparity is important for readers to understand how mature SimNet is. Lastly, an ablation study on the number of downstream tasks to train on is important for readers to understand how it affects the peroformance.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "**Major**:\n- Show results to justify why designing SCVN is necessary. For example, does it perform better than published learned stereo matchers for disparity prediction? Does it lead to better downstream tasks performances compared to published learned stereo matchers?\n- Show ablation study of the disparity auxiliary loss (line 169-174).\n- If there are two-stage approaches that predict “3D oriented bounding boxes of unknown objects” compare SimNet’s accuracy, runtime, and memory consumption to them.\n- Show disparity prediction and segmentation results on KITTI. Compare the results to other published learning-based methods for readers to understand how does SimNet compared to others.\n- Study whether the multi-task nature of SimNet helps/hurts the performance.\n- Compare the segmentation results to Xie et al. [6].\n\n**Minor**:\n- Line 264, Figure 6 only exists in the supplementary material but authors didn’t mention it.\n- Line 247, “different, real homes” -> “different real homes”.\n- In Figure 4 & 5, both the depth and the disparity are too small for readers to scrutinize. I think showing less examples while explicitly pointing out why SimNet is better would help readers a lot.\n"}}, {"review_id": "O0qkaBDNPI", "reviewer": "Reviewer_m12X", "summary": "This paper proposes an approach for sim2real transfer that can leverage suboptimal simulation data for downstream robotic manipulation tasks on unknown objects. The approach is based on a deep neural network that is trained on jointly optimizing a variety of perception-based losses from stereo images. To reduce the simulation gap, the model is trained on predicting an auxiliary disparsity map between its input images, which is then used as additional information for the later predictions. This incentivizes the model to focus on geometric reasoning rather than texture or depth. Experimentally, the authors show that this geometric reasoning improves downstream performance on object detection and manipulation tasks when compared to similar models that are trained without it.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The main idea of training an auxiliary disparsity map from stereo inputs is a conceptually simple but effective approach to utilize geometric information in sim2real transfer.\n- This allows for good performance on downstream real-world tasks even when the model is trained on sub-optimal simulations, thus mitigating the need for slow and expensive hyper-realistic simulations.\n- The approach is applied to different real-world tasks, showcasing its versatility.\n- Both the related work and the model description are presented clearly. The figures and schematics are very fitting. The model is concisely summarized in the schematic in Figure 2, and Figure 4 shows a conclusive quantitative example of the advantages of the approach.\n\n### Weaknesses\n\n- The model depends on a plethora of different training objectives. As such, it needs to tune a weight for each of those, making it less applicable in novel scenarios. Adding up a number of losses like this also comes with the risk of the network prioritizing some losses and ignoring others, thus potentially being less reliable if not carefully tuned.\n- On the same note, the impact of the different training losses for the final performance is not really explored beyond the auxiliary loss. An additional ablation study on these losses would be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents and explores the interesting idea of incentivizing geometric reasoning in a neural architecture by means of an auxiliary disparsity loss to improve sim2real transfer. The individual parts of the rather complex model are both intuitive and motivated nicely.\n\nThe approach is evaluated on different real-world scenarios and shown to achieve good performance. However, the experiments could be more convincing as they lack uncertainty measures (such as standard deviations of the performance over multiple trials) and some insightful ablations. It can be argued that the baselines that the method is compared to could have been stronger.\n\n\n========================================\n\n(Edited and copied from the reviewer response below) \nThe authors have resolved many of my concerns, while indicating that they intend to address most of the remaining ones if accepted. I can not evaluate intended changes, but still believe that the paper in its current state poses an interesting contribution. I will leave my score at `weak accept'.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- The baseline in Section 4.1.2. is chosen by \"sweeping over four different algorithms\", which are not explained further in the main paper.\n- The experimental results are presented as plain numbers without standard deviations, confidence intervals or other measures of uncertainty. This makes it hard to assess the stability of the approach as well as the qualitative difference between the approach and its baselines.\n- Section 4.4 reports mAP@ 0.5, while previous sections report mAP@ 0.25. This choice is not further explained/justified in the paper. Either making this consistent or explaining the reason for this change would help the experimental section.\n- The appendix mentions that the weights for the losses for the different models are chosen based on HyperBand. This should be part of the main paper. It should also be clarified how the other hyperparameters were chosen.\n- Contributions **(iii)** to **(v)** each refer to one experimental setup and the insights taken from it. This is more of a description and evaluation rather than a contribution. The insights (better transfer than baselines, useful for downstream policies) could instead be added mentioned in contribution **(i).**\n- The RGB-D baseline in Section 4.1.2 is chosen based on the performance for \"easy\" objects. Later on, this baseline performs bad on the \"hard\" objects, which could be an artefact of architecturally overfitting on the \"easy\" objects. The bad baseline performance (or rather the comparison of the model's performance to it) is a main selling point of the paper and highlighted in the abstract. As such, the baseline should be convincingly designed to be as strong as possible.\n- The car detection (Section 4.4) uses a \"Stacked\" baseline consisting of stereo images that is missing in previous sections. Considering the good performance of the \"Mono\" baseline in the previous section, it would be interesting to see how \"Stacked\" compares to the main model on the other tasks.\n- Some heads of the model use an L1 loss, others use a Huber loss. It is not clear why this distinction is made and why which loss is chosen.\n\n### Minor Issues\n\n- Table 1 shows the Perception performance of the model compared to baselines for \"easy\" objects. It would be interesting to also see the performance for \"hard\" objects.\n- The name *SimNet* is already used by NVIDIA for a simulation toolkit\n- The appendix is very bloated due to the figures having high resolution. The file itself is rather big, and some page takes a second to load\n- The model is presented as an end-to-end method in the appendix and the introduction. While one can argue that is is trained in an end-to-end fashion, this is different from end-to-end methods for e.g., robot policy learning. This distinction is briefly mentioned in the related work section, but not really cleared up.\n- The model feeds early features from the left image to its main backbone, but does not do so for the right image. This is presumably done because the left image is the basis for the output heads, but that is not explicitly mentioned in the paper.\n\n### Typos and Style\n\n- The model concept is explicitly described/explained in the abstract and each of the early sections. This feels somewhat repetitive and could be alleviated by shorter or more targeted descriptions.\n- Ll. 9, 10: The appendix mentions \"disparsity\" twice, once for the final output and once for the auxiliary loss. This distinction is not clear.\n- Ll. 29, 41: References to figures that are very far away from the current text\n- Ll. 35: \"artefacts\" → \"artifacts\"\n- Ll. 69: Contribution (ii) explicitly mentions single-shot prediction, but the rest of the paper mostly does so implicitly. If this is a selling point, it should maybe be more prominent in the paper.\n- Ll. 142: The double-subscript Notation for the images is a little cluttered. Maybe the image position (*l, r*) could instead be indicated via a superscript.\n- Ll. 158: Consecutive sentences start and end with an equation, which makes this hard to read.\n- Ll. 201: EPnP is used as an acronym but never written out.\n- Ll. 210: SVD is written in bold while other methods and/or acronyms are not.\n- Ll. 263: \"optically easy\" and \"optically hard\" should be made consistent with other parts of the paper, where the distinction is simply made between \"easy\" and \"hard\" objects.\n- Ll. 298: The reasoning \"since the ability...\" is stated as a fact, and should be supported by citations or other evidence.\n- Ll. 302: \"am\" → \"an\""}}]

Author rebuttal:

Meta-review: On one side, the reviewers agree that the experimental results presented in the paper are notable, giving good evidence that the proposed approach based on the SimNet architecture is useful and effective. On the other side, there are different perspectives regarding the scientific contribution of the work. In any case, a clearer distinction of the proposed algorithm to existing work (e.g. Xie et al [6], Liu et al (see reviewer oTrx)) and a more thorough motivation for the design of this new method are necessary to assess the contribution. What is it that the SimNet architecture can do that can not be done by similar ones? Also, the ablation studies mentioned by reviewer z3Gb are an important add-on to measure the impact of the approach. Auxiliary tasks are used more and more often, and intuitively, it is reasonable to use them, but an empirical evidence of their benefit is still required. It would be good if the authors could react on these two main points in their answer to gain more insight about the main contributions of their work.   

Post-rebuttal:
During the discussion phase, the authors could clarify most of the concerns raised by the reviewers. The additional results and the added ablation studies are very useful, showing the usefulness and significance of the approach. Also, the further distiction from existing works is convincing.  
Decision: Accept (Poster)