Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a perception model, SimNet, that leverages learned stereo matching to predict keypoints, oriented bounding boxes, and segmentation while only using low-quality simulation data during training. The authors perform extensive experiments in both indoor and outdoor environments and show that the proposed method can achieve better performance compared to monocular, depth-only, or RGB-D input.

strengths_and_weaknesses: **Strengths**: 
- The motivation is clear and sound.
- The technical presentation is clear.
- The experiments on the real world data of different domains show that the proposed method is versatile.

**Weaknesses**: 
- To me, the major contribution of the paper is unclear. 
  - The authors didn’t explain why they had to design Stereo Cost Volume Networks (SCVN, Section 3.1). Is it a better learned stereo matcher compared to existing works listed in Section 2.3?  Can authors replace it with an existing work? If so, does the performance of other heads (e.g., keypoints, segmentation) change?
  - How important is the disparity auxiliary loss (line 169-174)? The authors didn’t ablate SimNet to show its importance. In the supplementary material, Table 5 seems to indicate that incorporating this loss doesn’t improve other baselines much.
  - The second contribution indicates “the first network to enable single-shot prediction of 3D oriented bounding boxes of unknown objects”. Are there two-stage approaches? If so, how do SimNet’s accuracy, runtime, and memory consumption compare to them?
  - Line 71 “(iii) stereo vision experiments on simulated and real home scenes with annotations for oriented bounding boxes, segmentation masks, depth images, and keypoints” -> I don’t think running experiments on these known tasks itself is a contribution.

- Experiments:
  - Line 72 indicates that one of the contribution is “(iv) experiments on the KITTI stereo vision benchmark dataset that suggest that the network transfers significantly better than standard domain randomization and naive stereo concatenation” -> How does it compare to other state-of-the-arts (SOTAs) for 2D car detection that also report results on the same benchmark? Additionally, reporting results on other tasks will help readers understand the potentials and limitations of SimNet. As SimNet can perform segmentation and disparity prediction too, can authors show results on those tasks and compare to published methods on KITTI?
  - As SimNet is a multi-task framework, do these tasks help each other? I think the authors can ablate the tasks used to train SimNet to inform readers whether multi-task learning helps or hurts the performance in this case.
   - Line 108 the authors mentioned that the approach is comparable to Xie et al. [6], can you show the compared performance on segmentation?

summary_of_recommendation: I recommend rejecting the paper because 1) the ambiguous contribution of the paper and 2) the lack of experiments (see my comments above). I think authors need to perform a series of ablation studies to justify the design of the pipeline. Additionally, comparing SimNet to published works on famous benchmarks (e.g., KITTI) for tasks such as segmentation and disparity is important for readers to understand how mature SimNet is. Lastly, an ablation study on the number of downstream tasks to train on is important for readers to understand how it affects the peroformance.

issues: **Major**:
- Show results to justify why designing SCVN is necessary. For example, does it perform better than published learned stereo matchers for disparity prediction? Does it lead to better downstream tasks performances compared to published learned stereo matchers?
- Show ablation study of the disparity auxiliary loss (line 169-174).
- If there are two-stage approaches that predict “3D oriented bounding boxes of unknown objects” compare SimNet’s accuracy, runtime, and memory consumption to them.
- Show disparity prediction and segmentation results on KITTI. Compare the results to other published learning-based methods for readers to understand how does SimNet compared to others.
- Study whether the multi-task nature of SimNet helps/hurts the performance.
- Compare the segmentation results to Xie et al. [6].

**Minor**:
- Line 264, Figure 6 only exists in the supplementary material but authors didn’t mention it.
- Line 247, “different, real homes” -> “different real homes”.
- In Figure 4 & 5, both the depth and the disparity are too small for readers to scrutinize. I think showing less examples while explicitly pointing out why SimNet is better would help readers a lot.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents an architecture to predict object segmentation, oriented bounding boxes, keypoints and disparity from stereo RGB images provided as input. The training data is purely synthetic, generated in simulation via domain randomization. The trained system enables manipulation of objects in the real world, including transparent and reflective objects as well as non-rigid objects such as t-shirts. It can also achieve 2D car detection, as shown in experiments involving the KITTI benchmark.
strengths_and_weaknesses: Strengths:
- Surprising performance in manipulation of transparent, reflective and non-rigid objects.

Weaknesses:
- A few elements can be better explained to increase the clarity of the paper.
summary_of_recommendation: This work achieves remarkable results in manipulating challenging objects (transparent, reflective, non-rigid).

The release of their dataset is also a significant contribution.
issues: Figure 2: "The output of the SCVN is a low resolution disparity image fed in with features from the left image to a ResNet-FPN backbone and output prediction heads" --> can the authors please better explain what are the features from the left image? How are they obtained?

Can the authors please explain the dimensionalities Wo/8 X Ho/8 X 16, Wo/8 X Ho/8 and Wo/8 X Ho/8 X 6? This is in subsection "Oriented Boundin Boxes". It's not clear to this reviewer why the oriented bounding boxes prediction leads to these shapes.

Typos:

- The second component and a core contribution SimNet (Figure 2) --> contribution of
- However, the robot is only able grasp 35% of the “hard” objects with the RGB-D network --> able to grasp
- we can learn robust “low-level” features like disparity than enable sim2real transfer of “high-level” vision tasks --> that enable
- Each prediction head high uses the up-scaling branch defined in [40] --> What is meant here by "high"?
- Since our SCVN, produces a disparity image at quarter resolution --> no comma between "SCVN" and "produces"
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a multi-task network consisting of ResNet(-FPN) blocks and a stereo matching module that takes in stereo images and outputs different predictions such as segmentation, 3D bounding boxes, keypoints, disparity, etc. The network is trained with data generated from a non-photorealistic simulator together with domain randomization. In the robotic experiments, a T-Shirt folding task based on keypoints and unknown object grasping based on the predicted bounding boxes is tested. Furthermore, 3D/2D object detection is evaluated on human household scenes (including transparent objects) and the KITTI benchmark. They show for their network that stereo inputs outperform mono, depth or RGB-D inputs in all cases and that sim2real transfer works well.
strengths_and_weaknesses: Strengths:

- Several relevant robotic tasks are evaluated and the suitability of stereo data for these tasks is demonstrated
- Sim2Real transfer from non-realistic simulated data is shown
- Comparisons of input modalities using variations of their own network structure

Weaknesses:

- I did not learn very much. It is not a new finding that stereo can improve results when RGB-D sensors fail. Neither that disparity is a good auxiliary task for semantic predictions improving results over RGB. It is a nicely engineered system from existing components evaluated on robotic tasks. But the evaluation and discussion is lacking depth beyond "look, our system works".  
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects, CVPR 2020  
Real-Time Semantic Stereo Matching, ICRA 2020
- There is no comparison to existing approaches even when comparing on public benchmarks, e.g. on the 2d car detection of KITTI. It does not seem that results are competitive to state-of-the-art approaches regardless of modalities   
http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d  
Stereo R-CNN based 3D Object Detection for Autonomous Driving, CVPR 2019  
- The reader has no idea whether the modality comparisons hold in general or whether other RGB-D methods specifically tailored for fusing modalities would turn results around.
- The argument for efficient non-realistic data generation becomes quite questionable if hyperparameter tuning takes "20 single gpu instances for 48 hours". There are easy ways to generate photorealistic annotated stereo data in much less time, e.g. BlenderProc
summary_of_recommendation: I welcome system papers and I think the approach has potential, but the evaluation is lacking quite a bit. There are no comparisons to existing approaches, no real discussions of results and not much additional findings compared to literature. Therefore, it can merely serve as a proof-of-concept.
issues: - It would be nice to see the 3D predictions from a different angle, e.g. the 3d bounding boxes in a point cloud.
- The supplement is over 50Mb, please reduce the size
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an approach for sim2real transfer that can leverage suboptimal simulation data for downstream robotic manipulation tasks on unknown objects. The approach is based on a deep neural network that is trained on jointly optimizing a variety of perception-based losses from stereo images. To reduce the simulation gap, the model is trained on predicting an auxiliary disparsity map between its input images, which is then used as additional information for the later predictions. This incentivizes the model to focus on geometric reasoning rather than texture or depth. Experimentally, the authors show that this geometric reasoning improves downstream performance on object detection and manipulation tasks when compared to similar models that are trained without it.
strengths_and_weaknesses: ### Strengths

- The main idea of training an auxiliary disparsity map from stereo inputs is a conceptually simple but effective approach to utilize geometric information in sim2real transfer.
- This allows for good performance on downstream real-world tasks even when the model is trained on sub-optimal simulations, thus mitigating the need for slow and expensive hyper-realistic simulations.
- The approach is applied to different real-world tasks, showcasing its versatility.
- Both the related work and the model description are presented clearly. The figures and schematics are very fitting. The model is concisely summarized in the schematic in Figure 2, and Figure 4 shows a conclusive quantitative example of the advantages of the approach.

### Weaknesses

- The model depends on a plethora of different training objectives. As such, it needs to tune a weight for each of those, making it less applicable in novel scenarios. Adding up a number of losses like this also comes with the risk of the network prioritizing some losses and ignoring others, thus potentially being less reliable if not carefully tuned.
- On the same note, the impact of the different training losses for the final performance is not really explored beyond the auxiliary loss. An additional ablation study on these losses would be useful.
summary_of_recommendation: The paper presents and explores the interesting idea of incentivizing geometric reasoning in a neural architecture by means of an auxiliary disparsity loss to improve sim2real transfer. The individual parts of the rather complex model are both intuitive and motivated nicely.

The approach is evaluated on different real-world scenarios and shown to achieve good performance. However, the experiments could be more convincing as they lack uncertainty measures (such as standard deviations of the performance over multiple trials) and some insightful ablations. It can be argued that the baselines that the method is compared to could have been stronger.


========================================

(Edited and copied from the reviewer response below) 
The authors have resolved many of my concerns, while indicating that they intend to address most of the remaining ones if accepted. I can not evaluate intended changes, but still believe that the paper in its current state poses an interesting contribution. I will leave my score at `weak accept'.
issues: - The baseline in Section 4.1.2. is chosen by "sweeping over four different algorithms", which are not explained further in the main paper.
- The experimental results are presented as plain numbers without standard deviations, confidence intervals or other measures of uncertainty. This makes it hard to assess the stability of the approach as well as the qualitative difference between the approach and its baselines.
- Section 4.4 reports mAP@ 0.5, while previous sections report mAP@ 0.25. This choice is not further explained/justified in the paper. Either making this consistent or explaining the reason for this change would help the experimental section.
- The appendix mentions that the weights for the losses for the different models are chosen based on HyperBand. This should be part of the main paper. It should also be clarified how the other hyperparameters were chosen.
- Contributions **(iii)** to **(v)** each refer to one experimental setup and the insights taken from it. This is more of a description and evaluation rather than a contribution. The insights (better transfer than baselines, useful for downstream policies) could instead be added mentioned in contribution **(i).**
- The RGB-D baseline in Section 4.1.2 is chosen based on the performance for "easy" objects. Later on, this baseline performs bad on the "hard" objects, which could be an artefact of architecturally overfitting on the "easy" objects. The bad baseline performance (or rather the comparison of the model's performance to it) is a main selling point of the paper and highlighted in the abstract. As such, the baseline should be convincingly designed to be as strong as possible.
- The car detection (Section 4.4) uses a "Stacked" baseline consisting of stereo images that is missing in previous sections. Considering the good performance of the "Mono" baseline in the previous section, it would be interesting to see how "Stacked" compares to the main model on the other tasks.
- Some heads of the model use an L1 loss, others use a Huber loss. It is not clear why this distinction is made and why which loss is chosen.

### Minor Issues

- Table 1 shows the Perception performance of the model compared to baselines for "easy" objects. It would be interesting to also see the performance for "hard" objects.
- The name *SimNet* is already used by NVIDIA for a simulation toolkit
- The appendix is very bloated due to the figures having high resolution. The file itself is rather big, and some page takes a second to load
- The model is presented as an end-to-end method in the appendix and the introduction. While one can argue that is is trained in an end-to-end fashion, this is different from end-to-end methods for e.g., robot policy learning. This distinction is briefly mentioned in the related work section, but not really cleared up.
- The model feeds early features from the left image to its main backbone, but does not do so for the right image. This is presumably done because the left image is the basis for the output heads, but that is not explicitly mentioned in the paper.

### Typos and Style

- The model concept is explicitly described/explained in the abstract and each of the early sections. This feels somewhat repetitive and could be alleviated by shorter or more targeted descriptions.
- Ll. 9, 10: The appendix mentions "disparsity" twice, once for the final output and once for the auxiliary loss. This distinction is not clear.
- Ll. 29, 41: References to figures that are very far away from the current text
- Ll. 35: "artefacts" → "artifacts"
- Ll. 69: Contribution (ii) explicitly mentions single-shot prediction, but the rest of the paper mostly does so implicitly. If this is a selling point, it should maybe be more prominent in the paper.
- Ll. 142: The double-subscript Notation for the images is a little cluttered. Maybe the image position (*l, r*) could instead be indicated via a superscript.
- Ll. 158: Consecutive sentences start and end with an equation, which makes this hard to read.
- Ll. 201: EPnP is used as an acronym but never written out.
- Ll. 210: SVD is written in bold while other methods and/or acronyms are not.
- Ll. 263: "optically easy" and "optically hard" should be made consistent with other parts of the paper, where the distinction is simply made between "easy" and "hard" objects.
- Ll. 298: The reasoning "since the ability..." is stated as a fact, and should be supported by citations or other evidence.
- Ll. 302: "am" → "an"
review_title: null

Meta-review: On one side, the reviewers agree that the experimental results presented in the paper are notable, giving good evidence that the proposed approach based on the SimNet architecture is useful and effective. On the other side, there are different perspectives regarding the scientific contribution of the work. In any case, a clearer distinction of the proposed algorithm to existing work (e.g. Xie et al [6], Liu et al (see reviewer oTrx)) and a more thorough motivation for the design of this new method are necessary to assess the contribution. What is it that the SimNet architecture can do that can not be done by similar ones? Also, the ablation studies mentioned by reviewer z3Gb are an important add-on to measure the impact of the approach. Auxiliary tasks are used more and more often, and intuitively, it is reasonable to use them, but an empirical evidence of their benefit is still required. It would be good if the authors could react on these two main points in their answer to gain more insight about the main contributions of their work.   

Post-rebuttal:
During the discussion phase, the authors could clarify most of the concerns raised by the reviewers. The additional results and the added ablation studies are very useful, showing the usefulness and significance of the approach. Also, the further distiction from existing works is convincing.  
Decision: Accept (Poster)