Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper contributes a interesting approach and data collection pipeline for what is essentially a shape completion problem (predicting unseen point clouds) that specializes in see-through objects. This is a class of objects that are common and pose a significant challenge to RGB-D sensors we use in robotics. I would say this is an important and relevant problem with a solid approach presented.
strengths_and_weaknesses: Overall, I am a fan of this paper, good work! The paper addresses an important challenge in perception for manipulation, so props for that. The proposed approach and data-collection pipelines are carefully thought out and well executed. The results do seem quite compelling and I am interested in trying it out. Part of the challenge I feel is the relative recency of the field so benchmarking on actively collected data-sets is still emergent. It is good to see the role that a robot can play in data-collection. I have 2 suggestions to improve the paper:

1- Please provide a detailed discussion on why the work is relevant to manipulation. How good does our estimate of the complete point cloud of the object need to be for various tasks? For example, how well do we need to do shape completion for grasping? Is the existing approach overkill? This sort of discussion of will strengthen the ties of this paper to robotics more.

2- Please provide an evaluation of the normals predicted by the approach -- these have value for the calculation on antipodal grasps.

I also appreciate the acknowledgement of concurrent work as a footnote on the final page of the paper. I think the fact that it is mentioned is sufficient for the initial submission. If it is possible to compare against it for the final submission, that would be excellent.
summary_of_recommendation: I believe that handling transparent objects is an important step towards general manipulation. I think that this paper is attempting to address this challenge in a convincing way. I do  think this paper should be accepted, it is cool and serves as a good foundation to build upon -- even though personally I don't think the novelty in the network or its losses are that high.
issues: Please see strengths and weaknesses.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The main contribution of this paper is a new large-scale real transparent object dataset including 15000 RGB-D observations that contain RGB, raw depths paired with ground truth depth, instance segmentation, and object pose. The dataset is collected via an automatic procedure using AprilTags, transparent object CAD model, and robot arms. It also introduces a new glass depth estimation method that outperforms the existing approach on both the collected datasets and the existing datasets.
strengths_and_weaknesses: The presented paper proposed to leverage the distorted depth of the transparent objects from the raw depth, instead of discarding them as did in previous work. The quantitative evaluation shows that this way can achieve higher accuracy.

This paper also presents a new real-world transparent object dataset that can be very useful for the community. It is clearly explained how the dataset was automatically collected using their robotic platform. 

One concern regarding the presented dataset is that it only contains 6 objects and a similar background scene (plane). Looking at Figure 3 that shows the CAD model of the used transparent objects, it seems that it actually only contains two different shapes with 3 varying sizes. Considering there are two objects used in the validation and testing and 4 objects in the training, it is a bit worrying if the model is overfitted on the presented datasets. It will be better if the method can also be tested on the shapes in different shapes or on a different dataset to see if it can be generalised well. 


summary_of_recommendation: This paper presented a new method to estimate the depth of transparent objects from an RGB-D image pair. It outperforms the existing approaches. It also presented a new real-world RGB-D transparent object dataset, which will be very useful for the community.
issues: The main concerning issue is mentioned in the weakness part. 

Besides, the writing can be further improved. At least, the notations should be explained when they are used. For example, where does W^p, V^p in Equation (2) come from? It is unclear for me and the authors should not assume they are common notations.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a novel method of monocular depth completion for transparent objects. It introduces a novel architecture to be able to correct and fill in the gaps made by a typical RGB-D sensor. In order to achieve that they first mask the depth input and transform it to a point cloud, which is then feed to a 3D-CNN architecture. After that the result is projected back into the depth image and handed with the RGB image to a U-Net architecture to combine the features of both to create a complete depth reconstruction of the transparent object. Additionally, they provide a novel dataset called Transparent Object Depth Dataset (TODD), which contains 14,659 RGB-D images of different backgrounds, which was created via an automatized process, which makes it easier to create more data points in the future.
strengths_and_weaknesses: Strengths:
* a novel architecture combining a point cloud filling approach with an image based depth completion method to tackle the difficult task of transparent object depth completion
* a new dataset, which will be helpful in the future evaluation of methods in this area
* an automatized method to generate the said dataset 
* a nice in depth comparison to ClearGrasp [1]
* well written and most of the core concepts are well explained

Weaknesses:
* The biggest weakness I see is that the authors fail to mention, where the mask in Figure 2 comes from. It is not predicted as the edges of the mask look to perfect, but for a real world application it seems weird that the input requires a mask of the transparent object, as this is likely to not be available. Of course a second method could have been trained on that, but that is not mentioned in the paper, which means that in the current version, this can not be used directly on a robot. 
* Sadly, the authors do not mention how the automated dataset construction affects the performance of the network. As the april tags are visible in all RGB images and the objects are always on seven fixed positions in relation to the April tags, one could ask themselves if the network is just really good in locating the april tags and then therefore predicting objects on these seven positions? I would have loved to seen, in an ablation study what happens if those april tags are removed from the scene, to ensure that the network didn't just learn the relative positions to the april tags. Especially, as the amount of training/tested objects is quite low, so that the network could have been able to recognize the category and the given april tag position, which is a much easier task then reconstructing a depth image. 
* Figure 2 is hard to read, the upper and lower part should be visually separated, maybe by putting them into boxes. Additionally, why are the input images rotated in the top left corner, that makes it so much harder to understand. Furthermore, is not intuitive that the output from the top right corner is used in the bottom left corner. That should be visualized better.
* Not all figures are explained in the text, which is in my view a must. The paper contains six figures, but only 1, 2 and 6 are mentioned in the text. Figure 3, which is extremely crucial to understand how the objects are placed and 6D pose estimated is not explained in the text or even mentioned.  Figure 4 and 5 should also be mentioned.
* Section 4.2 and 4.3 introduce a lot of variables, which are not all explained in the text. If they are the same as in [46, 47] then this should be mentioned. In particular in 4.3 $V^p, W^p$ are mentioned but not explained. $W^i$ is used before, but there it seems to be the width of the image. 
summary_of_recommendation: I usually don't give weak responses to papers. But, I truly believe that this paper deserve a strong accept as the idea and the presentation is good. It just lacks a few points, which are sadly to important to be overlooked. The paper sadly omits the detail, where the mask comes from, which is used in Figure 2 as an input to the method. Without that mask the whole approach does not work and therefore an application in a real scenario is out of reach, as this conference highlights the importance of applicable robotic work. 

I hope the authors can clarify the usage of this mask as an input and how they plan to produce it in a real scenario, not just in the constructed dataset test. If they can show that it is also possible to train a mask detection network and use it as an input, with similar or even worse results, than I will change my review to a strong accept. 

After reading the perfect response from the authors, I change my recommendation to "Strong Accept". They were able to provide satisfactory results to all my questions.
issues: * As mentioned above, it must be clear how the mask is produced and where it would come from in a real scenario
* Figure 2 must be reorganized to make it easier for a reader to understand the approach
* All figures **must** be explained in the text, this is not optional.
* All mentioned variables **must** be explained or must be referenced if they are used in a similar fashion to a prior work. In particular in section 4.2 and 4.3
review_title: null

Meta-review: The paper presents an interesting and novel DL approach for an important problem, namely the detection of transparent objects from RGB-D sensors. While on one side, this novelty and the good results of the method are much appreciated by the reviewers, they also raise some important questions regarding the practical applicability of the approach and its tendency to overfit to the comparably small training data. It would be good if the authors could comment on the availabilty of the object masks and how this could be solved in practice, and also explain how the overfitting is addressed. For example, a qualitative evaluation on a data set without AprilTags and with different, random object positions could be considered to show the generalization ability of the approach. 

Post-rebuttal:
The discussion between the authors and the reviewers was very helpful. Especially the additional results, showing the usefulness of the approach even without a ground truth segmentation mask and without the AprilTags in the image, have led to a convincing contribution of the paper. The approach is very novel and a good inspiration for the community.
Decision: Accept (Oral)