Seeing Glass: Joint Point-Cloud and Depth Completion for Transparent Objects
Authors: Haoping Xu
Yi Ru Wang
Sagi Eppel
Alan Aspuru-Guzik
Florian Shkurti
Animesh Garg
Venue: CoRL2021 Oral (2021)
Keywords: Transparent Objects
Depth Completion
3D Perception
Data Collection

Abstract: The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literature, we also propose an automated dataset creation workflow that consists of robot-controlled image collection and vision-based automatic annotation. Through this automated workflow, we created Transparent Object Depth Dataset (TODD), which consists of nearly 15000 RGB-D images. Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets, including ClearGrasp, and that it also handles cluttered scenes when trained on TODD. Code and dataset will be released at https://www.pair.toronto.edu/TranspareNet/


Reviews:
[{"review_id": "i1wFqybe3z", "reviewer": "Reviewer_vich", "summary": "The paper presents a novel method of monocular depth completion for transparent objects. It introduces a novel architecture to be able to correct and fill in the gaps made by a typical RGB-D sensor. In order to achieve that they first mask the depth input and transform it to a point cloud, which is then feed to a 3D-CNN architecture. After that the result is projected back into the depth image and handed with the RGB image to a U-Net architecture to combine the features of both to create a complete depth reconstruction of the transparent object. Additionally, they provide a novel dataset called Transparent Object Depth Dataset (TODD), which contains 14,659 RGB-D images of different backgrounds, which was created via an automatized process, which makes it easier to create more data points in the future.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* a novel architecture combining a point cloud filling approach with an image based depth completion method to tackle the difficult task of transparent object depth completion\n* a new dataset, which will be helpful in the future evaluation of methods in this area\n* an automatized method to generate the said dataset \n* a nice in depth comparison to ClearGrasp [1]\n* well written and most of the core concepts are well explained\n\nWeaknesses:\n* The biggest weakness I see is that the authors fail to mention, where the mask in Figure 2 comes from. It is not predicted as the edges of the mask look to perfect, but for a real world application it seems weird that the input requires a mask of the transparent object, as this is likely to not be available. Of course a second method could have been trained on that, but that is not mentioned in the paper, which means that in the current version, this can not be used directly on a robot. \n* Sadly, the authors do not mention how the automated dataset construction affects the performance of the network. As the april tags are visible in all RGB images and the objects are always on seven fixed positions in relation to the April tags, one could ask themselves if the network is just really good in locating the april tags and then therefore predicting objects on these seven positions? I would have loved to seen, in an ablation study what happens if those april tags are removed from the scene, to ensure that the network didn't just learn the relative positions to the april tags. Especially, as the amount of training/tested objects is quite low, so that the network could have been able to recognize the category and the given april tag position, which is a much easier task then reconstructing a depth image. \n* Figure 2 is hard to read, the upper and lower part should be visually separated, maybe by putting them into boxes. Additionally, why are the input images rotated in the top left corner, that makes it so much harder to understand. Furthermore, is not intuitive that the output from the top right corner is used in the bottom left corner. That should be visualized better.\n* Not all figures are explained in the text, which is in my view a must. The paper contains six figures, but only 1, 2 and 6 are mentioned in the text. Figure 3, which is extremely crucial to understand how the objects are placed and 6D pose estimated is not explained in the text or even mentioned.  Figure 4 and 5 should also be mentioned.\n* Section 4.2 and 4.3 introduce a lot of variables, which are not all explained in the text. If they are the same as in [46, 47] then this should be mentioned. In particular in 4.3 $V^p, W^p$ are mentioned but not explained. $W^i$ is used before, but there it seems to be the width of the image. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I usually don't give weak responses to papers. But, I truly believe that this paper deserve a strong accept as the idea and the presentation is good. It just lacks a few points, which are sadly to important to be overlooked. The paper sadly omits the detail, where the mask comes from, which is used in Figure 2 as an input to the method. Without that mask the whole approach does not work and therefore an application in a real scenario is out of reach, as this conference highlights the importance of applicable robotic work. \n\nI hope the authors can clarify the usage of this mask as an input and how they plan to produce it in a real scenario, not just in the constructed dataset test. If they can show that it is also possible to train a mask detection network and use it as an input, with similar or even worse results, than I will change my review to a strong accept. \n\nAfter reading the perfect response from the authors, I change my recommendation to \"Strong Accept\". They were able to provide satisfactory results to all my questions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "* As mentioned above, it must be clear how the mask is produced and where it would come from in a real scenario\n* Figure 2 must be reorganized to make it easier for a reader to understand the approach\n* All figures **must** be explained in the text, this is not optional.\n* All mentioned variables **must** be explained or must be referenced if they are used in a similar fashion to a prior work. In particular in section 4.2 and 4.3"}}, {"review_id": "Oo2VYq9Jii3", "reviewer": "Reviewer_ieyv", "summary": "This paper contributes a interesting approach and data collection pipeline for what is essentially a shape completion problem (predicting unseen point clouds) that specializes in see-through objects. This is a class of objects that are common and pose a significant challenge to RGB-D sensors we use in robotics. I would say this is an important and relevant problem with a solid approach presented.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I am a fan of this paper, good work! The paper addresses an important challenge in perception for manipulation, so props for that. The proposed approach and data-collection pipelines are carefully thought out and well executed. The results do seem quite compelling and I am interested in trying it out. Part of the challenge I feel is the relative recency of the field so benchmarking on actively collected data-sets is still emergent. It is good to see the role that a robot can play in data-collection. I have 2 suggestions to improve the paper:\n\n1- Please provide a detailed discussion on why the work is relevant to manipulation. How good does our estimate of the complete point cloud of the object need to be for various tasks? For example, how well do we need to do shape completion for grasping? Is the existing approach overkill? This sort of discussion of will strengthen the ties of this paper to robotics more.\n\n2- Please provide an evaluation of the normals predicted by the approach -- these have value for the calculation on antipodal grasps.\n\nI also appreciate the acknowledgement of concurrent work as a footnote on the final page of the paper. I think the fact that it is mentioned is sufficient for the initial submission. If it is possible to compare against it for the final submission, that would be excellent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe that handling transparent objects is an important step towards general manipulation. I think that this paper is attempting to address this challenge in a convincing way. I do  think this paper should be accepted, it is cool and serves as a good foundation to build upon -- even though personally I don't think the novelty in the network or its losses are that high.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see strengths and weaknesses.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "5BK1yAYSnTk", "reviewer": "Reviewer_UETv", "summary": "The main contribution of this paper is a new large-scale real transparent object dataset including 15000 RGB-D observations that contain RGB, raw depths paired with ground truth depth, instance segmentation, and object pose. The dataset is collected via an automatic procedure using AprilTags, transparent object CAD model, and robot arms. It also introduces a new glass depth estimation method that outperforms the existing approach on both the collected datasets and the existing datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presented paper proposed to leverage the distorted depth of the transparent objects from the raw depth, instead of discarding them as did in previous work. The quantitative evaluation shows that this way can achieve higher accuracy.\n\nThis paper also presents a new real-world transparent object dataset that can be very useful for the community. It is clearly explained how the dataset was automatically collected using their robotic platform. \n\nOne concern regarding the presented dataset is that it only contains 6 objects and a similar background scene (plane). Looking at Figure 3 that shows the CAD model of the used transparent objects, it seems that it actually only contains two different shapes with 3 varying sizes. Considering there are two objects used in the validation and testing and 4 objects in the training, it is a bit worrying if the model is overfitted on the presented datasets. It will be better if the method can also be tested on the shapes in different shapes or on a different dataset to see if it can be generalised well. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presented a new method to estimate the depth of transparent objects from an RGB-D image pair. It outperforms the existing approaches. It also presented a new real-world RGB-D transparent object dataset, which will be very useful for the community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The main concerning issue is mentioned in the weakness part. \n\nBesides, the writing can be further improved. At least, the notations should be explained when they are used. For example, where does W^p, V^p in Equation (2) come from? It is unclear for me and the authors should not assume they are common notations. "}}]

Author rebuttal:

Meta-review: The paper presents an interesting and novel DL approach for an important problem, namely the detection of transparent objects from RGB-D sensors. While on one side, this novelty and the good results of the method are much appreciated by the reviewers, they also raise some important questions regarding the practical applicability of the approach and its tendency to overfit to the comparably small training data. It would be good if the authors could comment on the availabilty of the object masks and how this could be solved in practice, and also explain how the overfitting is addressed. For example, a qualitative evaluation on a data set without AprilTags and with different, random object positions could be considered to show the generalization ability of the approach. 

Post-rebuttal:
The discussion between the authors and the reviewers was very helpful. Especially the additional results, showing the usefulness of the approach even without a ground truth segmentation mask and without the AprilTags in the image, have led to a convincing contribution of the paper. The approach is very novel and a good inspiration for the community.
Decision: Accept (Oral)