Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors want to learn object shape from sensors on the robot. As a starting point, they apply a baseline method that uses solely RGBD data, using the prior work of Plausible Shape Sampling Network (PSSNet). This prior work first converts the RGBD images into voxel grids that are known-free and known-occupied. The PSSNet is then a VAE that encodes + decodes into a latent space. Latent vectors $\psi$ can then be sampled, decoded into occupancy probability estimates, and aggregated into object shapes.

This work aims to augment PSSNet with robot contact information. The initial belief of objects is represented as a particle filter of several samplings $\psi$ from the prior. As the robot moves in the environment, the contact sensors lead to contact constraints (either from free space or detected contacts). Given an already-trained PSSNet, the proposed method (CLASP) optimizes over $\psi$ to get new latents that are consistent with the contact constraints. The loss includes a term that encourages $\psi$ to stay close to the prior. Thus over the course of interaction, the potential object shapes become more constrained. This is then compared against other ways of integrating the contact information (such as rejection sampling).
strengths_and_weaknesses: I am not too familiar with shape completion, but the method seems reasonable. I did not understand what the OOD baseline meant - it is clear that RGBD data is augmented with contact information in some way, but it's not clear what this means. My interpretation was that contact information was propagated to the PSSNet in some way, and predictions from PSSNet were used directly, rather than doing a live constraint update like CLASP does, but if this is how that baseline works, the details are fairly sparse.

To me the main weakness is on the baselines. The main baseline used is rejection sampling, which the authors note is fairly competitive until they hit a point where the rejection sampler returns 0 valid object configurations. Given that rejection sampling was fairly effective, this makes me wonder how well the method does if the rejection sampling selects the object that violates the *minimum* number of constraints rather than requiring 0 constraint violation. It seems unfair to exclude comparisons to rejection sampling at all in the higher # of contacts setting. I am also curious about more complex MCMC algorithms in general for baselines.

The other main weakness of the method is that it is fairly reliant on the shape prior. As the authors noted, when they tried using a VAE-GAN for the shape prior, they found that average samples were more accurate, but samples were also much less diverse, and this harmed the contact info update. Thus, the effectiveness of CLASP seems tightly coupled to properties of the learned encoder / decoder...but this isn't really the focus of the paper, the paper is focusing on how to augment the shape prior, so I am not considering this for the review.
summary_of_recommendation: The method is very reasonable but I am unsure of its effectiveness.
issues: I would appreciate stronger comparisons to simple methods of using contact info to constrain the prior and would be willing to improve the rating if this were done.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a method to refine single-view shape completion predictions through a combination of free-space carving and robot collisions. The approach iteratively updates a believe over object shapes using a particle filter in the latent space of a shape completion network. It is demonstrated that the approach can improve the initial shape predictions, both in simulation and in a robotic experiment, for objects that are quite close to the objects used during training. 
strengths_and_weaknesses: Strength:

- The approach takes into account the ambiguities of single-view object reconstructions by refining a distribution over possible shapes
- The problem, i.e. how to update the believe of previously predicted 3D geometry when encountering robot collisions, is challenging and relevant
- It is shown that by reintroducing constraints such as very similar objects with simple shapes during training and testing, collision detection and free-space carving can improve results
- Both, the paper and the video contain a clear presentation of the chosen concepts
- The paper discusses problems and limitations of the presented approach that are related to the underconstrained nature of the problem. 
- Limited results can still be important to initiate further research.

Weaknesses:

- The tackled problem is severely underconstrained and we can therefore not expect good results:  
1.) Shape completion ambiguities through partial views  
2.) No knowledge of the contact position  
3.) No knowledge on the environment   
4.) Static scene assumption while dealing with collisions  
- The problem can only be made tractable by reintroducing a lot of constraints that severely limit the general applicability of the approach:  
1.) Objects during training and testing are very similar  
2.) Only simple shapes are considered  
3.) The objects are fixed to the ground so that they do not move  
4.) Multi-object setup is basically a wider version of a single object (stacked boxes)  
- The rejection sampling sometimes works better until an additional contact makes all shapes implausible. I would have liked to see some sort of soft rejection sampling as a baseline since the estimated contact positions are only broad approximations. 
- The baselines are mostly ablations of the presented method. Alternative baselines are not considered. E.g. since the shape completion networks anyways do not generalize over multiple categories, we could also use intra-class shape retrieval / parametric shape estimation / categorical pose estimation. Collisions at the back of the object could still be used to (softly) constrain the selection of the right pose/shape.
- The applicability of the approach in a downstream-task is not shown. At this accuracy, I could only imagine motion planning, but then you could also make the planner react to collisions and adapt its plan, directly. Or by just moving the camera / having two cameras to remove ambiguities.
- It is unclear how much CLASP helps the (often better) VAE-GAN results, I can only observe a gain in the simulation results of cheezeit (shallow) over number of contacts
summary_of_recommendation: I am leaning towards weak accept, mainly, because the paper and video are well crafted, but I could also understand if other reviewers lean towards rejection because the practicality is questionable. The method of iteratively refining latent codes is not new as also noted in the paper [30]. The authors try to use even less information to locate contact points (just collision information). I believe that to make the problem tractable and to achieve the accuracy needed in robotic applications we not only need better shape completion methods but also touch-sensitive skin. Trying to circumvent this via constraint optimization seems a bit hopeless but the general method might become interesting when these components are available. 
issues: As mentioned before, the paper is well written. Some of the general mathematical descriptions (e.g. Section 3) are not really adding anything to the text, but this is a matter of taste.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents Constrained Latent Shape Projection (CLASP), a method for refining object shape (originally inferred from vision) with information from robot contact/tactile information. The overall framework involves a sensor fusion which is based on a particle filter, and using CLASP --which is a projection of the particles/samples onto contact constraints in the latent space-- as the measurement update. The projection technique itself is quite interesting: it's similar to the training process of the neural network (via backpropagation algorithm), but instead of optimizing the weights, it optimizes the inputs. In the experimental section, ablation studies as well as comparison with other method such as VAE-GAN are performed on both simulation and a real robot.
strengths_and_weaknesses: Strengths:
(1) The constraint latent space projection technique is quite interesting: it's similar to the training process of a neural network (via backpropagation algorithm), but instead of optimizing the weights, it optimizes the inputs.
(2) In the experimental section, ablation studies as well as comparison with other method such as VAE-GAN are performed on both simulation and a real robot.
(3) Source code is provided in the supplementary materials.

Weaknesses:
(1) A missing reference: S. Luo, W. Mou, K. Althoefer and H. Liu, "Localizing the object contact through matching tactile features with visual map," 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 3903-3908, doi: 10.1109/ICRA.2015.7139743.
(2) (a) Figure 2 is not very clear: what are the pink and blue rectangles in between the "sample" arrow and the "f_{dec}" trapezoid? Maybe worth checking if other readers understand this figure well, including each components drawn in it. Does each component of the figure provide a useful information?
(b) Figure 1: bottom right figure (third row, vision + all contacts) is missing?
summary_of_recommendation: While the technique presented in the paper is interesting, the way of communicating it to the reader (e.g. via the figures) may need to be improved.
issues: Please see the points mentioned in the "Weaknesses" section above as well as a few additional suggestions below, and please try to address them. Thanks!

Additional suggestion(s):
(1) In the experimental results/video, it would be nicer if the authors could highlight regions of the point clouds that were cut/chamfered due to the additional contact information, to get a better sense/reasoning of how the trained model behaves.
review_title: null

Meta-review: This paper presents an extension to the PSSNet diverse object shape reconstruction work to incorporate reconstruction from contact sensing in addition to vision.

In the updated version the authors provide a comprehensive set of baseline comparisons and do a better job of situating it in the broader reconstruction literature.

As such I think this would make a nice addition to the conference.

I would recommend the authors do another pass through the comments of the reviewers to make sure everything is correctly addressed. I also think I suggested the wrong reference from Lu in my previous meta-review, as I intended the authors to include a discussion and reference of their reconstruction work ("Learning Continuous 3D Reconstructions for Geometrically Aware Grasping" https://ieeexplore.ieee.org/abstract/document/9196981)
Decision: Accept (Poster)