CLASP: Constrained Latent Shape Projection for Refining Object Shape from Robot Contact
Authors: Brad Saund
Dmitry Berenson
Venue: CoRL2021 Poster (2021)
Keywords: Shape Completion
Contact Sensing

Abstract: Robots need both visual and contact sensing to effectively estimate the state of their environment. Camera RGBD data provides rich information of the objects surrounding the robot, and shape priors can help correct noise and fill in gaps and occluded regions. However, when the robot senses unexpected contact, the estimate should be updated to explain the contact. To address this need, we propose CLASP: Constrained Latent Shape Projection. This approach consists of a shape completion network that generates a prior from RGBD data and a procedure to generate shapes consistent with both the network prior and robot contact observations. We find CLASP consistently decreases the Chamfer Distance between the predicted and ground truth scenes, while other approaches do not benefit from contact information.

CLASP: Constrained Latent Shape Projection for Refining Object Shape from Robot Contact
Brad Saund and Dmitry Berenson
Robotics Institute
University of Michigan
bsaund@umi ch. edu, dmitryb@umich.. edu
Abstract: Robots need both visual and contact sensing to effectively estimate the state of their environment. Camera RGBD data provides rich information of the objects surrounding the robot, and shape priors can help correct noise and fill in gaps and occluded regions. However, when the robot senses unexpected contact, the estimate should be updated to explain the contact. To address this need, we propose CLASP: Constrained Latent Shape Projection. This approach consists of a shape completion network that generates a prior from RGBI) data and procedure to generate shapes consistent with both the network prior and robot contact observations. We find CLASP consistently decreases the Chamfer Distance between the predicted and ground truth scenes, while other approaches do not benefit from contact information.
Introduction
You look into a cabinet and box of crackers. You reach in and attempt grab the box from the side, but your fingers hit something. Perhaps this box is larger than you thought? Your mental model of the box updates, you try a wider grasp, and you successfully retrieve your snack. Robots currently not so adept. While they can estimate the pose of known shapes [1] or estimate parameters of objects [2], they cannot yet fuse this visual and contact information to draw from the wide range of shape priors in the world. A robot could try to learn its next action directly from vision and force feedback instead [3], but this approach lacks the logic to generalize scenarios not seen training.
We propose a method that allows robots to mimic the process of updating object shape from contact information. A shape completion neural network first generates beliefs over possible object shapes based on visual RGBD data. The belief updates the object shape to be consistent with contact information gathered by robot moving in the scene. We make the realistic assumption that the RGBD) camera perceiving the scene suffers from sensor noise and occlusion. We assume the robot sense if it collides with an object, but not where the contact was made (i.e., sensorized skin). Many of the "cobot" platforms available today utilize this contact model to detect collision and stop before harming person.
Formally, this type of contact creates a contact manifold, thin space of shapes with boundary bordering the robot. Past work has projected shapes onto the contact manifold in object pose space [4] and robot configuration space [1], but both require known shape geometry. Our objective is to update the unknown shape geometry to satisfy contact constraints. Returning the cracker box example, the robot will be unsure the contact occurred at the top finger, the bottom finger, or perhaps the back of the hand or the elbow. Filling in all possible contact points would lead to absurd scenes with robot shells protruding from the cracker box. However, ignoring this contact information leaves the robot with the original belief of the thinner cracker box and explanation of why the attempted grasp failed. Our shape completion network generates a prior in latent shape space which can be decoded into shapes in workspace; however, shapes generated directly from this latent prior are unlikely to satisfy contact constraints.
Code is available at http:///shtttaoooooooooaaaaaaaat
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Initial
logaa
World
fens
fdee
Shapes proe
ORUT
Iterative Gradient Update (projection)
Beliefs
fave
vision only
Sample
1sl contact
all contacts
fave
Figure 1: A visual RGBD view of objects leave ambiguity final shape due to sensor noise and occlusion, which store set of sampled scenes in a particle filter. Contact information (pink) reduces ambi-
Figure 2: CLASP Architecture
Top: Shapes sampled from RGBD using PSSNet [5].
Middle Right: A robot motion detects free space (light
blue) and collision set (pink).
Middle Left: Latent samples are projected to satisfy
guity, and using CLASP the particles con- verge the true shape.
contacts (green). Ovals depict the latent prior.
Bottom: Final samples satisfy the contact constraints.
Our key insight is that latent samples from our neural network can be projected onto the contact manifold in the latent shape space using iterative gradient descent, creating shapes both likely under the visual prior and consistent with the contact information. We further expect these projected shapes to be closer to ground truth than direct samples not considering contact.
We accomplish this with our proposed Constrained LAtent Shape Projection (CLASP), which stores a belief over shapes in particle filter. Each particle represents a collection of latent object shapes which can be decoded into scene. Every new robot measurement of contact and freespace triggers an update on all particles. During each particle update, gradient steps are taken to increase the occupancy likelihood of the most likely point(s) explaining the contact(s), decrease the occupancy likelihood of the free points, and increase the latent likelihood under the shape prior.
We test this method both in simulation and live robot by constructing scenes of objects tabletop, generating robot motions that generate freespace and contact observations (Fig. 1), updat- ing the belief using CLASP well baselines, and comparing the set of sampled shapes to the ground truth scene. We find CLASP outperforms both ablations of CLASP, as well as the approaches of Rejection Sampling and directly updating the input the shape completion network. We also find that CLASP produces more accurate scenes than a VAE_GAN shape completion network.
2 Related Work
Shape Completion from Vision: The goal of Shape Completion is to predict full shape from single partial input. Recently, neural networks have become a popular method of shape completion. A common network architecture learns encoder
feature space followed by decoder to the shape output [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16. 17]. Scene Completion networks are trained on larger spatial volumes of occupied points and use similar architectures with adaptations to join information from multiple scales [18]. While most methods predict the single best estimated shape, build off work that uses variational autoencoder architecture to produce plausible and diverse shapes [5]. We draw from the vast work on shape and scene completion and contribute a method that improves scene estimates using contact information from robot.
Shape Completion from Touch: In different works, "touch" can refer to a single known contact point, contact configuration, force-torque measurements, rich tactile sensor. Work using the

---

definition of contact point or contact configuration typically uses touch to reduce the version space of shape possibilities [19], but such approaches cannot tractably capture the diversity of all shapes. Alternatively, some situations model known shapes with unknown poses [2]. Both classical Iterative Closest Point [20, 21] and neural networks [22, 23, 24] have been used to predict valid poses. To generate samples consistent with contact information, the Implicit Manifold Particle Filter projects sampled poses onto the contact manifold using an iterative approach [4]. Analogously, we project sampled particles onto the contact manifold in the latent shape space of our neural network.
Touch can also refer to the rich tactile sensors such as GelSight [25] or soft-bubble grippers [26], with input more analogous to images. Tactile patches can be directly mapped to visual features [27]. Alternatively, neural networks have used these sensors for material classification [28] and grasped pose estimation [29]. We do not assume our contact sensing has such rich information.
Combining Vision and Touch: A neural network can combine vision and touch (force + torque [3], or GelSight [30D using separate encoders to a latent space for each sensing modality alongside a decoder variety of spaces. We considered a similar encoder structure with a decoder to produce completed shapes, but this would require a large dataset of (Shapes x Contact + Freespace) mea- surements and the resulting network would be only applicable to the robot used for training. The loss of a neural network can be tweaked during training to bias towards priors similar to contact, such as connectivity and stability [31]. neural network can output a downstream objective, such as grasp success probability, instead of shape reconstruction, and thus may be successful for a diverse array of objects where accurate reconstruction not available [32]. Insteaed of neural network, surface reconstruction has be done using Gaussian Proccess (GP) prior fit to tactile measurements [33, 34).
Our method is most similar the work of Wang et al. [35], which uses gradient descent the latent space of a shape completion network to enforce touch constraints. Where that work uses a high-resolution GelSight tactile sensor refine shape details previously reconstructed from vision, our work focuses on ambiguous shapes (e.g. a box with unknown depth, or novel shapes not in the training data) and the lower information measurement of contact detection. We accomplish much larger shape updates by using a diverse set of predictions and novel projection loss function.
3 Problem Formulation
Consider robot R observing static scene composed of specific objects o, sampled from some distribution of objects O. The objects divide the workspace into occupied space Woce and free space Wfree W Wooe The robot has access to a training subset of O beforehand, but does know the specific objects 0j in the current scene.
The robot observes the scene with two distinct sensing modalities. In the visual modality, the robot views the scene from a stationary RGBD camera receiving color depth images Im. Due to sensor noise and occlusion these depth images offer incomplete and noisy measurement on the full region of Woce occupied by the obstacles. From the camera image we assume the scene can be segmented into distinct objects from O.
For the tactile modality, consider robot that is able to sense if it has made contact with any object, but not where along the robot surface the contact was made. We assume the contact does not move the objects. For a configuration in configuration space q E C, let R(y) C W denote the region of workspace occupied by the robot. A robot that has visited configurations (91,42,...) = Qferee CC C
without observing contact can carve out regions of known free space:
U qeQ
R(g)= = Wknown_free Wfree
(1)
For each configuration Icontact E Qcontact where contact is observed, there must be at least one object point in collision with the robot (and not in known freespace).
Vicontact E Qcontact 3 Peontact (R(geontact) \ Wennwn_free) Peontact E Wooc
(2)
Using existing nomenclature, each such region is called Collision Hypothesis Set (CHS) [36].
Our objective is to model the conditional occupancy p(Woce|O) Im, Qree:: Qrontact). Specifically, we desire stochastic function g(O, m,, Qiree, Qcantact) which generates sample Wace as similar as possible to the true conditional distribution. Since the true conditional distribution is unknown, in practice we seek to minimize the distance of drawn samples to the ground truth scene.
3

---

4 Method
Our approach is to use a particle filter storing a collection of latent shapes. We first segment the scene into distinct objects, then use an existing shape completion neural network to draw latent shape samples , for each object from p( O Im), initializing the particle filter. Each particle can be decoded into the objects in scene, thus the collection of particles represents the belief pl p(WocclO, Im). We propose Constrained LAtent Shape Projection (CLASP) as the measurement update, projecting these samples onto the constraints imposed by free and Qcantact. Our method is shown in Fig. 2, where the trapezoids fenc and fdee are the encoders and decoders of PSSNet [5].
4.1 Initial Belief
The RGBD camera images are passed segmentation algorithm, which yields distinct pixel re- gions in the image corresponding to different objects Oj. For each object oj the corresponding portion of the depth image converted first point cloud, then voxelgrids of known-oceupied and known-free space centered around the visible object points with a transform T3 mapping the voxelgrid to the workspace coordinates.
For each object 0y we use the Plausible Shape Sampling Network (PSSNet) [5] f to generate possible variational autoencoder. An encoder fenc maps the
shape completions. PSSNet is structured known-free and known-occupied voxelgrids to a mean and variance in latent space. A latent vector can be sampled and passed to the decoder fdee- which outputs a probability of occupancy for each voxel. Thresholding (e.g. > 0.5 for each voxel) yields a completed shape.
An object 0j that is representable by f can be stored compactly as b such that fdee(t's) oj. The transform Tj maps the completed shape into the workspace frame. A world is composed of static objects foi, 02....
O. A particle ó stores a specific world as a sequence of latent space vectors {U1, 122,... We sample worlds conditioned on only the depth-image observation by independently sampling latent vectors of objects. The initial belief is set of particles {ø1, 22.... &b generated from the information from the depth camera before any robot motion.
4.2 Projecting single object
Sampling particles using only camera information may yield worlds that are inconsistent with the robot contact information. For example, PSSNet may predict objects that extend far into occluded space that intersect regions the robot has moved through. Alternatively, PSSNet may predict objects that do not extend into occluded space, and so the robot may observe contact with no object to explain the collision. Predicting shapes from vision and robot contact in single pass would require a dataset specific to each robot and a specific set of motions.
To resolve these inconsistencies, sampled particles are projected onto the constraints in the latent space of the shape completion network, shown Fig. For sample of object j. f induces a workspace occupancy. Our constraints lie in the workspace, but we wish to project the latent space vector. Therefore, the projection is accomplished by optimizing a loss via gradient updates on vi while holding feee fixed, mirroring the process of training a neural network but optimizing the input instead of the network weights. Consider the unthresholded voxelgrid with values between and produced by the decoder: Sáee( WW
We optimize the loss: Loll = Lfree Loce + Eprior
The first term free penalizes all voxels predicted above a threshold ó that are known to be free.
[W) v. 5,0) Winoon (2, y. 2) = 1
free
(3)
otherwise
The second term Loee penalizes unexplained contact. Each contact Geontact must be caused by some object s.t. R(geontact) n Woce # W. however it is not obvious which object is responsible for the contact, or which voxel of the object contacted the robot. During optimization we consider a specific assignment of Coontact to object oj- We define the assignment process in Section 4.3. Because a single occupied voxel is enough explain contact, at each iteration the loss is optimized based on

---

the maximum prediction of occupancy overlapping with the collision hypothesis set.
Loc
Σ
max
1 max (R(2) (W))
(4)
DEQcontact
The final term Eprior penalizes deviation of v from the original distribution predicted by the encoder. Without this constraint, & can deviate arbitrarily, losing all dependence on the depth image and leaving the training domain of fdec. This would produce completions that longer look like objects. Eprior is weighted by a to maintain a similar magnitude of gradients to freee and Loce.
Eprior -a log (P(w))Senne
(5)
Sampling the occupancy for a specific object oy given Im, Q free and Pcontact is thus accomplished by sampling a 3 and optimizing until the constraints are satisfied. Projection can fail if an iteration limit, set to 100 steps, is reached without satisfying the constraints. For practical efficiency this failure can sometimes be detected early when gradient updates longer change the loss and the constraints are not satisfied. We use Adam [37] for optimization with a learning rate of 0.01.
4.3 Multi-object completion
CLASP stores an assignment of each Icontact to particular object oy for each full-scene particle When a measurement contains new contact icontact, it is assigned to specific object for each sampled particle as follows. First, the output of our shape completer is a finite-sized voxelgrid, typically smaller than the full scene. The new Jcontact cannot be assigned to any object where (yconntact) lies entirely outside the output region of the decoder /dee(()) Next, for each remaining j. projection is attempted for each et to satisfy the new lcontoct- If all attempts fail, we assume this Jcottact not caused by object j. For each full-scene particle i, specific assignment of Gcontact is randomly and uniformly selected from the remaining objects j that could possibly explain the contact.
Experiments
We evaluated scenarios of different objects to determine if CLASP improves the estimate of the scene using robot contact information. We tested ablations of CLASP to evaluate the importance of the latent prior and constraint satisfaction. We also tested alternative approaches CLASP that did not rely projection. Finally, we compared CLASP on two different network architectures and trained on multiple datasets. We trained separate instances of PSSNet [5] on Axis-Aligned Boxes (AAB), YCB [38], and ShapeNet mugs [39] (training details in Section A.1).
5.1 Robot Contacts
To generate contact measurements we moved the right arm of robot composed two Kuka iiwa arms with Robotiq 3- f-ngger grippers. We generated scenes by manually placing simulated objects from AAB, YCB, or ShapeNet on a virtual table at about camera height. The known voxels passed to trained PSSNet generate set of possible completions.
Simulation:
We generated robot motions to gather information by moving near and sometimes contacting the objects using the procedure described in the appendix A.2. The first motion typically sweeps known free space rather than making contact. The second or third motion intentionally makes contact with the object.
The physical kinematics of our robot matched the simulated robot. Kinect depth camera mounted at the "head" position generated the RGBD images. A calibrated motion capture system provided transforms between the Kinect and robot frames. We segmented the RGB image using the CSAIL semantic-eegmentttinnnnyeeee Each segmentation was converted into a voxelgrid and fed to PSSNet as in simulation to generate sample worlds. The same procedure was used to generate robot motions as in simulation.
Live Robot:
library [40] which we retrained on YCB objects.
5

---

Ablittios for Simulation Mug: Shapener mugs
Ablations for Live Cheezit: YCB
PSSNet CLASP
PSSNiu CLASP:ignune
0.65
projectione
Chumter Dalatie (tm)
Dsstance (em)
0.60
0.55
0.50
0.45
0.40
(9)
1. [00
2 1200
(1) Total Contacts)
003
(0)
2 [43
3 (2)
(2)
[2)
(43
10 (73
Basclines fo Simulation Mug: Shapenet mugs
Basetines for Live Cheezit: YCB
0.9
PSSNet - CLASP
Edit
Rejection Samoling PretNet Soft Pejeection Sampling
Chammeer Distance (cml
PSSNet 000
Chamter Distance
GAN
0.0
0.5
0.4
0.003
1:403
2.00)
3-22)
5:63) (4) (7)
Figure 3: Boxplots showing the Chamfer Distance from sampled particles to ground truth. The mean, middle quartiles (boxed colored region), and outer quartiles excluding outliers are shown. Rejection Sampling and VAE_ WAE_GAN occasionally produced no valid shapes, in which case no box is displayed.
MultiObject: YCB
PSSket CLASP
3.4
PSSket Clasp: No contact disambiguation
CLASP
Chanperr bitance (an)
3.2
3.0
0.6
6.4
[00 1 (1) 3(1) 4
5/33 6 7 & Contacts)
Figure 5: The Deep Cheezit (left), Mug (Mid- dle). Live Cheezit and Live Pitcher (Right) scenes. The occupied (black) and known free (not shown) voxels from vision with contact (transparent red) and robot free (not shown) voxels are used by CLASP to generate com- pleted shapes (green).
Figure 4: Boxplot results for the multiobject PSSNET CLASP: No CONTACT DIS- AMBIGUATION fails to project any samples for observations and beyond, there not cor- responding box.

---

On the live robot, contact was determined at each configuration by checking if the measured external torque exceeded a threshold 2.m per joint. This threshold was large enough to avoid generating false positives while remaining sensitive enough to detect contact with secured objects.
5.2
Scenes
We tested four scenes in simulation and two on the live robot. Each scene consisted of a single object secured to the table in front of the robot. We also tested scene of multiple YCB objects (Fig. 6). Contacts occurred with occluded sec- tions of the objects, with examples shown in Fig. 5. both simulation and the live robot, table occupancy
not considered when eval- uating the quality of the completions.
Simulated Scenes: The first pair of scenarios used a single YCB Cheezit box (Shallow) and stack of three Cheezit boxes (Deep). These setups generated similar depth images but dif- ferent ground truth shapes. Both used net- works trained on the AAB dataset. The Simu- lated Pitcher from YCB was positioned with the handle occluded from view and used networks trained on the full YCB dataset. The Simulated Mug from ShapeNet also had the handle oc- cluded from view and used networks trained on all mugs in ShapeNet. The handles these objects were localized through contact.
Figure The multiohject scene with the YCB Cheezit box and Drill on table (not shown) just before the first contact.
Live Scenes:
The Live Cheezit also consisted of stack of three boxes, and again the Live YCB Pitcher had the handle occluded. Both scenes used networks trained on the full YCB dataset. The Cheezit boxes were attached together and the pitcher was taped to prevent motion during contact. Simulated objects were manually aligned the live scene to approximate the ground truth live objects and were used for evaluation.
5.3 Baselines
We compared our proposed method to the following alternatives. DIRECT EDIT directly adds or removes voxels to satisfy the contact information. REJECTION SAMPLING samples latent space vectors from the distribution predicted by the encoder, then decoded these into 3D objects and rejected any samples not satisfying the contact free space constraints. SOFT REJECTION SAM- PLING selects and then Directly Edits the least violating samples in the cases where all samples are rejected. For OOD (Direct Out-Of-Distrro Of-Distribution Prediction), occupied voxels with the contact known free and occupied voxels as input to PSSNet. While other methods did not have access to the true contact point, allowed this method this advantage and added the true contact point directly to the known occupied voxels from the depth image. Finally, while other approaches used the PSSNet shape completion network, tested using VAE_GAN [17] network. This network tends to produce better average but less diverse samples. Section A.3 describes these baselines in more detail.
combined the visual known free and
We also tested ablations of our method. CLASP: IGNORE PRIOR tested removing the loss term Eprior. CLASP: ACCEPT FAILED PROJECTIONS tested accepting all projections, even those that do not satisfy the contact constraints. To test our contact assignment the multi-object case (Sec- tion 4.3), CLASP: No CONTACT DISAMBIGUATION determined if projection of latent thy could satisfy each new Gemntact as in CLASP, then assigned each goontant to all feasible objects j. This resulted in scenes explaining a single Goontact with multiple objects.
100 particles were sampled in each method, with the threshold of L.rre set at é 0.4 and the weighting of Eprior set at 0.01.
7

---

5.4 Results
Single-object scenes were tested on all baselines and ablations using PSSNet trained with the ap- propriate dataset. Fig. 3 compares the Chamfer Distance (CD) [41] of each accepted sample to the ground truth for selected scenes. Plots for all scenes are shown in Section A.3. We consider a different analysis in Section A.4. We find that REJECTION SAMPLING often performs well during the first few observations with zero or one contact. However, REJECTION SAMPLING soon fails to return any valid samples with two or more contacts. We see that OOD produces completions that are typically much worse than the original completions from only vision. The initial estimate (observation 0) from VAE_GAN are hit-or-miss. All networks saw the YCB Pitcher during training, and VAE_GAN recalls the pitcher more accurately than PSSNet during testing to the point where contacts are unnecessary. However, the recall of VAE_GA GAN in the other ambiguous scenarios is worse than PSSNet and projection to the contact constraints often fails, leaving no sampled shapes. The results justify choice of PSSNet over VAE.GAN for CLASP, as VAE.GAN is unable to sufficiently adjust to the contact information.
Considering ablations of CLASP, ACCEPT FAILED PROJECTIONS performed as well initially (when no projections fail) and significantly worse as the number of observations increases. Ignoring the la- tent prior during projection also performs worse and occasionally produces shapes that qualitatively look less like objects compared to completions from other methods.
Across all scenes, CLASP performed similarly to the best of all other methods with 0 or 1 contacts and the best with multiple contacts. CLASP successfully used the robot contact information in all scenarios to reduce the CD between the predicted and ground truth shapes in all scenarios. Robot measurements with a contact typically caused a larger reduction in CD than measurements with only freespace information. Numerically, the CD reduced most in the Cheezit scenarios, with a reduction of the mean from 0.5cm 0.1cm for the Shallow and from 0. 14c4m to 0.08em for the Deep. The CD reduction in the pitcher and mug scenarios was significantly smaller, as the general shape of the pitcher and mug could be predicted from the image. The prediction of the occluded handle was improved with contact. The trend of improvement in the live scenes matched the simulation. However, the numeric error of the live scenes was much larger, perhaps caused by imperfect transfer of the learned shape completer from training in simulation to prediction on live Kinect data as well as imperfect alignment of the robot frame to the Kinect frame.
In the multi-object scene (Fig. 4) the VAE GAN method achieves a better completion from the RGBD data, but our proposed method produces better samples after 2 contacts. Our proposed con- tact assignment (Section 4.3) outperforms naively satisfying the contacts whenever possible.
6 Discussion and Conclusion
While we model CLASP using a particle filter and would like to have the Bayesian estimate of the scene given all observations, acknowledge many non-Bayesian approximations. Particle filters approximate Bayes filters, but the 100 particles we sample may not be a sufficient coverage of the latent shape space. CLASP projects samples, which does not preserve Bayesian estimates.
We demonstrated a method for estimating shape completions initialized with purely RGBD visual data, then updated from observations of robot arm moving through unknown regions and sensing contact. We stored the belief of the scene as particle filter of latent vectors from shape com- pletion network and used CLASP to enforce shape consistency with the robot observations. Most importantly, we showed that CLASP improves the estimate object shape using these contact ob- servations. Our results further showed that CLASP performs better than ablations of CLASP and alternative methods. We hope CLASP will be used within a larger robotics framework where rea- soning over environment uncertainty based on shape priors aids in accomplishing larger goals.
Acknowledgements: This work was supported in part the Toyota Research Institute and by NSF grant 1SS-1750489 and ONR grant authors.
This article solely reflects the opinions of its
8

---

References
[1] M. Klingensmith, M. Koval, S. Srinivasa. N. Pollard, and M. Kaess. The manifold particle filter for state estimation on high-dimensional implicit manifolds, April 2016.
[2] K. Desingh, S. Lu, A. Opipari, and O. C. Jenkins. Factored pose estimation of articulated objects using efficient nonparametric belief propagation. ICRA, 2019.
[J Lee, Y. Zhu, Zachares, M. Tan, Srinivasan, S. Savarese. L. Fei-Fei. A. Garg, and Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. JEEE Transactions on Robotics, 2020, doi: 10.1 11190000.00001111 201..2959445.
[4] M. C. Koval, N. S. Pollard, and S. S. Srinivasa. Pose estimation for planar contact manipulation with manifold particle filters. IJRR, 2015.
[5] B. Saund and D. Berenson. Diverse plausible shape completions from ambiguous depth images. CoRL, 2020.
[6] Zhirong Wu, S. Song, A. Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao. 3D shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
[7] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3D-R2N2: unified approach for single and multi-view 3D object reconstruction. In ECCV, 2016.
[8] R. Girdhar, D. Fouhey, M. Rodriguez, and A. Gupta. Learning predictable and generative vector repre- sentation for objects. ECCV, 2016.
[9] Wu, Zhang, X. Zhang, Zhang, T. Freeman, and B. Tenenbaum. Learning shape priors for single-view 3D completion and reconstruction. In ECCV, 2018.
[10] Wu, Y. Wang, T. Xue, X. Sun, W. T. Freeman, J. Tenenbaum. MarrNet: 3D Shape Reconstruction via 2.5D Sketches. In Advances In Newral Information Processing Systems, 2017.
[11] M. Michalkiewicz, E. Belilovsky, M. Baktashmotlagh, and A. Eriksson. A simple and scalable shape representation for 3D reconstruction. arXiv, 2020.
[12] C. Wen, Y. Zhang, Z. Li, and Y. Fu. Pixel2mcsh++: Multi-view 3D mesh generation via deformation. ICCV, 2019.
[13] H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In ICCV, 2019.
[14] H. Fan, H. Su, and L. Guibas. A point set generation network for 3D object reconstruction from single image. CVPR, 2017.
[15] Y. Yu, Z. Huang, F. Li, H. Zhang, and X. Le. Point encoder gan: A deep learning model for 3D point cloud inpainting. Neurocomputing, 2020.
[16] B. Yang. S. Rosa, A. Markham, Trigoni, and Wen. Dense 3D object reconstruction from a single depth view. In TPAMI, 2018.
[17] Wu, C. Zhang, Xue, B. Freeman, and J. Tenenbaum. Learning a prohabilistic latent space object shapes via 3D generative-aadverarii modeling. In NIPS, 2016.
[18] L. Roldão, R. de Charette, and A. Verroust-Blondet. 3d semantic scene completion: a survey. ArXiv, abs/2103.07466,, 2021.
[19] J. Jung. Active shape completion using tactile glances. In Master's Thesis: Ludea University of Technol- ogy, 2019.
[20] P. Besl and N. D. McKay, A method for registration of 3-D shapes. TPAMI, 1992.
[21] H. Yang, J. Shi, and I.. Carlone. TEASER: Fast and Certifiable Point Cloud Registration. arXivv, 2020.
[22] V. Narayanan and M. Likhachev. Discriminativel--uuie guided deliberative perception for pose estimation of multiple 3D object instances. In RSS, June 2016.
[23] X. Deng, A. Mousavian, Y. Xiang, F. Xia, T. Bretl, and D. Fox. PoseRBPF: A rao- no-blackwwellized particle filter for 6D object pose estimation. In RSS, 2019.

---

[24] T. Hodan, D. Barath, and J. Matas. EPOS:
Estimating 6d pose of objects with symmetries. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[25] W. Yuan, S. Dong, and E. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geome- try and force. Sensors, 2017.
[26] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake. Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation. In RoboSoft, 2019.
[27] S. Luo, W. Mou, K. Althoefer, and H. Liu. Localizing the object contact through matching tactile features with visual map. ICRA,201 2015.
[28] W. Yuan, Y. Mo, S. Wang, and E. Adelson. Active clothing material perception using tactile sensing and deep learning. ICRA, 2018.
[29] N. Kuppuswamy, A. Alspach, A. Uttamchandani, S. Creasey, T. Ikeda, and R. Tedrake. Soft-bubble grippers for robust and perceptive manipulation. In IROS, 2020.
[30] Y. J.-. Zhu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In CVPR, 2019.
[31] W. Agnew, C. Xic, A. Walsman, O. Murad, C. Wang, P. Domingos, and S. S. Srinivasa. Amodal 3d reconstruction for robotic manipulation via stability and connectivity. CoRL, 2020.
[32] Q. Lu, M. V. der Merwe, and T. Hermans. Multi-fingere active grasp learning. IROS, 2020.
[33] Z. Yi. R. Calandra, F. Veiga, H. van Hoof. T. Hermans, Y. Zhang, and J. Peters. Active tactile object exploration with gaussian processes. IROS, 2020.
[34] S. Dragiev, M. Toussaint, and M. Gienger. Uncertainty aware grasping and tactile exploration. ICRA, 2013.
[35] S. Wang. J. Wu, X. Sun, W. Yuan, W. Freeman, J. Tenenbaum, and E. Adelson. 3d shape perception from monocular vision, touch, and shape priors. In IROS, 2018
[36] B. Saund and D. Berenson. Motion planning for manipulators in unknown environments with contact sensing uncertainty. In ISER, 2018.
[37] D. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning
Reprexentations, 2014.
[38] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Yale- CMU-Berkete dataset for robotic manipulation research. IJRR, April 2017.
[39] A. X. Chang, T. Funkhouser. L. Guibas, P. Hanrahan, Q. Huang. Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information Rich 3D Model Repository. Technical Report arXiv, 2015.
[40] B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision, 2018.
[41] H. Barrow, J. Tenenbaum, R. Bolles, and H. Wolf. Parametric correspondence and chamfer matching: Two new techniquesfor image matching, IJCAI, 1977.
[42] F. S. Nooruddin and G. Turk. Simplification and repair of polygonal models using volumetric techniques. IEEE Transactions on Visualization and Computer Gruphics. 2003.
[43] P. Min. binvox. http://www. wttp::/wwww.rtit.n.t/// com/binvox, 2004- -0000.
[44] D. Berenson, S. Srinivasa, and Kuffner. Task space regions: A framework for pose-constrained manip- ulation planning. International Journal Robotics Rexearch, 30(12):1435 1460, October 201
[45] B. Saund, S. Chen, and R. Simmons. Touch based localization of parts for high precision manufacturing. In ICRA, 2017.
[46] A. Hermann, F. Drews, J. Bauer, S. Klemm, A. Roennau, and R. Dillmann. Unified GPU voxel collision detection for mobile manipulation planning. In IROS, 2014.
10


Reviews:
[{"review_id": "wD2_Cxx49V", "reviewer": "Reviewer_URGm", "summary": "The paper presents a method to refine single-view shape completion predictions through a combination of free-space carving and robot collisions. The approach iteratively updates a believe over object shapes using a particle filter in the latent space of a shape completion network. It is demonstrated that the approach can improve the initial shape predictions, both in simulation and in a robotic experiment, for objects that are quite close to the objects used during training. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The approach takes into account the ambiguities of single-view object reconstructions by refining a distribution over possible shapes\n- The problem, i.e. how to update the believe of previously predicted 3D geometry when encountering robot collisions, is challenging and relevant\n- It is shown that by reintroducing constraints such as very similar objects with simple shapes during training and testing, collision detection and free-space carving can improve results\n- Both, the paper and the video contain a clear presentation of the chosen concepts\n- The paper discusses problems and limitations of the presented approach that are related to the underconstrained nature of the problem. \n- Limited results can still be important to initiate further research.\n\nWeaknesses:\n\n- The tackled problem is severely underconstrained and we can therefore not expect good results:  \n1.) Shape completion ambiguities through partial views  \n2.) No knowledge of the contact position  \n3.) No knowledge on the environment   \n4.) Static scene assumption while dealing with collisions  \n- The problem can only be made tractable by reintroducing a lot of constraints that severely limit the general applicability of the approach:  \n1.) Objects during training and testing are very similar  \n2.) Only simple shapes are considered  \n3.) The objects are fixed to the ground so that they do not move  \n4.) Multi-object setup is basically a wider version of a single object (stacked boxes)  \n- The rejection sampling sometimes works better until an additional contact makes all shapes implausible. I would have liked to see some sort of soft rejection sampling as a baseline since the estimated contact positions are only broad approximations. \n- The baselines are mostly ablations of the presented method. Alternative baselines are not considered. E.g. since the shape completion networks anyways do not generalize over multiple categories, we could also use intra-class shape retrieval / parametric shape estimation / categorical pose estimation. Collisions at the back of the object could still be used to (softly) constrain the selection of the right pose/shape.\n- The applicability of the approach in a downstream-task is not shown. At this accuracy, I could only imagine motion planning, but then you could also make the planner react to collisions and adapt its plan, directly. Or by just moving the camera / having two cameras to remove ambiguities.\n- It is unclear how much CLASP helps the (often better) VAE-GAN results, I can only observe a gain in the simulation results of cheezeit (shallow) over number of contacts", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am leaning towards weak accept, mainly, because the paper and video are well crafted, but I could also understand if other reviewers lean towards rejection because the practicality is questionable. The method of iteratively refining latent codes is not new as also noted in the paper [30]. The authors try to use even less information to locate contact points (just collision information). I believe that to make the problem tractable and to achieve the accuracy needed in robotic applications we not only need better shape completion methods but also touch-sensitive skin. Trying to circumvent this via constraint optimization seems a bit hopeless but the general method might become interesting when these components are available. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "As mentioned before, the paper is well written. Some of the general mathematical descriptions (e.g. Section 3) are not really adding anything to the text, but this is a matter of taste.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "rJYoPBWJ1an", "reviewer": "Reviewer_wjGX", "summary": "The paper presents Constrained Latent Shape Projection (CLASP), a method for refining object shape (originally inferred from vision) with information from robot contact/tactile information. The overall framework involves a sensor fusion which is based on a particle filter, and using CLASP --which is a projection of the particles/samples onto contact constraints in the latent space-- as the measurement update. The projection technique itself is quite interesting: it's similar to the training process of the neural network (via backpropagation algorithm), but instead of optimizing the weights, it optimizes the inputs. In the experimental section, ablation studies as well as comparison with other method such as VAE-GAN are performed on both simulation and a real robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) The constraint latent space projection technique is quite interesting: it's similar to the training process of a neural network (via backpropagation algorithm), but instead of optimizing the weights, it optimizes the inputs.\n(2) In the experimental section, ablation studies as well as comparison with other method such as VAE-GAN are performed on both simulation and a real robot.\n(3) Source code is provided in the supplementary materials.\n\nWeaknesses:\n(1) A missing reference: S. Luo, W. Mou, K. Althoefer and H. Liu, \"Localizing the object contact through matching tactile features with visual map,\" 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 3903-3908, doi: 10.1109/ICRA.2015.7139743.\n(2) (a) Figure 2 is not very clear: what are the pink and blue rectangles in between the \"sample\" arrow and the \"f_{dec}\" trapezoid? Maybe worth checking if other readers understand this figure well, including each components drawn in it. Does each component of the figure provide a useful information?\n(b) Figure 1: bottom right figure (third row, vision + all contacts) is missing?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the technique presented in the paper is interesting, the way of communicating it to the reader (e.g. via the figures) may need to be improved.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Please see the points mentioned in the \"Weaknesses\" section above as well as a few additional suggestions below, and please try to address them. Thanks!\n\nAdditional suggestion(s):\n(1) In the experimental results/video, it would be nicer if the authors could highlight regions of the point clouds that were cut/chamfered due to the additional contact information, to get a better sense/reasoning of how the trained model behaves."}}, {"review_id": "JkFYq1NIOh_", "reviewer": "Reviewer_82yA", "summary": "The authors want to learn object shape from sensors on the robot. As a starting point, they apply a baseline method that uses solely RGBD data, using the prior work of Plausible Shape Sampling Network (PSSNet). This prior work first converts the RGBD images into voxel grids that are known-free and known-occupied. The PSSNet is then a VAE that encodes + decodes into a latent space. Latent vectors $\\psi$ can then be sampled, decoded into occupancy probability estimates, and aggregated into object shapes.\n\nThis work aims to augment PSSNet with robot contact information. The initial belief of objects is represented as a particle filter of several samplings $\\psi$ from the prior. As the robot moves in the environment, the contact sensors lead to contact constraints (either from free space or detected contacts). Given an already-trained PSSNet, the proposed method (CLASP) optimizes over $\\psi$ to get new latents that are consistent with the contact constraints. The loss includes a term that encourages $\\psi$ to stay close to the prior. Thus over the course of interaction, the potential object shapes become more constrained. This is then compared against other ways of integrating the contact information (such as rejection sampling).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not too familiar with shape completion, but the method seems reasonable. I did not understand what the OOD baseline meant - it is clear that RGBD data is augmented with contact information in some way, but it's not clear what this means. My interpretation was that contact information was propagated to the PSSNet in some way, and predictions from PSSNet were used directly, rather than doing a live constraint update like CLASP does, but if this is how that baseline works, the details are fairly sparse.\n\nTo me the main weakness is on the baselines. The main baseline used is rejection sampling, which the authors note is fairly competitive until they hit a point where the rejection sampler returns 0 valid object configurations. Given that rejection sampling was fairly effective, this makes me wonder how well the method does if the rejection sampling selects the object that violates the *minimum* number of constraints rather than requiring 0 constraint violation. It seems unfair to exclude comparisons to rejection sampling at all in the higher # of contacts setting. I am also curious about more complex MCMC algorithms in general for baselines.\n\nThe other main weakness of the method is that it is fairly reliant on the shape prior. As the authors noted, when they tried using a VAE-GAN for the shape prior, they found that average samples were more accurate, but samples were also much less diverse, and this harmed the contact info update. Thus, the effectiveness of CLASP seems tightly coupled to properties of the learned encoder / decoder...but this isn't really the focus of the paper, the paper is focusing on how to augment the shape prior, so I am not considering this for the review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method is very reasonable but I am unsure of its effectiveness.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I would appreciate stronger comparisons to simple methods of using contact info to constrain the prior and would be willing to improve the rating if this were done.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an extension to the PSSNet diverse object shape reconstruction work to incorporate reconstruction from contact sensing in addition to vision.

In the updated version the authors provide a comprehensive set of baseline comparisons and do a better job of situating it in the broader reconstruction literature.

As such I think this would make a nice addition to the conference.

I would recommend the authors do another pass through the comments of the reviewers to make sure everything is correctly addressed. I also think I suggested the wrong reference from Lu in my previous meta-review, as I intended the authors to include a discussion and reference of their reconstruction work ("Learning Continuous 3D Reconstructions for Geometrically Aware Grasping" https://ieeexplore.ieee.org/abstract/document/9196981)
Decision: Accept (Poster)