VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects
Authors: Youngsun Wi
Andy Zeng
Pete Florence
Nima Fazeli
Venue: CoRL 2022 Poster (2022)
Keywords: Deformable Object Manipulation
Multimodal Representation Learning

Abstract: Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation (VIRDO) via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo-tactile feedback, and (ii) generalization to unseen objects and contact formations. 

VIRDO++: Real- Visuo-Tactile Dynamics and Perception of Deformable Objects
Youngsun Wi
Andy Zeng²
Pete Florence Nima Fazeli
Robotics Department, University of Michigan RRobotics at Google
(yswi, nfz) afz)@umiz ch. edu (andyzeng, peteflorence @google.......oooooooo coms https: http:://wwwwmmmt. wwwntlain...........mmm.mmmm.mae com/virdopp
Abstract: Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present novel approach for, and real-world demonstration of, multimodal visuo- tactile estemmsstiotion and dynamics prediction for deformable objects. Our ap- proach, VIRDO++, builds recent progress in multimodal neural implicit rep- resentations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief distribution of deformation within a trajectory, and (ii) enables practical real-world application by removing the need for contact patches. In the context of two real-world robotic tasks, we show: (i) high-fidelity cross-modal state-estimation and prediction of deformable objects from partial visuo- -actile feedback, and (ii) generalization to unseen objects and contact formations.
Keywords: Deformable Object Manipulation, Multimodal Representation Learning
Raw RGB
Observation
State Estimation
Prediction
RGB w/ Prediction
<
Fig. 1: Given partial point cloud observations (left). and multimodal sensor measurements (end effector wrist reaction wrench and pose). VIRDO++ can accurately predict the 3D geometry (magenta) and reaction forces of deformable objects such as spatulas (top) and bike chains (bottom) conditioned on the next robot action under severe occlusions which can be artificial (red dotted box) natural due to obstacles (c.g., from other objects).
Introduction
Deformable objects are ubiquitous in our everyday lives the clothes we wear, the food we eat, and many of the tools we use are just few examples. As such, helpful robots of the future may benefit from the ability to master dexterous manipulation of deformable objects. the heart of this mastery is the interplay between object geometry and force transmission, perceived by vision and touch. Deformable object manipulation difficult due to complexity of this interplay (e.g., infinite dimensional state-spaces and nonlinearity) and the ensuing challenges in perception (e.g., partial observability and occlusion) and controls [2, 3, 4, 5]. Multimodal visuo-tactile representations can help address many of these challenges by exploiting mutual information and complementary cues.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Existing deformable object manipulation approaches typically use one modality (mostly vision) and rely on finite element/particle- based techniques [6. 7, 8, 9, 10. 11, 12, 13] or leverage deep learning for visual affordance/latent dynamics learning [14, 15, 16, 17, 18, 19]. The former methods typically rely on privileged knowledge (e.g., occluded or unknown boundary conditions) and stop at system identification, limiting their ability to refine the underlying physics model by learning from data. Latter methods lack (i) tactile feedback which is far more informative of contact than vision, and (ii) structured representations which can limit generalization and increase sample-complexity,
Here, we build on recent work [1] that provides a framework for visuo-tactile state estimation using multimodal implicit neural representations. This approach offers number of advantages including multimodal sensory fusion, direct integration with raw sensory feedback, and computational speed. However, [1] does not address (i) deformation predictions (dynamics) and (ii) is not directly appli- cable to real-world robotics settings. The first limitation is due to the lack of a dynamics module and an emphasis on static scenes. The second limitation is due to the need for the contact patch which are typically impractical to measure in real-world robotic settings.
The primary contribution of this work is (i) a novel approach to, and (ii) real-world demonstration of, performing multi-modal visuo-tactile state estimation and dynamics prediction for deformable objects. Fig. shows how our proposed approach uses partial views and tactile feedback for defor- mation prediction and state-estimation in real-world applications. Our contributions are:
1. Representation of deformable object dynamics conditioned on actions: Multimodal neural implicit representation with action-conditioned dynamics to predict future deformations. To demon- strate the utility of the dynamics model, we introduce a state estimation algorithm based on particle filtering to maintain belief distribution over object deformations.
2. Real-world demonstrations on two challenging tasks: Scraping with a spatula and chain ma- nipulation. We demonstrate both 3D geometry estimation in heavy occlusion and action-condiiioned deformattin//racctoo wrench prediction. This is enabled by introducing a novel contact latent vec- tor that eliminates the need for explicit contact patch information used in [1]. We use these tasks to demonstrate generalization to unseen objects and novel env environments.
2 Related Work
Deformable object modeling, Recent studics have attempted to address challenges in system identi-
fication and high computation costs in conventional continuum mechanics [6, 7, 8, 9, 10, 11, 12, 13]. To ease the burden of system identification, prior studies infer the physics models' parameters based on fighfiidliity physics engines [20, 21, 22] and simple foroe-deoormation relationship (e.g. Hooke's law) [23]. The inherent limitations of these approaches are (i) that performance is con- fined to the underlying physics model, often with strong assumptions on the objects (uniform den- sity/elasticity) and the force-deformation relationship (linearity), and (ii) relies on access to privi- leged information such as occluded or unknown boundary conditions. More recent approaches pro- pose computationally efficient clastic object modeling with minimum system identification using, for example, a potential energy propagation [24] and geometric motion estimates [25]. However, these methods suffer from balancing between model approximation and computation cost as well as maintaining seamless integration with robotic sensing modalities (RGB-D cameras and F/T sensors). In this study, we adopt data ddiven approach using neural networks [1, 26, 27]. Learning directly from observation, our approach does not require object parameters (Young's 's modulus, Poisson's 's ra- tio) or strong assumption of object composition. Even so, it can capture complex non-linear elastic behaviors and exhibit fast computation times due to the architecture of the neural networks.
3D geometric representation of deformable objects. Most recent studies have adopted discrete geometry representation for deformable objects with finite resolution such as meshes [7. 10, 28] and key points [15, 21], sometimes accompanied with graph neural networks [29, 27]. These approaches have been used largely for ID and 2D objects (rope, cloth, top view of cylindrical objects). Among them, structured representations (e.g.. Mesh or GNN) are advantageous at tracking connectivity, mainly used for deformable objects with large self-occlusions (e.g. folded cloth or knot) [30, 31]. On the other hand, we address general 3D geometry representations which are not confined to specific types of objects and can work with large occlusions (e.g., wok-scraping). Our work builds on dense implicit geometric representation which include signed distances fields [1, 26, 32), occupancy [33], volume density [34], and dense descriptors [35]. These approaches have several advantages over their discrete counterparts in dealing with large occlusion, high resolution surface representation
2

---

Deformable Object Dynamies
Geometry Representation
1.
fur
A2
Action A
Modio
Modse
$
P F
O
AF
Fig. 2: Overview. VIRDO++ is composed of an implicit signed- -egedddistance field representation of geometry (left), informed by deformation dynamics model (right), where dotted lines indicate hyper networks that decode embeddings into network weights.
[36], parameterization of 3D geometries [37, 38), and are useful for downstream tasks (e.g., contact location detection similar to Sec. 4.5). We extend these representations to real-world multimodal deformable object dynamics and perception by integrating touch and robot actions.
Methodology
Our approach, shown schematically in Fig. 2, is composed of a latent deformation dynamics model and an implicit dense geometric representation. Using Hidden Markov Models as an analogy, the former component plays the role of the hidden state transition, and the latter is the observation model. We expand on the latent deformation dynamics in Sec. 3.1 and discuss the implicit geometric representation in Sec. 3.2.
3.1 Deformable Object Dynamics
VIRDO++ represents deformable object states using latent object (or) and force (zz) codes. The object code condenses the undeformed shape information into a feature representation that can in- terpolate to unseen object variants. The force code is generated by the Force Module (F) which encodes boundary conditions as ž; = F(fr, Ct Pt) where f: R is the reaction wrench at the wrist, Pt E IR is robot end-effector pose, and Ct E RR is the contact latent vector, where lc is the dimension of the contact latent vector. The contact latent vector C₁ is learnable latent vector that can change over time whose primary role is to store contact information. However, it can also implicitly integrate information about the object's physical properties (e.g., stiffness) and historical information about the object's states and deformations. This embedding generalizes the original VIRDO [1] in terms of both (i) no longer needing known contact patch locations, unlocking the ca- pability to run VIRDO in the real-world and (ii) representing additional useful information beyond just the contact location as was the case in VIRDO. In more detail, VIRDO [1] uses ground truth contact patches encoded by a PointNet encoder [39] directly which required access to this privileged information. Here, the contact patch and encoder are replaced by the learnable contact latent vector.
The Action Module (A) predicts the next step's boundary conditions given the current (latent) state and robot action: A(a, 2t, ax) = fi+1, Ct+1 where f E F is the predicted reaction wrench the wrist, e the predicted contact latent vector, and a, E RR is the robot action (Carte- sian displacement). This predictive capability enables state-estimation and downstream tasks such as contact location detection crucial for deformable object manipulation. Though we did in- clude in this paper, the predictive ability of VIRDO++ also enables planning for deformable object manipulation. VIRDO lacked this functionality and could only infer the object geometry statically.
3.2 Geometry Representation
VIRDO++ decouples the 3D geometric representation of deformable objects into an undeformed (nominal) shape signed-distance field (SDF) and set of deformation fields, similar to [1. 38). The deformation field is 3D vector field that, when summed with the deformed object SDF results in the nominal shape SDF. More precisely, any query point E 1 in the robot wrist frame belonging to the deformed shape SDF can be mapped back to the corresponding point x' of the nominal shape SDF by applying the point-wise deformation field Ar
Ax x'.
Nominal Shape Representation: The Object Module O(æ') the parametric representation the nominal shape SDF the object shape with no external contacts corresponding to Ax O. Here, we use feedforward neural network to parameterize the SDF [37] and write (( V (ox) = a where is the signed-distance at query point x' and Vi (ax is a hyper-network that predicts the weights
3

---

of O conditioned on the object code a. The object code and hyper-network weights are learned end-to-end an auto-decoder approach [37]. We simplify our notation to O(z') for the remainder.
Deformation Field Representation: The generalized geometric representation of a deformable object is given by SDF()) = Os (a: (( Va(z,, o+)) s. Here, the Deformation Module ((x (lz/Vale (Z,, o4)) Aæ produces object's deformation field given force and object code pairs. Similar to the Object Module, Deformation Module's weights are given by the hyper-network da
3.3 Training and Loss Formulation
The training dataset for VIRDO++ is set of trajectories S = (T1, T2, J where each tra- jectory Tj = {(Po, fa, Po, ao), (Pi: PI, tion and action tuples. Here Pj is a partial point cloud. Our first step is to pretrain a nominal shape representation (i.e., hyper-network Vo and object code Q). The loss for training m nomi- nal shapes is Cnominal = Ledf + Ci + Xehhyper as in [1], where Llatent = - l| |
Pi,ai) ai),..... (pi, fk, Pk, ag)} has k sequential observa-
Liyper = Di=1 ( 112, and:
Ladf
(clam((O(x))). - (aamp(s",6]| + (a-
DE
n" the ground truth normal vector, parameter that clips SDF predictions, and s* is the ground truth signed distance.
After the pretraining, we train the rest of the modules using the three losses Eseo eored and gre summed over fixed horizon (w) while freezing V and the object code a:
Liot Σ
(C9 >a NART
The geometry representation loss is defined as:
_0
Σ
| ( + A CD(P De
(P),
(1- 0 (x) nx)),n')
correspondence
#E
minimum correction
normal aligning
(2)
A
Clip(Ow..r (z (z).
-)cipp((
signed distance regression
where x' = x Dv, (x), is the 3D querying space, Pu C S2 is the on-surface region, P {plp Φo} is an unordered set of the on-surface points, and P the ground truth nominal shape point cloud. Next, we add reaction wrench prediction loss as Lpred = - The prediction loss for c++1 is implicitly handled by Eq. 2 because the contact latent vector prediction is recursively used as next step's input during training. In contrast, we do replace f(+1 with real measurement in Eq. 2. Finally, regularization losses L,e9 Ag /g||z: la + A1v Aaul/cel/2 + A1rll AI aala 22). which regularize the force codes, contact latent vector, and the weights of v.a (x) respectively.
3.4 Inference and State-estimation Algorithm
To maintain a belief distribution over the object deformation, we develop a particle fister-based inference algorithm, Alg. 1. Each particle represents contact latent vector which can be used to reconstruct the full 3D object geometry. The Refine step updates the particle set by performing gradient descent on ming, L (P, Č,, ft. with the visible object point cloud. The particles then weighted according to their wrench error prediction with Weight-Functice unf exp(->(fi ft)) and resampled with probability proportional to this weight vector. Here, ft the reaction wrench prediction of the 7th particle and ft a ground truth measurement. Finally, the particles are propagated through the Force and Action Modules, a small amount of uncertainties applied ~ N 0.01) to the particles and the algorithm is repeated. (see algorithm variations in Appendix Sec. A.2.2)
defined as

---

Algorithm Particle Filter based Inference Algorithm
1: procedure
2: Co + ( "b.-...6--- ~ N 0.01)
> Init Particles
3:
for each scan in Trajectory Length do
4:
C, + Refine (,,, C₁ ft, Pt)
w, - Weight-Function ft. ft 1
5:
6:
C, + Re Re-sampling(w)) C₁)
7:
Ze f Force-Module( C₁ fú, Pt,
8:
{ fn @.+1). (ftt1 (f)... (f (G+ f- Action-Module (Z,, a, at)
9:
t ++1
10:
end for end procedure
11:
4
Experiments
For real robot experiments, we investigate two classes of objects: spatulas and bike chains. These objects exhibit different deformation behaviors: (i) spatulas are representative of bendable elastic objects that retain their shape when external forces are removed, whereas (ii) bike chains (with the 2 end points controlled by a bi-manual manipulator) can undergo significantly larger deforma- tions (e.g., s,, from gravity) requiring algorithms that can reason over free-space while estimating in- contact shapes and forces. Spatula deformations are generally local and experience significant reac- tion wrenches, while chain deformations are much larger but with much smaller reaction wrenches.
4.1 Spatula Manipulation Dataset
Train and test objects are shown in Fig. 3. The training set consists of spat- ulas, each with 672 samples (N = 28, k 24), using 9:1 train/test split (25 train and test trajectories) for representation learning. The test dataset con- sists of 2 held-out spatulas, each with 72 samples (N = 3. k 24), and is used to evaluate zero-shot transfer and not used for representation learning.
At the beginning of each trajectory (t 0), the end-effector moves to ran- dom world Cartesian position in a bounding box size (0.1 x 0.24 x 0.06) meters and the tool is brought into contact with the table. Within tra- jectory, the tool maintains contact while sampling a random action from (ar, U: 2, r.P.)) y) = (0.02, 0.02, 0.01, 0.06, 0.025, 0.04) where translations are in meters and rotations in radians. Wrench measurements are averaged over 0.05 [s] and the point cloud is registered to the wrist frame. The method only requires a full point cloud from the nominal geometry of the object. This is collected by hold- ing the object in front of the camera and rotating. Deformed object geometries are collected from a single camera viewing the front side of the spatulas without occlusions and are all partial. P is then the result of segmenting the object pointcloud and normalizing the scale into a bounding box of size 2 x 2 x 2/ centered around (0, 0, 0). The test dataset point clouds P contain occlusions at the bottom 0.15 spatula-heigh as shown in the top observation panel in Fig. For hardware, we use Franka Emika's 's Panda, wrist mounted ATI Gamma F/T sensor, and Photoneo PhoXi 3D L..
Fig. 3: Train and test objects differ in terms of (i) geometry, and (ii) how they react un- der wrench.
4.2 State Estimation and Predictions
Here, we evaluate the deformation prediction (Sec. 3.1) and state-estimation (Alg. I) accu- racy of VIRDO++ on training and test tra- jectories. Fig. 4 shows an example of state- estimation for test trajectory where the ground truth (black) and reconstruction (magenta) are overlaid. We note the high state-estimation accuracy indicated by the overlay agreement, where the average distance between the mea- sured points and reconstructions is within 0.03 [m] of normalized scale, just 1% of the ob- ject length.
CD (x (x10*)
Spatula w/ table
Bike Chain
Train Est.
0.777 (0.215)
0.708 (0.331) 0.762 (0.224)
Train Pred.
0.830 (0.330)
Test Est.
0.934 (0.210)
0.760 (0.234)
Test Pred.
1.041 (0.348)
1.247 (0.391)
Table. I: VIRDO++ can accurately estimate object ge- ometry deformations under significant occlusions, sured terms mean Chamfer distances (CD) (std.) [m²) scaled as (x10*) across 5600 on-surface points between predicted and ground truth reconstructions.
5

---

The quantitative results, Tab. I, show VIRDO++ is able to faithfully predict and estimate object deformations and reaction forces despite significant occlusions. Force and torque norm errors are less than 0.567 [N], 0. [Nm] (train) and 1.130 [N], 0.135 [Nm] (test) respectively (see Appendix Sec. A.3.3 and Sec. A.3.4 for details).
Our ablation study, Tab. 3, helps further anchor the scale of these state-estimation and dynamics prediction errors. On state estimation, for example, our full VIRDO++ method achieves Chamfer distance (x x10") of normalized geometry as 0.93 [m=]] The baseline model without a novel contact latent vector input has 27% increased chamfer distance error as 1.18 [m²*] for geometry estimation. Without the baseline does not reason about contact during inference, resulting in larger errors during -steemsstimtiin. Further if we assume the object is rigid and reduce our state estimation to a "rigid pose estimation" pipeline, the Chamfer distance x 10") increases considerably to 16.19 [m²].
The geometry predictions slightly less accurate than state-estimation because the geometry pre- dictions are computed by the learned dynamics using previous observation and action, while state- estimations use the current measurement to update the prediction. Despite this, the prediction Cham- fer distances are within 14% of the state estimation results. Moreover, the train and test trajectories results show 25% difference in Chamfer distance. This interesting because the test trajectories are not only from unseen tool configurations but also with occlusions (see Dataset). This suggests that learned dynamies of VIRDO++ generalize effectively to unseen contact formations with occlusions. State Estimation
G.T.
G.T.
1=0
1=1
t= 2
1=3
t
t=
Fig. 4: Even under occlusions (generated by cropping out data within the red dotted box), our approach can accurately infer the how the geometry of test spatula deforms over time (1= 0 to 6), indicated by high overlap agreement between the prediction (magenta) overlaid the ground truth (black).
4.3 Evaluation on Downstream Manipulation Task Extrinsic Contact Detection
Central to compliant tool manipulation is estimating and controlling the contact between the tool and environment. For example, when scraping a wok with a spatula, the robot must reason over the contact formation (c.g., point, line, or patch) to ensure the spatula is scraping properly at the desired region on the wok. In this section, we evaluate VIRDO++ ability to estimate extrinsic contact features; specifically, the contact line created as the spatula is brought into contact with the table. To this end, we compute the signed distance values of the tables surface (= 0) to compute contact cluster. This requires only a single feed-forward path where the input is an N x 3 matrix, where N the number of query points. From the contact cluster, we extract a contact line by selecting the two points furthest apart similar to [40, 41, 42]. The evaluation metrics are the average Euclidean distance 3 - 14, 7 ll. where I; is the detected contact lines, 17 is the ground truth contact lines, and i = 0, refers to right and left ends of each contact line respectively. Ground truth contact lines were obtained from ground truth pointcloud without occlusions by applying contact mask This contact feature estimation task is particularly interesting is the inverse problem of [1]: estimating [1]'s privileged input (i.c., contact location) by inferring Cy.
As shown in Tab. 2 and Fig. 5, VIRDO++ can accurately detect contact locations with less than 8 mm error on flat table for the current and the following time step regardless of occlusions at the tip. As in Sec.4.2, test trajectories from 4 spatulas with 15% occlusion were used for the evaluation. Tab.2 shows that the average contact line detection errors from all 4 spatulas are less than [mm] during estimation and are less than 8 [mm] for predictions in Euclidean error. Similar to the result in Sec. 4.2, estimations are 1 [mm] more precise than predictions because they are optimized using the sensor measurements. Considering that the total length of the spatulas are about 200 [mm], this

---

Contact line estimation row)
7
t=1
t-2
t=3
t=4
Acdisn Modile
Riweaan FReine (Ag
Contact line prediction conditioned on action (bottom row)
Fig. 5: VIRDO++ detects the line of contact throughout an alternating sequence of "estimation - prediction + estimation - ...". Bayesian filtering increase the accuracy of contact location detection the red lines (detected contact line) gets closer the dotted blue lines (ground truth) following the green arrow, In each figure, yellow plane: table location, magenta spatula: reconstructed geometry from VIRDO++ red line: detected contact line, blue: ground truth contact line. All axis are [m] scale world coordinate.
L2 Error [mm]
VIRDO+
w/o C₁
Rigid body asmp.
Contact Est.
6.647 (4.806) 9.872 (5. (5.725)
21.203 (7.031)
Contact Pred.
7.198 (4.439)
9.356 55.788
22.572 (6.990)
Table. 2: Having latent embedding as input, VIRDO++ can decrease about 33 % of error for estimations and 23 % for predictions. Morcover, rigid body assumption is worst case scenario producing erro.
is less than 4% of error in total length. Estimating extrinsic contacts is a challenging task for rigid
object [43] and further complicated by object compliance; however, our approach proves effective due to its high-quality 3D reconstructions (details of error distribution in Appendix Fig. 12). Tab. 2 also quantifies performance against ablations (without e and Rigid-body assumption) illustrating the importance of the contact embedding and reasoning over compliance.
4.4 Generalizations Tasks
This section demonstrates VIRDO++ 's generalization ability unseen objects and their dynamics. For each unseen object, the generalization dataset contains nominal point cloud and trajectories, each with 24 transitions with occlusion. Here, we directly apply the same pre-trained model from Sec. 4.1 to the generalization test dataset.
4.4.1 Object code inference for Unseen Objects
VIRDO++ is able to represent nominal geometries unseen dur- ing training within distribution. Fig. 6 shows reconstructions of both seen and unseen object in Fig. 3 using single pretrained Object Module. For the unseen objects, we start from a randomly initialized object code A00 0.01) and update the code by gradient descent with Adam. We use the same loss function from Eq. with pre-trained VIRDO+ while freezing the weights of the network. The first row of Tab. 3 shows the Chamfer distance error of the unseen objects alongside training objects, indicat- ing the relatively small drop in performance. When comparing Fig. and the ground truth in Fig. 3, we note that the test reconstructions are close to the ground
Fig. 6: VIRDO+ reconstruct train and test objects with high fi-
delity (same order as Fig. 3).
truth aside from some fine edge details.
7

---

G.T.
Geometry Prediction
UELLE
UEELE!
1=0
1=2
5
Fig. 7: VIRDO++ accurately estimate the bike chain geometry (predictions overlaid in magenta) under occlusions due to objects at the front. Note that the chain intermittently makes contact and rests on the table, which is correctly predicted by VIRDO++ despite significant occlusion of where the chain makes contact.
4.4.2 Zero Shot Dynamics Prediction and State-estimation Generalization
With the object codes obtained in Sec. 4.4.1, VIRDO++ is able to predict the unseen object's dy- namics based their nominal shape. Here, we evaluate both prediction and state-estimation per- formance as in Sec. 4.2. We note that the observations of the unseen objects include occlusions as discussed in Dataset. Tab. 3 shows comparable performance between training and test objects with a small drop in accuracy, highlighting our model's generalization capability. Sec. A.3.6 in Appendix provides intuition for the Chamfer distance values. Further, we evaluate VIRDO++ 's ability to per- form high-fidelity geometry prediction and estimation even in novel environment (Wok scraping) without 3D model. Details can be found in Appendix Sec. A.4.
CD (x10*)
VIRDO++
w/o cz.
Rigid Asm.
Test Obj.
Test Obj.
Geo. Nom.
0.65 (0.08)
1.08
1.29
Gen. - S.F.
0.93 (0.21)
1.18 (0.35)
16.66 (11.51)
1.39 (0.20)
1,56 (0.29)
Gen. - Pred.
1.04 (0.35)
1.09 (0.25)
15.16 (10.57)
1.54 (0.45)
2.40 (1.08)
Wr. Pred.
1.14 (1.05)
1.04 (1.01)
1.63 (0.89)
1.50 (1.00)
2.14 (1.75)
Table. 3: (Column 1-3) Ablation study using the train object's unseen trajectories. Numbers indicate mean
(std.) of each error distribution. Chamfer distance used 5600 points and multiplied by 10 (Column 4-5)
Generalization results to unseen objects using the same pretrained VIRDO++ Geo. = Geometry, Nom. Nominal, S.E. State Estimation, Pred. Prediction, Wr. Wrench calculated as []f: frll.
4.5 Bike Chain Application
In this experiment, the chain ends are grasped by two arms. One end of the chain is moved ran- domly through the vertices of an equally-spaced x 6 x 6 grid fitting inside of 0.1 0.18 x 0.2 meter bounding box. The visited vertices of this grid make up the training dataset. The test dataset generated by sampling 20 random positions uniformly from the bounding box. The end-effector orientations were fixed; therefore, Pt, ar E IR3 As shown in Fig. 7, VIRDO+ is able to pre- dict/estimate both free-space and in-contact chain geometry. The in-contact phase occurs when the arm moves sufficiently far down such that the chain lays on the table. We report the quantitative results of geometry estimation and dynamics prediction in Tab. and Appendix Sec. A.3.3. The empirical results are comparable to spatula experiments despite significantly larger shape variations. Details on wrench prediction errors in Appendix Sec. A.3.4.
5 Limitations
Our proposed approach has a number of limitations: 1) Non-rigid grasps: VIRDO+ assumes that the gripper rigidly holding the object. When it comes to non-rigid grasping, we need estimate relative motion of the objects w.r.t the gripper and the force transmission is now a function of this relative motion. One potential solution is use collocated tactile sensor feedback [44, 45, 46] with F/T sensing to reason over reaction forces conditioned on the grasp. 2) Inertial effects: our ap- proach assumes quasi-static manipulation (negligible accelerations). During fast motions, the tools' inertial effects may also impact deformations and wrench measurements. When applicable, we sug- gest supplementing VIRDO++ 's input with end-effector accelerations. 3) Generalization to novel environments: While we provide an example of environment generalization (Appendix Sec. A.4), the performance lower than our other demonstrated applications. VIRDO++ may benefit from a 3D model of the environment to integrate inter-penetrate losses [23] or collision checkers [47].
8

---

Acknowledgments
This work was in part supported by Robotics at Google under the Google Faculty Research Award 2021 and the National Science Foundation (NSF) grant NRI-2220876. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Bibliography
[1] Y. Wi. P. Florence, A. Zeng, and N. Fazeli. VIRDO: visio-tactile implicit representations deformable objects. CoRR, abs/2202.00868,, 2022. URL https://arxxa org/abs /2002 00868.
[2] Yin, A. Varava, D. Kragic. Modeling, learning, perception, and control methods deformable object manipulation. Science Robotics, 6(54), 2021.
[3] Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar. Robotic manipulation and sensing formable objects in domestic and industrial applications: a survey. The International Journal of Robotics Research, 37(7):688-715 2018.
[4] V. E. Arriola-Rios, P. Guler, F. Ficuciello, D. Kragic, B. Siciliano, and J. L. Wyatt. Modeling of de- formable objects for robotic manipulation: A tutorial and review. Frontiers in Robotics and A1, 7:82, 2020.
[5] Y. C. Hou, K. S. M. Sahari, and D. N. T. How. A review modeling of flexible deformable ob- ject for dexterous robotic manipulation. International Journal of Advanced Robotic Systems, 16(3): 17298814 419888449944899, 2019.
A. Essa, Sclaroff, and Pentland. unified approach for physical and geometrie modeling for graphies and animation. Computer Graphics Forum, volume 11, pages 129-138. Wiley Online Library, 1992.
[7] F. Ficuciello, A. Migliozzi, E. Coevoet, A. Petit, and Duriez. Fem-based deformation control for dexterous manipulation of 3d soft objects. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4007-4013. IEEE, 2018.
[8] B. Frank, C. Stachniss, R. Schmedding, M. Teschner, and W. Burgard. Learning object deformation models for robot motion planning. Robotics and Autonomous Systems, 62(8):1553-1174 2014.
[9] M. Müller and M. H. Gross. Interactive virtual materials. In Graphics interface, volume 2004, pages 239-246 2004.
[10] A. Sengupta, A. Krupa, and E. Marchand. Tracking of non-rigid objects using rgb-d camera. In 2019 IEEE International Conference on Systems, Man and Cyberneties (SMC), pages 3310-33 3317. IEEE, 2019.
[11] A. M. Howard and G. A. Bekey. Intelligent learning for deformable object manipulation. Autonomous Robots, 91):551-58, 2000.
[12] A. Numberrger, A. Radetzky, and R. Kruse. A problem specific recurrent neural network for the descrip- tion and simulation of dynamic spring models. In 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. 98CH36227), vol- ume 1, pages 468 473. IEEE, 1998.
[13] D. L. Tonnesen. Dynamically coupled particle systems for geometric modeling, reconstruction, and ani- mation. Citeseer, 2000.
[14] B. Thach, Y. Cho, A. Kuntz, and T. Hermans. Learning visual shape control of novel 3d deformable objects from partial-view point clouds. arXiv preprint 1rXiv:211000468) 2021.
[15] B. Thach, A. Kuntz, and T. Hermans. Deformernet: A deep learning approach to 3d deformable object manipulation. arXiv preprint arXiv:2107.088k6 2021.
[16] T. Weng, S. M. Bajracharya, Y. Wang, K. Agrawal, and D. Held, Fabricflownet: Bimanual cloth manipu- lation with a flow-based policy. In Conference on Robot Learning, pages 192-202. PMLR, 2022.
[17] X. Lin, Z. Huang. Y. Li. J. B. Tenenbaum, D. Held, and C. Gan. Diffskill:: Skill abstraction from dif- ferentiable physics for deformable object manipulations with tools. arXiv preprint 17275 2022.

---

[18] D. Seita, N. Jamali, M. Laskey. A. K. Tanwani, R. Berenstein, P. Baskaran. S. Iba, J. Canny, and K. Goldberg. Deep transfer learning of pick points on fabric for robot bed-making. arXiv preprint arXiv: 0e.11880900980 0900009810..... 2018.
[19] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Visuospatial foresight for physical sequential fabric manipulation. Autonomous Robots, pages 1-25, 2021.
[20] P. Sundaresan, R. Antonova, and J. Bohg. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects, arXiv preprint arXiv:2204.0313 2022.
[21] R. Antonova, J. Yang, P. Sundaresan, D. Fox, Ramos, and J. Bohg. A bayesian treatment of real-to- sim for deformable object manipulation. IEEE Robotics and Automation Letters, 7(3):5819-588 2022. doi: 10.1 1e.110.11199R... 200./.....22..311555
[22] Y. Narang, K. Storey, I. Akinola, M. Macklin, P. Reist, I., Wawrzyniak, Y. Guo, A. Moravanszky, G. State, M. Lu, ct al. Factory: Fast contact for robotic assembly, arXiv preprint 2xXx:2205.03332, 2022.
[23] A. Petit, S. Cotin, Lippiello, and B. Siciliano. Capturing deformations interacting non-rigid objects using rgb-d data. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 491 197. IEEE, 2018.
[24] J. Zhang, Y. Zhong, J. Smith, and C. Gu. Neural dynamics-based poisson propagation for deformable modelling. Neural Computing and Applications, 31(2):1991-1101 2019.
[25] Y. Wang, D. McConachie, and D. Berenson. Tracking partially-occluded deformable objects while enfore- ing geometric constraints. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14199-14205... IEEE, 2021.
[26] P. Palafox, A. Božič, J. Thies, M. Nießner, and A. Dai. Npms: Neural parametric models for 3d de- formable shapes. In Proceedings of the IEEE/CVF International InterrationalCoffeennneee on Computer Vision (ICCV), pages 12695- 12705, October 2021.
[27] X. Ma. D. Hsu, and W. S. Lee. Learning latent graph dynamics for deformable object manipulation. arXiv preprint arXiv:2104. 12149, 2021.
[28] 1. Santesteban. N. Thuerey, M. A. Otaduy, and D. Casas. -upervidd collision handling via generative 3d garment models for virtual try-on. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11763-11773, June 2021.
[29] H. Shi, H. Xu, Z. Huang, Y. Li, and Wu. Robocraft: Learning to see, simulate, and shape Masto-plastic objects with graph networks. arXiv preprint wrxiv:2205.02900 2022.
[30] Z. Huang, X. Lin, and D. Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation. arxiv preprint arXiv:2206.0288 2022.
[31] C. Chi and S. Song. Garmentnets: Category-level pose estimation for garments via canonical space shape completion. In The IEFE International Conference on Computer Vision (ICCV), 2021.
[32] J. Ortiz, A. Clegg, J. Dong, E. Sucar, D. Novotny, M. Zollhoefer, and M. Mukadam. isdf: Real-time neural signed distance fields for robot perception. arXiv preprint arXiv:2204.022%% 2022.
[33] M. Niemeyer, L.. Mescheder, M. Oechsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In International Conference on Computer Vision, Oct. 2019.
[34] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021.
[35] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Roboticy and Automation (ICRA), pages 9411- 1- 9418. IEEE, 2020.
[36] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[37] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
10

---

[38] Y. Deng, J. Yang, and X. Tong. Deformed implicit field: Modeling 3d shapes with learned dense corre- spondence. CoRR, abs/201 1.36600 2020. URL https://arxivy http:///aaariv..rgaaaa////sssss/ org/abs/2011 13650.
[39] C. R. Qi, H. Su, K. Mo, and I.. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings the IEEE conference computer vision and pattern recognition, pages 652-660. 2017.
[40] D. Ma, Dong, and A. Rodriguez. Extrinsic contact sensing with relative-motion tracking from dis- tributed tactile measurements. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11262 11268, 2021. doi: 100.0..000///RA4450022020200 1961188.
[41] Q. Li, SchAVarmann, Haschke, and H. Ritter. control framework for tactile servoing. In Proceed- ings of Robotics: Science and Systems, Berlin, Germany, June 2013. doi:10. 10..1560///S.00111111 201...X0....
[42] Van der Merwe, D. Berenson, and Fazeli. Learning the dynamics of compliant eol-ennvron interaction for visuo-tactile contact servoing, 2022.
[43] A. Sipos and Fazeli. Simultaneous contact location and object pose estimation using proprioceptive tactile feedback. arXiv preprint 2rXiv:2206.01245, 2022.
[44] Taylor, S. Dong. and A. Rodriguez. Gelslim3. O: High-resolution measurement of shape, force and slip in compact sactile-sensing finger. arXiv preprint w:Xiv:2103.12269 2021.
[45] Yin, G. M. Campbell, Pikul, and M. Yim. Multimodal proximity and visuotactile sensing with a selectively transmissive soft membrane. In 2022 IEEE 5th International Conference on Soft Robotics (RoboSoft), pages 802 808. IEEE, 2022.
[46] N. Kuppuswamy, A. Alspach, A. Uttamchandani, S. Creasey, T. Ikeda, and R. Tedrake. Soft-bubble grippers for robust and perceptive manipulation. arXiv preprint arXiv:2004.0300 03691, 2020.
[47] P. Mitrano, D. McConachie, and D. Berenson. Learning where to trust unreliable models in an unstruc- tured world for deformable object manipulation. Science Robotics, 6(54):eabd8170 2021.
[48] H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun. Grnet: Gridding residual network for dense point cloud completion. In European Conference Computer Vision, pages 365-381. Springer, 2020.
[49] J. Schauer and A. Nüchter. Collision detection between point clouds using an efficient kd tree implemen- tation. Advanced Engineering Informatics, 29(3):440-458 2015.
[50] J. Klein and G. Zachmann. Point cloud collision detection. In Computer Graphics Forum, volume 23, pages 567-576. Wiley Online Library, 2004.
[51] R. Weller. A brief overview of collision detection. New Geometric Data Structures for Collision Detection and Haptics, pages 9-46, 2013.
[52] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint 200.:::00000.03033 2020.


Reviews:
[{"review_id": "peCHQwkPusy", "reviewer": "Reviewer_y9NZ", "summary": "This paper builds on top of the prior work on VIRDO and proposes a framework for predicting deformation of deformable objects conditioned on the multi-modal observation such as partial point cloud and force beedback. Compared to prior work of VIRDO, the new appraoch removes the assumption on known contact patches, learns the deformation dynamics and develops a gradient-based state estimation method able to handle occlusion. The proposed method is demonstrated on real manipulation trajectories.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The paper makes a clear contribution compared to prior works on VIRDO. I appreciate the added component of the deformation dynamics model, which is essentially a latent dynamics model that avoides the need of known contact information. \n2. The method is sufficiently demonstrated on real-world trajectories, showing the effectiveness of state estimation and dynamics prediction.\n3. The paper is well presented and the supplementary video is very helpful.\n\n### Weakness\nI think the paper is a clear accept and the below weaknessnes are mainly suggestions to further improve the paper. \n\n1. While the paper claims that the contect embedding c predicts the contact, it is actually not directly supervised the same as the prediction of reaction wrench. The contact embedding is only trained implicitly through other losses. As mentioned in the paper, contact embedding may also contain other information like the stiffness. It would be better to try reading out the contact information from c, or change the name of this component to something like a latent vector. \n\n2. The paper can make a better connection to prior works. There are prior works on dealing with occlusion for other deformable objects such as cloth [1,2,3], where self-occlusion can be an important issue. The current description in line 248-250 can be misleading. \n\n3. Does the method require full point cloud information during training? What forms of data are collected for training in the real world? Is there a multi-camera setup in place?\n\n4. Finally, in the beginning I thought the dynamics model is used for planning in the two real world tasks demonstrated but in fact, they are only prediction experiments. It would be better to clarify this in the paper.\n\n[1] Garmentnets:Category-level pose estimation for garments via canonical space shape completion\n\n[2] Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation\n\n[3] Learning visible connectivity dynamics for cloth smoothing\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper proposes an interesting method that overcomes some limitations of a prior work. The paper is well presetned with convincing experiments and thus I recommend acceptance. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see above. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "YSdx2VrZ8RW", "reviewer": "Reviewer_fX59", "summary": "The paper presents a visuo-tactile state estimation and dynamics model for deformable objects, built on the previous work VIRDO. This work improves VIRDO in two aspects: (1) it removes the need for priviledged information (contact patch) by estimating a contact embedding, which enables real world robotic application; (2) it learns a implicit dynamics model. Based on the this multimodal neural implicit representation, the authors introduce a state estimation algorithm by maintaining a belief distribution over object deformations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* It addresses the biggest limitation of VIRDO - cannot be directly applied to real-world robotics settings due to the need for priviledged information, which is a significant contribution.\n* It integrates an implicit dynamics model with VIRDO seamlessly, and show the advantages of proposed visuo-tactile representation with robotic application (state estimation), in real-world. The results look great and seem to be robust to occlusion, thanks to tactile information.\n\nWeaknesses:\n* The paper proposes an implicit dynamics model for deformable object, but doesn't compare it with an explicit counterpart. For example, one may extract a mesh from predicted SDF and learn an explicit dynamics model on the mesh. \n* Some of the design choices are not ablated. For example, why do you choose to use hypernetwork for deformation and object module? One alternative is to concatenate query point with the feature and feed them to a MLP (like DeepSDF)\n* It's not evaluated on downstream manipulation tasks. I'm not so sure about how to interpret the state estimation results. It will be nice if the author can evaluate the proposed method on a manipulation task, to give us a sense of how accurate the state estimation and dynamics model need to be to, for example, flip a steak with a spatula.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a visuo-tactile representation for deformable object that can be applied to real world. The proposed approach is not groundbreaking, but well executed. I believe this will be interesting for the wider robot learning community, especially deformable object manipulation. However, I have some concerns regarding the experiment section. I would be happy to update my rating if more results can be shown.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* The plots in Fig.5 should be annotated with the names of the quantities. For example, the first one should be \"Geometry prediction and state estimation\", and second one should be \"force and toque prediction\"."}}, {"review_id": "JKf5JY6AGWI", "reviewer": "Reviewer_8Dpk", "summary": "This work builds on recent progress in multimodal neural implicit representations for deformable object manipulation by proposing (1) an action-conditioned dynamics model to predict future deformations and (2) a state estimation algorithm to maintain a belief distribution over object deformations. The work performs two real-world tasks to demonstrate generalization to unseen objects and novel environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths :\n(1) the work enables real world application of deformable manipulation. \n(2) training, loss formulations, and algorithms are clearly presented\n\nweaknesses : \n(1) the work lacks originality, it heavily builds upon the predecessor work (VIDRO). It relies on the original VIDRO architecture and replaces (a) contact patches with contact embedding and (b) adding an additional dynamics model (to focus on non-static scenes). \n(2) real-world experiments aren't fully convincing. The paper shows interpolation on limited geometries which is insufficient to draw conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I read the paper thoroughly and used my best judgement in assessing the paper.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "For their current work, I'd suggest adding more experiments with newer geometries (eg: other objects besides spatula, bike chain), thorough interpolation to intermediate sizes / shapes (at least 3-5).\n\nThis work is highly incremental, and I am not convinced with the technical novelty of the work, so would like to see if authors considered any other approaches to model architecture itself.\n"}}, {"review_id": "3PcfHt3pAYt", "reviewer": "Reviewer_NDvN", "summary": "The paper proposes a deformation dynamic model of deformable objects and its complementary model of geometry estimation using multi-modal visuo-tactile information,, which is demonstrated in two real world tasks, spatula manipulation and bike chain application. ", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main weakness of the paper is the presentation of the paper is not well organized and clear and the wording of the paper makes the contents difficult to understand. The paper needs to be improved significantly to convey the search findings more effectively.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presentation of the paper is not well organized and clear and the wording of the paper makes the contents difficult to understand. Hence I recommend rejecting the paper.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. The section 4 on related work should be placed right after the introduction for the sake of understanding the relevant background.  \n2. In the paper, the authors should describe models or methods with technique details instead of using meaningless words conveying limited information. For example, in the abstract, the phrase \"a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information.\"  fails to tell the key contributions of the paper with strong supporting details.\n3. Some of the figures and tables should be formatted more properly. For example, Fig.2, 3 and Table 2.\n4. The conclusion on the generalization capability of the proposed model in the spatula manipulation task is not well supported by the results in Table 2a and Fig. 7. Some of the measuring scores double for the test object 2 compared to the training object."}}]

Author rebuttal:

Decision: Accept (poster)