Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper builds on top of the prior work on VIRDO and proposes a framework for predicting deformation of deformable objects conditioned on the multi-modal observation such as partial point cloud and force beedback. Compared to prior work of VIRDO, the new appraoch removes the assumption on known contact patches, learns the deformation dynamics and develops a gradient-based state estimation method able to handle occlusion. The proposed method is demonstrated on real manipulation trajectories.
strengths_and_weaknesses: ### Strength
1. The paper makes a clear contribution compared to prior works on VIRDO. I appreciate the added component of the deformation dynamics model, which is essentially a latent dynamics model that avoides the need of known contact information. 
2. The method is sufficiently demonstrated on real-world trajectories, showing the effectiveness of state estimation and dynamics prediction.
3. The paper is well presented and the supplementary video is very helpful.

### Weakness
I think the paper is a clear accept and the below weaknessnes are mainly suggestions to further improve the paper. 

1. While the paper claims that the contect embedding c predicts the contact, it is actually not directly supervised the same as the prediction of reaction wrench. The contact embedding is only trained implicitly through other losses. As mentioned in the paper, contact embedding may also contain other information like the stiffness. It would be better to try reading out the contact information from c, or change the name of this component to something like a latent vector. 

2. The paper can make a better connection to prior works. There are prior works on dealing with occlusion for other deformable objects such as cloth [1,2,3], where self-occlusion can be an important issue. The current description in line 248-250 can be misleading. 

3. Does the method require full point cloud information during training? What forms of data are collected for training in the real world? Is there a multi-camera setup in place?

4. Finally, in the beginning I thought the dynamics model is used for planning in the two real world tasks demonstrated but in fact, they are only prediction experiments. It would be better to clarify this in the paper.

[1] Garmentnets:Category-level pose estimation for garments via canonical space shape completion

[2] Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation

[3] Learning visible connectivity dynamics for cloth smoothing



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the paper proposes an interesting method that overcomes some limitations of a prior work. The paper is well presetned with convincing experiments and thus I recommend acceptance. 
issues: Please see above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a visuo-tactile state estimation and dynamics model for deformable objects, built on the previous work VIRDO. This work improves VIRDO in two aspects: (1) it removes the need for priviledged information (contact patch) by estimating a contact embedding, which enables real world robotic application; (2) it learns a implicit dynamics model. Based on the this multimodal neural implicit representation, the authors introduce a state estimation algorithm by maintaining a belief distribution over object deformations.
strengths_and_weaknesses: Strengths:
* It addresses the biggest limitation of VIRDO - cannot be directly applied to real-world robotics settings due to the need for priviledged information, which is a significant contribution.
* It integrates an implicit dynamics model with VIRDO seamlessly, and show the advantages of proposed visuo-tactile representation with robotic application (state estimation), in real-world. The results look great and seem to be robust to occlusion, thanks to tactile information.

Weaknesses:
* The paper proposes an implicit dynamics model for deformable object, but doesn't compare it with an explicit counterpart. For example, one may extract a mesh from predicted SDF and learn an explicit dynamics model on the mesh. 
* Some of the design choices are not ablated. For example, why do you choose to use hypernetwork for deformation and object module? One alternative is to concatenate query point with the feature and feed them to a MLP (like DeepSDF)
* It's not evaluated on downstream manipulation tasks. I'm not so sure about how to interpret the state estimation results. It will be nice if the author can evaluate the proposed method on a manipulation task, to give us a sense of how accurate the state estimation and dynamics model need to be to, for example, flip a steak with a spatula.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a visuo-tactile representation for deformable object that can be applied to real world. The proposed approach is not groundbreaking, but well executed. I believe this will be interesting for the wider robot learning community, especially deformable object manipulation. However, I have some concerns regarding the experiment section. I would be happy to update my rating if more results can be shown.
issues: * The plots in Fig.5 should be annotated with the names of the quantities. For example, the first one should be "Geometry prediction and state estimation", and second one should be "force and toque prediction".
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work builds on recent progress in multimodal neural implicit representations for deformable object manipulation by proposing (1) an action-conditioned dynamics model to predict future deformations and (2) a state estimation algorithm to maintain a belief distribution over object deformations. The work performs two real-world tasks to demonstrate generalization to unseen objects and novel environments.
strengths_and_weaknesses: strengths :
(1) the work enables real world application of deformable manipulation. 
(2) training, loss formulations, and algorithms are clearly presented

weaknesses : 
(1) the work lacks originality, it heavily builds upon the predecessor work (VIDRO). It relies on the original VIDRO architecture and replaces (a) contact patches with contact embedding and (b) adding an additional dynamics model (to focus on non-static scenes). 
(2) real-world experiments aren't fully convincing. The paper shows interpolation on limited geometries which is insufficient to draw conclusions.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I read the paper thoroughly and used my best judgement in assessing the paper.
issues: For their current work, I'd suggest adding more experiments with newer geometries (eg: other objects besides spatula, bike chain), thorough interpolation to intermediate sizes / shapes (at least 3-5).

This work is highly incremental, and I am not convinced with the technical novelty of the work, so would like to see if authors considered any other approaches to model architecture itself.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a deformation dynamic model of deformable objects and its complementary model of geometry estimation using multi-modal visuo-tactile information,, which is demonstrated in two real world tasks, spatula manipulation and bike chain application. 
strengths_and_weaknesses: The main weakness of the paper is the presentation of the paper is not well organized and clear and the wording of the paper makes the contents difficult to understand. The paper needs to be improved significantly to convey the search findings more effectively.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The presentation of the paper is not well organized and clear and the wording of the paper makes the contents difficult to understand. Hence I recommend rejecting the paper.
issues: 1. The section 4 on related work should be placed right after the introduction for the sake of understanding the relevant background.  
2. In the paper, the authors should describe models or methods with technique details instead of using meaningless words conveying limited information. For example, in the abstract, the phrase "a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables practical real-world application by removing the need for privileged contact information."  fails to tell the key contributions of the paper with strong supporting details.
3. Some of the figures and tables should be formatted more properly. For example, Fig.2, 3 and Table 2.
4. The conclusion on the generalization capability of the proposed model in the spatula manipulation task is not well supported by the results in Table 2a and Fig. 7. Some of the measuring scores double for the test object 2 compared to the training object.
review_title: null

Meta-review: metareview: The authors propose a method to predict the reaction wrenches at the wrist of a robot arm and contact embeddings of deformable objects in manipulation tasks. The proposed approach builds on prior work, where most importantly, the need for privileged information was removed.

Strength:

    -Exploiting force measurements at the wrist for predicting object deformations is a relevant and important topic.
    -Compared to the prior work, the proposed extension enables real-world applications.
    -This feature is exploited in two impressive real-world experiments.

Weakness:

    -It is an incremental approach.
    -No comparison to other approaches, not even in simulations.
    -The selected real-world manipulation experiments do not demonstrate the full benefit or the limitations of the proposed approach. How would the learned model work on other tasks as shown in Fig. 8 using the wok.
    -The description is often vague, e.g., the terms 'privileged information' or 'contact embedding' should be defined explicitly.
    -The description can be improved, e.g., Fig 5 has no axis labels, KDE is not defined, and what is x1e3? What is the value of the contact embedding?

Update: The authors added more explanation and additional experiments which greatly improves the paper. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)