Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper propose a framework for deformable object manipulation with a dexterous multi-finger hand. It consider six tasks in simulation. The proposed framework solves these tasks by first collecting a few demonstrations by teleoperation, then it trains a skill dynamics predictor and action decoder for planning towards a given goal. The skill abstraction is defined by a fixed temporal interval. Finally, the planned action sequence is refined by gradient descent since the simulator is differentiable. The authors demonstrate success in the six tasks and it outputperform several alternatives such as reinforcement learning and imitation learning.
strength_and_weaknesses: Strength:
1) This paper is (to the best of my knowledge) the first work studies how to do deformable object manipulation with a multi-fingered hand. Previous methods either do deformable object manipulation using parallel jaw grippers or rigid object manipulation with multi-fingered hand.
2) The writing is pretty clear and complete. It’s easy to read and understand the main message of the paper. The figures are also nice and well-designed.

Weakness:
1) Since I consider this paper as an engineering effort to build a practical system, I’m worried about if the results can be easily transferred to the real world. Specifically, this paper assumes perfect point clouds both for input and goals. However, it is unlikely to obtain perfect point cloud observations in the real world due to sensor noise. It is also hard for the user to specify point clouds as the end state.
2) The authors does not promise code release. I think realeasing code upon publication can greatly improve the paper’s impact.
clarity,_quality,_novelty_and_reproducibility: As I summarized in the above section, this paper is well-written and clear. Conceptually it is not novel but it contains good engineering to build a system. I’m also concerned about the reproducibility because if the author does not promise to release their code. 
summary_of_the_review: In summary, this paper is worth reading for a large part of the community. Though it focuses more on a system design and implementation, it can still benefit the community a lot.

----
Score Increase from 6 to 8 after discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the task of dexterous manipulation of deformable objects and introduces a pipeline involving gathering human demonstrations, learning representations for the scene and the demonstrations, and augmenting novel data samples via a differentiable simulator. The proposed pipeline, DexDeform, starts with a small set of demonstrations to learn scene representations and skill abstractions, a dynamics model, and an initial policy. It then iteratively runs training and exploration stages where a gradient-based trajectory optimization is applied via a differentiable simulator to augment available samples. The paper presents evaluations on a variety of tasks, including soft object folding, wrapping, and flipping with a single or dual Shadow hand. 
strength_and_weaknesses: Strengths:

1- The paper is well-motivated. The concerns raised at the beginning are later addressed. 

2- The optimization objective in Eq. 1 leverages the implicit scene representation, nicely overcoming the challenges of modeling soft objects.

3- Few demonstrations are enough to initialize the models, and it is straightforward to gather them by using the proposed teleoperation setup.

---

Weaknesses:

1- What is the design choice behind using a non-dynamic skill abstraction network? I think a temporal variant with a learnable prior would be more useful and allow for sampling of the next latent sample z. A sampled latent trajectory could be used in the exploration phase instead of augmenting the targets and running optimization in the observation space.

2- It is not clear if the prior of the skill abstraction network is used. I assume the latent skill samples in Eq. 1 are randomly initialized from the standard Gaussian prior. Could the authors clarify this?

3- I think the evaluations could be fairer. The baseline models take the input from the scene in point cloud representation. The implicit scene representation learned by the ConvONet and the scene embeddings learned by the VAE are trained in isolation by using standard training objectives. I believe the baseline models could use the same latent embeddings as the proposed method.

4- This is in line with my previous comment. The baselines and how they are implemented seem rather straightforward. A goal-conditioned policy (see [1*] for grasping rigid objects as an example) could be well suited for the tasks introduced in this paper. The goals could be represented by latent scene embeddings. This is not a weakness per se, but a suggestion. I think stronger baselines would improve the quality of the submission.

[1*] Christen, Sammy, et al. "D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. It is clear to understand the problem, motivation, and contributions of the paper. The task and components of the pipeline are novel.
summary_of_the_review: Despite the weaknesses I mentioned, I think it is an overall good paper. It provides useful insights into the problem as well as intriguing solutions. I am leaning toward accepting this paper. I'm looking forward to getting further clarification and experimental evidence from the authors.

-----
**Post rebuttal update:** The rebuttal addresses my concerns, and I do not have any further questions. I think this is an interesting paper for the community. I am in favor of accepting this paper. I raised my score from 6 to 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores the task of dexterous manipulation with human demonstrations. The paper introduces an interesting set of simulated benchmarking tasks, along with a baseline approach to solve these. The benchmark builds on prior work (Plasticinelab), using a simulated shadow robot hand. The proposed baseline relies on human demonstrations captured using a leap motion controller, and mapped to shadow hand using an ik solver. An implicit representation of the scene (observed in point cloud form) is learned using a point cloud encoder decoder model. this representation is then combined with actions, and a sequence of these mapped into another latent space, a dynamics model is learned using this representation, and used for goal-based planning.
strength_and_weaknesses: Strengths:

The proposed set of benchmarks is interesting, and the results appear to show that the proposed approach performs well at these.

Weaknesses:

The biggest concern I have here is that the description of the proposed approach is rather unclear, and I struggled to follow section 2.4. It is not at all clear to be how the different components in the model connect, and how the baseline approach is trained or used at run-time. It would be helpful to provide a figure illustrating this, and to standardise notation to align with Fig 2. Notation needs a lot of care, for example Locc is seemingly also referred to L(g,h) on page 4, and this makes it hard to keep up with and follow much of the method. I appreciate that some of this is in the appendix, but feel section 2.4 could do with a bit of a rewrite in service of the reader, as at present the complexity here limits reproducibility and does not do the method justice.


Minor points:

Par 2: "...consider the problem of deformable object manipulation with the Shadow Dexterous hand." please rephrase to a *simulated* shadow dexterous hand.

The paper repeatedly claims to provide the first dexterous manipulation benchmark for deformable objects (to the best of the authors knowledge), but learning based approaches have been widely used in robotics literature, along with many deformable manipulation tasks (rope, cables, cloth folding). Plasticinelab, which this work builds on, requires the manipulation of deformable objects. Is the definition of dexterous manipulation used here specifically down to the use of a multi-fingered hand?

Can you provide additional detail about the environments? Eg. actions (are these joint torques?) and point clouds (are these rendered with respect to the camera view point? do they factor in occlusions) etc.

Section 2.3 "Given a novel deformable shape, how do we solve it..." Rephrase - I don't think you can solve a shape.

Section 3.3: "We illustrate that pure EMD might not be a good measure for soft bodies with large topological variations Feydy (2020)" - I don't see an illustration of this. If EMD is not a good metric, shouldn't the benchmark report alternatives too?

clarity,_quality,_novelty_and_reproducibility: The paper could do with some improvement of clarity, particularly around the method for this to be reproducible. I would recommend a bit of a rewrite for clarity, particularly focusing on notation, and explaining the method (perhaps rethinking what is in the appendix and what isn't). This also applies to the abstract, which doesn't really convey the approach very clearly.

The paper proposes an interesting set of deformable object manipulation tasks and a method of learning these, which could be useful for others to build on and test their algorithms. The method itself does not seem to be particularly novel.
summary_of_the_review: This paper introduces an interesting benchmark for goal oriented dexterous deformable object manipulation tasks, alongside a latent learning-based planning approach. The explanation of the latter needs to be improved.

---- Post rebuttal updates ----

I think the authors have added sufficient clarity and addressed most of my concerns, hence am raising my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper studies the manipulation of deform-able objects with a dexterous hand. They combine human demonstrations, with differentiable physics guided exploration, abstraction, and refinement. The approach is evaluated experimentally in 6 tasks

Strengths:
- Novel application domain
- Complete framework
- Works well experimentally

Weaknesses:
- Real-world experiments: the proof is in the pudding - especially since this setting is very hard to simulated realistically (de-formable objects / contacts)
- Quite a complex system with many moving components
justification_for_why_not_higher_score: Overall the paper seems to try to do a bit too much for the given space-limit. E.g. the designed teleportation setup only seems tangentially realted to the rest. The paper focuses on a specific setting, while the ideas and methods are potentially more broadly applicable. It remains a bit unclear what task specific and what more generally applicable.
justification_for_why_not_lower_score: All reviewers argue for acceptance.
Decision: Accept: poster