Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a new framework to learn long-horizon control policies on multi-tool deformable object manipulation tasks. A differentiable physics simulator is combined with a trajectory optimizer to gather expert demonstrations of short-term skills. The skill expert trajectories are then used to learn a neural skill abstractor. Finally, a planner learns to assemble the skill policies to solve the long-term task by solving intermediate goals.
main_review: ### Strengths

- The paper is well written and easy to follow. The provided videos showing the results highlight the difficulty of each task and the method ability to solve them.
- Splitting the long-horizon and multi-tool nature of the task into different short-horizon single-tool stages is a natural and intuitive way to manage to solve long and complex tasks.
- Using a VAE to learn a low dimensional latent space to encode the RGB-D observations is a clever way to allow generating intermediate goals and thus splitting the long-horizon task into multiple more manageable short-horizon tasks.
- The proposed tasks are fairly complex and convincing to show how their approach work for the defined problem.

### Weaknesses

- For the optimization problem in (2), shouldn't you maximize the cost function C(k,z) instead of minimizing it?
- A video comparison of the resulting policies to the chosen baselines would be nice (RL, BC and trajectory optimizer). This could illustrate where the other methods fail.
- What are the possible limitations of this approach in practice? Is the goal representation as an RGB-D image practical for example for real robot applications? How critical is resetting tools to initial poses at the end of each skill policy execution?

### Typos
- Page 6: "We also present the psedocode of our method in Algorithm" → **psedocode** to **pseudocode**.
- Page 8. "On the other hand, RL performs poorly on tasks, showing that it is difficult to learn A good policy directly from visual input." → Capital **A** to **a**
summary_of_the_review: The paper proposes a novel approach to solve a challenging problem: long-horizon multi-tool deformable object manipulators tasks. The idea can be beneficial for the robotics community and seems to be extendable to quite different tasks in the same setting.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper provides a framework for learning how handle deformable objects. Specifically, the authors introduce a 3-step process where (1) in a differentiable simulator low-level skills are learned, then (2) the learned skill are converted from state observations to image observations via behavioral cloning, while training a feasibility detector that helps finding the right skill and tool for a goal state and lastly, (3) planning a long-horizon task by searching for both the low-level skill and latent variable (representing an intermediate goal).
main_review: **Strengths:**

- I really like the idea of transfer-learning between diff-simulator and image-based policy, as well as searching over the latent for planning intermediate goals.
- The video results are quite impressive.
- The paper is very focussed - there's enough theory to understand the idea, there's enough implementation detail to get an understanding how to implement this, there aren't many experiments but the ones that are there are reasonably novel.

**Weaknesses:**

- W.1 Maybe this is just me but the writing could be a bit more polished for accessibility. It took me 2 reads for it to "click" that when you write "neural skill abstractor" in the abstract and intro, what you mean is that you're learning image-based control from a state-based ("expert") policy with the purpose of applying this in an image-based setting. I think just adding a sentence or two to explain this or create more of an intuition why we need to learn this would go a long way. The same is true imho for the fact that you're discovering intermediate goals via searching over z. 
- W.2 There's a couple typos in the main body of the text but I'm sure you can fix those. Also in the first sentence, calling elder care "deformable object manipulation" is... interesting. 
- Honestly, that's all my criticism. The paper is pretty good.

**Nitpicks/Questions:**

- When mentioning differentiable physics and planning therein, it would be nice to also mention the recent work on GradSim (Jatavallabhula, Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine et al. "gradSim: Differentiable simulation for system identification and visuomotor control." ICLR 2021.)
- Why did you decide to train all network components simultaneously with one big loss term? Did you try training them separately? Are there any upsides/downsides to that?
summary_of_the_review: UPDATED (2021-11-21): The authors answered my questions and updated a few sections of the paper. I think the work is now in a great state and should be highlighted.

OLD: I'm pretty happy with the paper. It's very novel, well-written, and would make for a great contribution to the robotic manipulation literature.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine a local controller with a global planner for long-horizon deformable object manipulation with different tools. The controller is modeled as a neural network that takes an RGBD image as input and outputs the control signal. The global planner iterates all possible tools and optimizes a utility function, which is the multiplication of predicted feasibility and reward function. Differentiable physics is used to generate demonstrations for short-horizon skills. All observations are embedded in a continuous latent space, on which the optimization is further performed.
main_review: This paper designs a pipeline to learn multistage deformable body manipulation tasks. It is hard to simply apply differentiable physics in this task since there are lots of local optima and even discrete variables - the choice of different tools. This task is also challenging for reinforcement learning because the process can be complex, the reward function can be sparse, and the control variables contain both continuous and discrete values. 

The authors find a clever way to circumvent the challenges. 1. Differentiable physics is used to optimize local trajectories, such that demonstrations are generated efficiently and then used to train an NN-based controller. 2. A global planner is articulated as optimization of both the intermediate states and the choice of tools. The state is in a low-dimensional continuous latent space encoded by a VAE network. 3. Two NNs are learned to approximate the feasibility and reward function, which are used when scheduling the global motion.

When reading this paper, I have several concerns,
1. Typos: 'psedocode' -> 'pseudocode', 'there strong baselines' -> 'three strong baselines'.
2. In Equation 2, should it be maximizing $C(k,z)$? and should $\bar{r}$ be negative or positive?
3. When optimizing $z_i$ in equation (2), is it efficient/correct to first treat the problem as unconstrained and then project variables back to the constraint set? Can you directly solve it as a constrained optimization? 
4. What could be the possible limitation and future work of this method.
summary_of_the_review: This paper formulates a pipeline of applying differentiable physics to long-horizon tasks. The experiment setting is very challenging. It deals with sensory observation, deformable body, variable tools, and complex tasks. The proposed method solves those problems reasonably well as shown in the video. I think their method has the potential of generalizing to other similar tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to learn long-horizon control policies. It uses differentiable physics simulator to generate skills for the latter imitator to learn. To complete the long-horizon task, the proposed method first exhaustively search for tools/skills to be used in each step. Next, it optimizes the intermediate observation states to maximize the feasibility and the reward through the trajectory. At last, the policy of each small step is obtained by feeding start/end state pairs to the imitator. the results show that the proposed method is much better than both traditional RL and the trajectory optimization offered by differentiable physics alone.
main_review: Strengths:

 - The method uses differentiable physics for obtaining trajectories that are required for the imitators to learn, which eases the difficulty of designing a good reward function during RL.
 - Different skills and tools here are separately considered, decoupling the problem into smaller ones.

Weaknesses:

 - Is a skill dependent on the tool alone or the observations together as well? Specifically, if you have different start/end positions of the same short-horizon task using only one tool, are they considered the same skill?
 - Why not training the VAE alone? Generally speaking, this module should be standalone.
 - How do you determine H in each long-horizon task? How does different H in one task impact the performance?
 - The method exhaustively search over all combinations of different skills in each small step. Does it mean that in some specific configurations where H and num_skills are large, the proposed method can actually be even slower than the trajectory optimization provided by differentialbe physics? I would like to see more discussions or experiments regarding this matter.
 - How did DiffSkill perform on single tool experiments?
 - Why Trajectory Opt has 0.544 improvement and 0% success rate in 'Tool A Only' while the numbers changed to 0.385/20% in 'Multi-Tool'? Aren't they positively correlated?
 - It seems that all previous baselines somewhat fails in the experiments designed. I wonder how they would perform in simpler ones where they actually have achieved something meaningful (i.e. success rate>0), and how the proposed method would compare in these examples.
summary_of_the_review: The paper addresses the challenging problem of long-horizon RL by using differentiable physics and imitation learning. It is a inspirational method in terms of the way differentiable physics is coupled in large-scale complex problems and the decoupling of different sub-steps. However, there are some issues described above that are not clearly demonstrated in the main text nor in the supplementary document. So, I decided to give a score of 5 as the initial rating. I am happy to raise my score if my concerns (together with those from other reviewers) are properly addressed.
review_title: null

Meta-review: Manipulating deformable objects is an up-and-coming topic in robotics and machine learning, and it creates many interesting scientific and real-world challenges. The paper looks into long horizon tasks of manipulation of deformable objects, using an interesting mix of more local trajectory optimization and differentiable physics. The reviewers agree on the interesting significance of the suggest work, all above acceptance threshold, but also a bit bimodal in terms of “just above” vs. “solidly good”. Thus, the paper appears an useful and discussion-provoking accept for ICLR.
Decision: Accept (Poster)