Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method of learning to manipulate a wide range of objects above and below the palm of the dexterous shadow-hand. Their method involves training a teacher policy with privileged information from the simulator (such as the true ground-truth state information), and then using it to distill a policy with Dagger into a student policy that learns to take actions in joint space directly from joint positions and a RGBD point cloud of the object. They compare both MLP and RNN architectures, and train and test with and without domain  randomization, on both the YCB and EGAD object datasets. They also make architectural contributions in their visual policy architecture by designing a sparse 3D convolutional version of the IMPALA architecture, which takes raw point clouds as input and outputs actions.
strengths_and_weaknesses: Strengths:

- The sheer scope and variation across objects tested with this method, and the range of different policy architectures and approaches tested makes this paper extremely thorough in its analysis of this reorientation task.
- The authors deliberately choose joint position control with RGBD sensors to make their method easier to transfer to a real robot environment.
- The use of the gravity curriculum to successfully train the hand to manipulate the object upside-down is also a considerable feat and seems to work well in simulation. It remains to be seen if this policy is transferrable to a real environment.
- Their method also shows the ability to zero-shot transfer across different object datasets (with YCB -> EGAD showing better transfer results)

Weaknesses:

- The authors mention that their current training strategy terminates the episode if the time limit is reached, or the hand is successful at reposing the object to the goal orientation. However, this does not guarantee that the final state of the object is stably grasped by the hand. This is apparent given one of the learned strategies for the upward-facing hand throws the object up with a spin to get it to the correct pose. While it is an interesting emergent skill, it is a limitation that their method does not guarantee that the object can be used by the hand after being reposed.
- The authors claim that position control is preferred to torque control for sim-to-real transfer. However since their method is not actually tested on a real robot, and they do domain randomization to be invariant to joint and tension dampening, the stated reasoning is not fully consistent. 
- The student network operates with an RGBD point cloud and position of all the hand joints. Point clouds are in general noisy and might contain even more noise than tested in simulation with domain randomization when factoring the occlusions and interaction between the hand and the object. The transferability between sim and real of this policy remains a considerable challenge.
- The notable issue of reorienting symmetric objects with RGBD sensors discussed in the paper makes their method less robust for certain kinds of objects. It might be worth exploring whether using object keypoints can help reduce some of these effects.
summary_of_recommendation: I would like to recommend this paper purely on its merits for having extensively explored how to enable dexterous reposing of objects with model-free reinforcement learning methods using 3D Convolutions and teacher-student training. While they do not show their method working on a real hardware platform, due to the current state of the pandemic, they do extensively test their method with different architectures and object datasets. Additionally, they demonstrate the generality of their approach by showing zero-shot policy transfer across these datasets. One limitation of their method include the fact that the environment terminates episodes after successfully reorienting the object to the goal pose without seeing if it falls/can hold that position after the episode ends. 
issues: - The tables are a bit difficult to quickly scan for which method shows the best performance, bolding the best performing method might be informative in drawing conclusions later from the results.
- There are multiple instances in the text where there are object variations in the scale, mass, and aspect ratios of the objects that are scattered throughout the sections (3, and 4.1). It would be helpful if the success rates for each of these different variations were compiled in a table in the appendix (C or D), so that their effects relative to one another on the performance of the method could be communicated more clearly.
- Table D.5 shows the performance of their vision model trained and tested with random transformations done on the point cloud. It would be useful to show the effect of these perturbations with an ablation study showing the effect that each of these transformations (and all of them combiend) on a model that is trained without them.
- I was unable to find the experimental results on the zero-shot training performance on objects weighing up to 500g referenced on line 212 in the paper and appendix.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper includes two main contributions. First, the authors developed an approach to transfer (using DAGGER) a teacher policy trained (using model-free RL) with full state information to a student policy trained with only sensory information. To achieve this, they exploited the fact that in simulations the state information and the correlated sensory information are simultaneously available, which enables the student to imitate the teacher’s action and learn a policy that maps sensory information to the same actions. They applied this design to in-hand reorientation of objects and demonstrated that their approach allows learning of successful reorientation for a much larger set of object shapes than has been attempted by the research community. Second, the authors also developed an effective way of training the agent to do in-hand reorientation with the robotic hand facing downwards. To achieve this, the authors had to develop both a lift policy that is used to identify good initial poses and a “gravity” curriculum where the gravitational acceleration is changed gradually from 1.0 to -9.8. 
strengths_and_weaknesses: Strengths

i.	The transfer approach developed by the authors is a promising approach. It is capable of leveraging increasingly better simulators and increasingly bigger datasets or model sets of diverse physical objects. The fact that the authors could get reorientation to work for a much larger set of object shapes than before could be believably attributed to the effectiveness of their approach.

ii.	Getting reorientation to work well enough with the robotic hand facing down is not trivial. The authors did a fairly thorough study of this highly specific but very challenging case.

iii.	The empirical results reported are largely believable.

iv.	The system integration, while not involving physical robots, is nontrivial.

Weaknesses

i.	There seems to be two papers here: the first one about the transfer method and the second one about learning reorientation when the robotic hand is facing downward. Either of these have enough content and importance. What is unfortunately is that the authors completely failed to relate these two aspects of the paper/project in an organic way. See Issues.a below for more discussion.

ii.	It should be noted that the effectiveness of the transfer method is not fully validated in the sense of supporting a full transfer to physical robot. (Of course getting Shadow hand to work physically with deep RL is in itself a huge challenge.) The reason this ultimate transfer to the real world is important is because it will test how heavily the transfer depends on the full coincidence of state information and sensory information that is likely only obtainable in simulation. The method's strength won’t be adequately established until evaluated against a full transfer to the real world. Similarly, the method's alleged “simplicity” (which the authors never made adequately explicit, but presumably is another way of calling the straightforward exploitation of the coincidence of state and sensory info in simulation) may become a liability if transfer to the real world cannot be made to work well.

iii.	The draft in its current state is quite rough with various writing and formatting issues (pointed out in Issues below). But the presentation overall suffers from a degree of being scattered. In addition to the major issue of there being two projects/papers in one, some other examples include specifics and motivations for variations such as MLP vs. RNN being missing.

summary_of_recommendation: In spite of the weaknesses identified earlier and the numerous issues identified below, this reviewer believes that the authors have done enough and achieved enough to warrant attention from and discussion by the broader CoRL community. The presentation in the paper, which needs to be significantly improved in the final submission, does manage to get the main contribution across as compelling research. Either of the two major parts of the paper could stimulate and engender much meaningful research conversations.
issues: a.	Title: “A Simple Method for Complex In-Hand Manipulation”. From the text, it is not clear what the “method” is. Is it the distillation of policies trained with state information into ones that do not use state information? If so, how do the various arrangements needed for learning manipulation with hand facing down fit into the method? Also not clear is what the authors mean by “Simple”. In what sense is the method simple?

b.	Regarding the fact that this paper includes two projects: (1) transfer and (2) in-hand reorientation with hand facing down, the authors made no attempt to link the two themes. For (2), the authors ended up developing a lot of machinery (lifting policy, gravity curriculum, table support etc) to investigate and to get it to work. Doesn’t all these under (2) make “the method” very complex? Or if (1) and (2) are really different, maybe we should do two separate papers? If (1) and (2) are related, how should or could the transfer method be adapted in a curriculum context?

c.	In the eyes of this reviewer, the authors may have missed a great opportunity to exploit a hand-orientation curriculum. That is, instead of artificially tweaking gravity, we could much more naturally make hand orientation change according to a curriculum. (See o. and r. below for more.) This could allow the authors to eliminate most of the complexity noted above in b. above. Moreover, there might also be an opportunity of linking the two parts of the paper together, if a hand-orientation curriculum could be organized with the curriculum for the teacher requiring fine-grained sampling of orientations but that for the student requiring much more coarse-grained sampling of orientations (or something like that).

d.	Figure 1: presumably misplaced above the Abstract.

e.	Line 3 in the Abstract: “a simple model-free framework” -- the whole abstract contains zero information about either the method (as in the title) or the framework. Nor does it give any sense to the word “Simple” in the title and here. Presumably, the unique or novel features of the method or the framework should be explained or at least noted in the Abstract as well as how such features make the method or framework simple.

f.	Lines 18-19: “These two tasks are harder because the object is in an intrinsically unstable configuration”. This does not seem to be true of the case where the object being manipulated is supported by a table.

g.	Line 40: Among the desiderata, (d) could be more clearly formulated in explicit contrast with approaches that require accurate state measurement (e.g. through motion capture).

h.	Line 43: “building a simple framework”, again no real information was given as to why and how the framework is simple in this Introduction.

i.	Line 63-66: This sentence adds nothing. It seems to be a wasted sentence because it started to talk about how the student policy is trained and then deferred to Section 2.2.1 immediately. It is also not clear how this sentence connects with the next one that starts with “However, …”

j.	Line 94: should introduce delta q_t^target explicitly.

k.	Section 2.2: it is probably going to help a lot if a table is included to directly compare the observations available to the teacher (Full State) and those available to the student (Reduced State).

l.	Figure 2: gap is needed between the main text and the caption for Figure 2.

m.	Lines 189-190: theta^bar under the two settings are respectively 0.1 and 0.2. Any principled reason why? Also, d^bar_C is set to 0.01. How does that combine with theta^bar = 0.2?

n.	Line 192: “reorients” => “reorient”.

o.	Section 42 Lines 235-245: The discussion in this paragraph is important but seems more narrow than warranted. First, the authors state “A natural question to ask is then does this still hold true when the hand is flipped upside down.” (Lines 236-237) But a more natural question seems to be if the hand’s orientation deviates from facing upwards at all. There are many possible orientations the hand could take. Facing up and facing down are just two extremes. In fact, the gravity curriculum proposed by the authors could be possibly replaced by a hand orientation curriculum that could be much more natural and potentially enable learning of policy structures much more generalizable. Second, “Unsurprisingly, the agent is unable to learn …” is possibly only unsurprising if the hand is putting in the extreme orientation of facing downward. The agent has no opportunity of learning effect of gravity for different orientations, with the two studied in the paper being only two extremes occupying a whole continuum of rich structures.

p.	Section 4.2.1: some comment/information about the strategy learned by the agent is expected. For example, does the agent leave the object to be supported by the table alone without any contact between maneuvers? If so, how that may have simplified the learning for the agent? If not, how does the support help? The reader is left wondering …

q.	Lines 267-270: the method of training the agent to learn a lifting policy and then use it to determine initial pose, while effective, seems to add a lot of complexity to the method. The reader is left to wonder why the method/framework is still characterized as “simple”.

r.	Line 287: “we start the training with g = 1 m / s^2,” -- This means there is (anti-)gravitational force moving the object upward into the hand. Why is this needed? How could this be justified?

s.	Line 309: “high success rate” -- there is no reference / baseline / comparison offered to support the claim that the success rate is high.

t.	Line 351: “this policy” => “the teacher policy” might be more clear.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a system in simulation for in-hand reorientation of novel objects. The work leverage privileged information from the simulator to train a model-free teacher policy and then applies Dagger to learn a model-free student policy which has access to only sensor inputs that are generally available in the real world.

strengths_and_weaknesses: Strengths
+ I consider this a 'systems' paper as it builds on mostly known algorithmic advances but effectively puts them together in a 'simple' manner to demonstration fairly strong results on a hard in-hand reorientation task.
+ The experiments are quite comprehensive and the paper makes good faith effort in arguing the applicability of the method to a real setting despite the results demonstrated in sim only.
+ Results are impressive and will serve the community well as a strong starting baseline for future work.

Weaknesses
- w1: I expect/encourage the authors to release code, models, and datasets otherwise much of the value of the contribution would be lost.
- W2: Insights gained from experiments were quite helpful, it would be good to have a more extended discussion on the limitations.
summary_of_recommendation: Overall, this is a solid paper that makes advancements in in-hand manipulation.
issues: W2
- Besides the domain randomization tried, I believe there will likely be more considerations needed for zero shot sim2real to be possible -- this is just based on the fact that while simulation photo-realism has come a long way existing simulators still massively struggle (and make strong assumptions) when it comes to simulating contact. One could sacrifice sim speed for fidelity, but that would likely not serve well for a model-free RL pipeline.
- The simulation/task seem to get cut off when the success condition is triggered, but it seems that some of the executions were mid motion. This in practice would lead to overshooting goal, instability, dropping object, etc. A fix to consider would be to add some zero velocity criteria (so object is stable in-hand) on top of the current goal reward.
- How do we scale model-free RL to real world and lead to more 'natural' motions?
    - An observation from the experiment videos is that there tends to be a lot of flaying and extra finger motions, likely since the reward is not capturing these.
    - What about ego centric perception and thus heavy occlusion when working in the real-world.
    - How do we continue to learn online in the real world since we won't have access to privileged information anymore.
The above questions are obviously hard and open research problems, but some discussion in the context of the limitations of this paper would be beneficial.
review_title: null

Meta-review: This paper presents impressive results on in-hand manipulation using imitation learning techniques in simulation.

The reviewers and area chair found the rebuttal to answer most of the lingering questions and concerns brought up by the first round of reviews.

We found this paper to be an extremely impressive systems paper that I recommend for acceptance.
Decision: Accept (Oral)