Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper employs Rapid Motor Adaptation to tackle the problem of policy generalization in in-hand manipulation tasks. The paper mainly describes results from a new experimental setup of in-hand manipulation. Generalization technique is along the similar lines of RMA, which tries to extract extrinsics vector from proprioceptive history of interaction with the world (environment). This extrinsics vector is hence used by a policy network to adapt its behavior to sensed changes in the environment. 
strengths_and_weaknesses: ### Strengths
- The paper sufficiently motivates the problem statement.  
- The paper presents an interesting application of an adaptation technique previously demonstrated for a different experimental setup. 
- The qualitative hardware results seem sufficiently convincing and validate the applicability of the underlying method.
- The quantitive results (Figure 3) aptly explain how different methods respond to environmental perturbations. 
- The presented experiments cover impressive variations in object shapes and dimensions. 
- The ablation study is thorough and provides sufficient details about the training and testing procedures. 

### Weaknesses
- Most importantly, the paper only presents a new application of an existing idea and lacks any new ideas. 
- Furthermore, it does not present new insights on the existing idea (for instance, why having a separate extrinsic vector prediction network is a good design choice over directly training a meta-learning based policy with adaptive behavior).  
- The paper and the underlying method’s adaptive functioning is limited by the ability to simulate environment/robot variations. It would be interesting if authors could provide any insights on method’s response to variations not previously seen during training. (For instance, by freezing one of the known variable during training and seeing how the method responds during testing.)
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper presents a new and interesting experimental setup of an exisiting method but lacks originality in ideas or providing deep insights. 
issues: - Although the paper demonstrated a novel use case of an existing method, it doesn't present any new idea or new insights from an existing idea which is not ideal. It would be interesting if it could additionally comment on new insights gained from running the underlying method for a new experimental setup. 
- It is not apparent to me how the policy behavior changes in response to a different environment (i.e., environmental variation). It would be helpful if this can be explicitly demonstrated as a video / plot. As an example, it would be interesting (at least to me) to see how the policy behavior changes with and without continuously using the adaptation module (e.g., by latching its output / using the same output as obtained from a different object).  
- Could you also add qualitative results from the DR, SysID, NoAdapt, and Periodic baselines? In case I missed it, please point me to the correct videos. 
- Lines (42-43): What type of gaits emerged from the baselines? Could you talk more about why having an adaption module could have led to smoother gaits?
- Lines (103-104): Could you explicitly states these changes?
- It would be greatly appreciated if you could expand on the limitations section. 

I am looking forward to the discussion!

### Minor issues
- Lines (7-9): For a reader unaware of RMA, it can be slightly misleading what is being estimated by the method. 
- Line (36): It has some grammatical issues. (Please also look at the remaining introduction section as it has additional grammatical errors.)
- Line (37): It is not clear to me what `adaptively estimate` means. Does it imply continuous updates?
- Line (56, 57): I am not sure, but: Starting with citation number might be aberrant with the template. You might want to double check.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an algorithm to achieve in-hand object rotation through finger-gaiting using linkage-based robot hand. In the base policy training phase, a base policy was trained jointly with an object property encoder using proximal policy optimization with a custom-defined reward function. The object property encoder takes privileged object information and outputs an extrinsics vector, which, combined with the current robot joint position and the previous predicted robot action, are then used as the input of the base policy to output the next robot action. The next phase trained an adaptation module to output an estimated extrinsics vector using the proprioception history as input, in order to lift the dependency to privileged object information during deployment. The algorithm was tested in real world experiments using an Allegro hand, in comparison with different baseslines.
strengths_and_weaknesses: **Strengths:** 
* Very clear presentation, especially section 3 with the help of Fig. 2.
* Multiple real world experiments demonstrating the effectiveness of the proposed method.

**Weaknesses:**
* The demonstrated object rotation is 2D. This would have been fine if it was well addressed in the paper (either in the scope or in the limitation). 
* Because the manipulation is 2D, some of the experimental objects do not really demonstrate generalizability, because it is the 2D cross section that matters. Objects of different shapes could have very close cross sections. 
* Not sufficiently encode object geometry: there is no object geometry information encoded in the object property encoder because it was not used as input for training. This could also explain the fairly limited generalizability for 2D manipulation. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The authors are solving the general object reorientation problem. While the proposed method aims to show general 3D reorientation that can be generalized to different objects, the experiment results only demonstrated object reorientation around two opposing axes (essentially 2D reorientation). This also made the object generalization less impressive, because objects such as the badminton and the cup have similar 2D cross section as a sphere. However, the method is still valid and was executed very well even with the limited scope.  
issues: * The outcome shows a potentially narrower scope compared to what was claimed in the paper. Based on the videos from the project page, the proposed method only achieved 2D manipulation: even though the objects are 3D, the rotations are all around the vertical axis ([0, 0, 1] and [0, 0, -1]). It is recommended to either claim the correct scope of the problem or provide further demonstrations to support the claim of general reorientation. 
* The effective 2D aspect ratio of all objects are very close to 1.  Although there are a few long objects present in the experiment, their cross sections (where the fingers are in contact) have aspect ratio close to 1. This is okay as long as properly addressed in the limitation section. What would happen if the objects being manipulated (or their cross sections where the fingers are in contact with) have more extreme aspect ratios?  What is limiting the object generalizability? It will be very interesting to see the extrinsics vector of an more extreme shape compared to the nice shapes.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a system that learns an adaptive controller for in-hand manipulation, trained in simulation and deployed on a real robotic hand. The proposed approach relies on reinforcement learning in simulation, and does not require retraining or fine-tuning on the real robot, thanks to an adaptation module. The proposed system also relied on the fact that objects' characteristics, such as their shape and mass, as perceived by the fingertips proprioception can be compressed into a low dimensional embedding. Such representation then allows generalisation to unseen objects.
strengths_and_weaknesses: The paper presents a solid execution of an important topic in robot manipulation, that is in-hand manipulation. In particular, the proposed solution generalises across objects and is shown to be reliable on a real robotic hand. The proposed solution is simple and clean, although it has limitations (mentioned by the authors as well), by being only relying on proprioception.
The paper is written and structure in a clear way, and it is pleasant to read. Figures, diagrams and videos are well made and help clarify the contributions. 
The paper includes a clear description of the experimental setup, and implementation details are reported (also in the supplementary material submitted).
Experiments include a comprehensive set of ablations and comparisons, which make the contribution of the paper more solid.

What is the (quantitative) effect of the gap between simulation and real setups? While usually more relevant for visual-related features, the domain gap for proprioceptive measurements could be limited. Is this true in your case? what is the main discrepancy playing a role in your experiment? As your perception stack is not based on visual clues, the appearance of the object in simulation should not affect your transfer, however mass or COM could be more relevant in your setup. Which object characteristic posed the biggest challenge for the sim-to-real transfer using your method?
The reward definition proposed includes several terms that are also commonly used to regularise actions in RL: do you have ablation studies or empirical results to highlight which component is mostly affecting the final behaviour?
Can you motivate the choice of using 2 time steps for your $o_t= (x_{t−2:t}, a_{t−3:t−1})$ in your policy?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents a solid design, implementation and execution of a method to address in-hand manipulation on a real robot hand with a variety of different objects. Although the components of the proposed framework are individually not novel on their own, their combination and the results presented are significant and the analysis conducted is thorough. 
issues: Discussion on the simulation-reality gap in the context of the presented experimental setup: which part of your perception and control stack are less robust?
Reward formulation: which terms have the most impact?
Policy: motivate choices.

Minor: typos p.6 "The NoAdapt also *has performs* poorly...", " Our method *can achieves* consistent"
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper applies the training pipeline of Rapid Motor Adaptation to the setting of in-hand object rotation. Note that in this paper, the `in-hand object rotation` actually means the robotic hand keeps rotating the objects along the fixed $z$-axis, not in the full $SO(3)$ space. The results show that the allegro hand can quasi-statically rotate around 30 different objects. The real-world experiments look nice.
strengths_and_weaknesses: Strenghts:

* The overall learning pipeline is simple and convincing.
* The real-world experiments are nice and show that the policies actually transfer.
* It's nice to see that policies trained with only rotating a cylinder can rotate other objects to some extent.


Weaknesses:

* The paper is a direct application of a published method to a new task. The method itself is unchanged from the prior works. 
* One baseline that is missing is the ablation of the adaptation module. While the paper has a baseline called NoAdapt, I think there is another baseline missing. The aspect that's missing is whether we actually need a separation of an adaptation module and a policy module. The paper currently regresses the output of the adaptation module to the privileged state embedding. Is this necessary? In other words, can we just train a temporal network $\pi(x_{t-30}, a_{t-30}, ..., x_{t-1}, a_{t-1}, o_t)$ that takes as input the history of observation and outputs the action command (no explicit latent regression)? 


Claims/Statements that need to be revised and made clear:
* The learned gait is a quasi-static gait and the rotation is only around $z$-axis. From the demo videos, it's safe to say that at any time, the object is in a relatively stable pose with multiple finger support. This is sufficient for rotating the objects around the $z$-axis, which is not a bad thing. I think this is also why the policy works for different objects. The fingers have good support on the object being rotated, the policy only needs to figure out when contact happens between the fingertips and object (which can be inferred from the history of observations) and then move the fingers one by one.  The point I wanna make is that this task is much easier than dynamic object reorientation in full $SO(3)$ space that many prior works explored. So I don't think it is appropriate to criticize prior works (on in-hand object manipulation) on the lack of generality or sim2real transfer without stating what this paper is actually doing. They are tackling in-hand manipulation tasks with different levels of difficulty.

* The paper claims that the hand can rotate both convex and non-convex objects around $z$-axis. However, this is misleading. It is true that the objects used in the paper have different shapes. But what matters is the local geometry on which the contacts happen. For example, the paper uses bottle, paper towel, hourglass, sake cup, etc. At first glance, these objects look different. But because the paper only trains the hand to rotate objects around the vertical axis, the hand only needs to touch a very small convex (or almost convex) contour of the objects. And we can also see that on these objects (paper towel, hourglass, sake cup, shuttlecock, etc.), the rotation axis is carefully chosen such that other geometry of the objects does not affect the rotation. And the fingertips almost always work on the convex part of the objects. In this sense, these different objects used in the paper actually look quite similar. In addition, considering the fingertips of the Allegro hand are big, so the local contact geometry differences on the objects won't affect the motion as much. Therefore, I think it is misleading to claim that the policy can rotate both convex and non-convex objects. This should also be clearly discussed in the Limitation section.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I think the paper is working on an important problem. Dexterous manipulation is challenging. As a starting point, the paper addresses the problem of rotating objects around $z$-axis in a quasi-static way. The results are nice even though the method itself is completely from another published paper. However, I would require the authors to revise some misleading claims in the paper to make the writing more accurate. I am happy to raise my score if the writing is improved.
issues: Listed above.
review_title: null

Meta-review: metareview: This paper describes a system that is trained in simulation to manipulate cylindrical objects. Then it generalizes to manipulate a variety of objects in the real world with no fine-tuning. It uses a low-dimensional representation of the object shape as perceived by the robot's finger tips to achieve this.

Strengths:

Addressing an important problem of dexterous manipulation.
Concrete results on a real physical robot, performing sim-to-real transfer.
Weaknesses:

More extreme shapes with larger aspect ratios are harder to rotate.
Rotation is only in two dimensions and manipulation is quasistatic; the limitations of the approach need to be clearly addressed in the paper.

confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)