Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors used a hierarchical setup to learn complex manipulation tasks. Instead of using raw joint controls, the the action space is composed of a selection of primitive actions including lifting, pushing, and dummy primitive action ,etc. Then at high level, a learned policy decides to choose which motion primitives and outputs the corresponding parameters. The authors demonstrated that using their method, RAPS, the robot arm can solve many benchmark manipulation tasks within shorter wall clock time, significantly outperform a few STOA baselines. 
main_review: The main strength of this paper is that, using the RAPS architectures, many challenging tasks can be solved effectively in sim after a few hours of training. By constraining the action outputs to the motion primitive space, the framework is also insensitive to the specific learning algorithms being used. The paper is generally well written, though a few details warrant more discussion.  The following are may comments to address:

(1) The dummy primitive. The authors said "As the dummy primitive operates on the high level horizon for Hk steps when called. Therefore, it cannot execute every trajectory that a lower level policy could. " This is a bit confusing. If the dummy primitive is already the raw 6D end-effector pose, together with the gripper state they should represent all possible states of the manipulator. If the primitive cannot be finished within "Hk" steps, one can simply resume the same inputs again. I think more clarifications are needed here.

(2) The parameterization of the primitives. The authors mentioned that the arguments of the primitives can be sampled from a learned distribution. However, the dimension of each primitive's arguments is still different, even if one uses a multi-dimensional Gaussian. More discussions are needed to clarify how arguments are encoded in this scenario. 

(3) Choices of primitives. I think a detailed discussion on the importance of each primitive is needed. For example, in order to solve a single task, how many primitives are effectively used? What is the usage percentage/activation time for each of them? Also, a bar plot showing how each primitive is selected over time would be interesting. 

(4) On top of (3), one may perform an ablation study on the minimum set of primitives needed to solve a particular task effectively.

(5) Cross robot transfer. These primitives used in appendix seem to be agnostic to robot geometry (unlike joint pose spec). Can we demonstrate how well a policy trained with one robot can be transferred to another robot that is morphologically different?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Low-level actions, such as torques or joint-velocities, provide a hard-exploration challenge for applying reinforcement learning (RL) to robotic systems. For example, it might be impossible to reach and grasp an object solely due to random torques or joint velocities. To overcome this difficulty, one can define high-level actions to enable a meaningful exploration. In this paper, the authors introduce parametrized action primitives, which encode a robotic movement and use those actions for learning with RL.
The parametrized actions are just manually encoded primitives, such as moving the arm up, reaching a position $x$ and grasp, and so on.
Therefore, the system proposed by the authors requires expert knowledge of the system, the task, and robotics.
This simplification of the task results, according to the paper, in a more efficient RL. 
main_review: Albeit I strongly agree with the authors that RL cannot be performed on "low-level" action (especially if one wants to learn directly on a real system), I struggle to find any novelty or contribution.

The proposed parameterized actions are - in my understanding - just mere trajectory planners. I do not find any novelty in their introduction nor their use in RL. It is not surprising that the proposed framework is more efficient than baselines since the framework requires much more human effort. The authors argue that these action primitives are easy to implement (perhaps easy for a person who can program and understand a robotic system) and transferable between robots. 
Naturally, the authors claim that this framework works with different RL techniques, such as model-based, model-free. Again, this is not surprising since what the authors are doing is to frame a different MDP where instead of low-level actions, we have high-level ones.
I wonder why the authors did not choose perhaps a smarter parametrization of robotic movements. The literature about movement primitives is abundant. Dynamic movement primitives (DMPs) [1] or probabilistic movement primitives (ProMPs) [2] are good examples of parametric movements that can be used along with RL. Traditionally, movement primitives can be automatically extracted by existing data using simple regression and segmentation techniques when needed.

In short, the authors propose just a higher-level action MDP, where any RL technique will benefit the abstraction. The method is not novel, does not give any new insights about RL or robotics. In my opinion, the proposed solution completely lacks novelty. Movement primitives have already been used in the context of reinforcement learning [3, 4]. 
I would recognize some novelty if the authors had proposed a novel technique for extracting movement primitives, or a new reinforcement learning technique specifically designed to use movement primitives.
In my perspective, none of the two has been done. 

-------------- UPDATE ---------------
After the discussion with the authors, I am willing to give more credit to their work. 


[1] Ijspeert, Auke Jan, et al. "Dynamical movement primitives: learning attractor models for motor behaviors." Neural computation 25.2 (2013): 328-373.
[2] Paraschos, Alexandros, et al. "Probabilistic movement primitives." Advances in neural information processing systems (2013).
[3] Stulp, Freek, and Stefan Schaal. "Hierarchical reinforcement learning with movement primitives." 2011 11th IEEE-RAS International Conference on Humanoid Robots. IEEE, 2011.
[4] Li, Zhijun, et al. "Reinforcement learning of manipulation and grasping using dynamical movement primitives for a humanoid-like mobile manipulator." IEEE/ASME Transactions on Mechatronics 23.1 (2017): 121-131.
limitations_and_societal_impact: The authors discussed the limitations of their work.
The societal impact was not addressed, but, in all fairness, I believe was not necessary.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel action space for efficient learning of robotic manipulation tasks. In contrast to typical joint space and task space control, the proposed action space consists of a set of hand-designed agent-centric primitive skills (e.g. lift arm for $d$ cm, close grippers to $d$ cm, and rotate wrist for $\theta$) with continuous parameters (e.g. $d$ and $\theta$). Since the primitive skills are agent-centric (i.e. conditioned only on a robot state), the primitive skills can be used across diverse environments and tasks. With this appropriately designed action space, a high-level policy learns to choose which primitive skill and corresponding parameters to execute from visual inputs and sparse rewards.

The exhaustive experiments on three robotic manipulation benchmarks and three different RL algorithms demonstrate improved sample efficiency of the proposed action space. The proposed method outperforms baselines with joint space control, task space control, and learned skills from offline data or random interactions.
main_review: **Originality**: This paper proposes a novel action space that consists of multiple manually designed primitive skills, each of which is parameterized with continuous variables. This idea is somewhat similar to the idea of motion primitives but could be designed more general to different environments and tasks. 

**Clarity**: The paper is clearly written and easy to follow. It would be great if more details about the primitive skills and how to derive this set of skills are discussed in the main paper.

**Quality**: The experiments are thorough and the proposed method successfully tackles many challenging manipulation tasks, which is impressive. 

**Significance**: The primitive skills are tailored to the robots and tasks, which requires domain knowledge and human efforts for designing them. The paper claims that once the primitive skills are designed, they can be transferred to other tasks and environments, but it might be still limited to similar tasks and environments, which may require new set of primitive skills for new tasks and environments. However, the proposed method seems timely as this is a simpler alternative of widely used task space and joint space control, and provides significant improvements in exploration and sample efficiency.

The paper claims that the proposed action space is suited to learn from visual inputs and sparse rewards. Intuitively, this is true as the set of well-designed primitive skills can always provide meaningful transitions, so reduce the exploration burden. However, this is generally very challenging to learn a policy from high-dimensional visual inputs and sparse reward. Especially given that the action space is not smaller than the original action space (either joint torque control or end-effector control), training the high-level policy may require significant engineering efforts. Could you elaborate whether this was not the case or there were some important implementation or training details.

The proposed action space is tailored to the robots and tasks. The reviewer wonders how diverse set of tasks can be covered with the proposed action space, which set of tasks cannot be handled with the proposed action space (e.g. velocity, force sensitive tasks like catching a ball, pouring, etc), and how this action space is scalable, as more number of primitive skills and parameters would be required for complex tasks but it makes learning of the high-level policy difficult.

It is great to compare sample efficiency in terms of both "wall clock time" and "training steps". However, the comparisons on "wall clock time" could be highly dependent on implementation of each method and environment. Moreover, the "training steps" can be prone to the proposed method as the proposed method uses longer-horizon low-level skills compared to baselines. If we want to simulate real-world scenarios, the environmental steps (the number of low-level interactions) can be the best evaluation metric as real-world interaction is the most slow, dangerous, and costly component in real-world RL. It would be great to see the comparisons over the number of interactions.
limitations_and_societal_impact: Adequately addressed.
review_title: null

Meta-review: The reviewers had some concerns that were all addressed by the authors. Especially, the authors did new experiments and provided a lot of new results. All these new results are extremely useful to support the claims of the authors and *should be included* in the final version of the paper. For this reason, the paper will have to be significantly revised but, on the basis of the discussion, the reviewers are confident that this final version will describe a strong enough contribution. 
Decision: Accept (Poster)