Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an experimental evaluation of episode-based RL (ERL) on a series of simulated tasks, starting from a reacher, box pushing, ball in a cup, to table tennis. The paper starts by proposing a trust-region based ERL algorithm, and evaluates it against step-based approaches on setting with sparse reward functions, and show it performs better than PPO, even with dense rewards.
strengths_and_weaknesses: Strengths:
1. Well motivated problem setting
2. A large range of simulated environments used in the evaluation

Weaknesses:
1. The contributions are minor beyond experimental evaluation in simulation. The paper reformulates and existing approach for an episodic setting, and evaluates it using a motion primitive algorithm.
2. Since all evaluation is in sim, and most experiments take the order of 10M steps, real-world transfer of the approach is hard to determine.
3. The experiments are somewhat hard to understand. The states and action spaces are not well-defined, the cost functions are not clear, and the plots are extremely hard to read. For example Figurers 2,3 don't have titles or legends which makes it hard to understand the results. I wrote my own legends on the figures to make sense of the analysis. I would recommend authors to update these figures to make them easier to follow.
4. Why do authors compare against PPO, but not TRPL? Since PPO does better than BBRL-PPO, I would suspect that TRPL might do better than BBRL-TRPL. Authors should add that comparison, and use it as oracle with dense rewards maybe?
 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper's contributions are minor, and the applicability to real robots is questionable due to large number of samples needed. The paper has some really interesting simulation environments though, and I appreciate such a thorough experimentation with competitive baselines.
issues: 1. It would be good to add an additional baseline of TRPL in the comparisons.
2. Authors should improve the quality of figures and experimental section
3. Overall technical contributions are somewhat minor.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a movement primitive based approach to reinforcement learning - instead of learning a function approximator with state conditioning, the policy is parameterized as a contextual movement primitive that translates a parameter vector to an open loop action trajectory to be deployed in the environment.  The advantage of this kind of approach is that it allows for generating smoother trajectories, and has applicability in situations where it is natural to define sparse or non-markovian rewards. Extensive experiments show the benefit of the approach in non-markovian reward settings and its relevance to energy minimization.
strengths_and_weaknesses: **Strengths**

1. The paper propose a new method based on trust region optimization for movement primitive parameterized policies that allows for successful learning on manipulation tasks using reinforcement learning. The importance of trust region optimization for policy optimization is shown by an ablation.
2. The method is very relevant to the robotics community where using movement primitives can naturally speed up learning while ensuring more energy efficient and smoother trajectories.
3. The paper contains extensive experiments in simulated domains that are relevant to robotics. The results on meta-world demonstrate convincingly that the method can learn and achieve a higher asymptotic performance than step-wise RL methods which was surprising. The proposed method also allows to incorporate non-markov or sparse rewards while training which can lead to more aligned and energy efficient behaviors.

**Weaknesses:** 

1. Paper/Method clarity: 
a. In the method section 3.3, it is not clear how the trajectory generator converts a parameter to a state-based trajectory. Since it forms a integral part of their method it might be important to clarify as a background or in appendix.  
b. Does the inference of state-based trajectory use any dynamics or inverse-dynamics information? In that case the applicability of the method is quite limited and the baselines compared should incorporate dynamics knowledge.
c. How is the PD controller tuned? Are the samples used for tuning PD controller counted in the sample efficiency plots?
2. Baselines:
a. Efficient off-policy baselines like SAC are missing for a fair comparison in the experiments for domains like meta-world.
    
    b.  Can the authors give an intutive understanding of why PPO gets stuck in local optima for the meta-world tasks?
    
3. Engineering movement-primitives: 
a. How does the movement primitive scale to high-dimensional or long horizon environments? When the state space is large the importance ratio used in PPO may suffer from large variance because unlike step-based PPO now we have reweighting of large dimensional states?
    
    b. How is the movement primitive decided? The optimal policy’s state visitation should lie in the space of valid movement primitives and cannot be decided without oracle information and is hard to approximate in tasks like locomotion?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is very relevant to the robotics community and presents a method to incorporate movement primitives in the RL framework. The paper has extensive experiments which makes the method convincing but the method writing needs more work and clarity. The ability to incorporate structural prior in policy in order to address sparse reward and non-markovian tasks in nicely addressed in this work.
issues: My concerns can be found in the weaknesses section above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a deep episode-based reinforcement learning algorithm by adapting an existing step-based RL algorithm (differentiable trust region layer) to instead select a single action per episode, where the action involves selecting the parameters for a parametrized motion primitive which will be executed for the full episode. The authors then demonstrate that the proposed episode-based RL algorithm can more easily learn from sparse rewards for a number of continuous robotic control tasks in simulation. The main contribution of this work appears to be the first deep contextual episode-based RL algorithm and experimental results suggesting the utility of such an algorithm for robotic tasks. 
strengths_and_weaknesses: Strengths:

(1) While the algorithmic innovation with respect to the differentiable trust region layer work appears minimal (all that seems to be required is applying this method for one step and feeding the results to a parameterized motion primitive), the idea to develop a deep episode-based RL algorithm definitely seems to be novel and well motivated for robotic tasks.

(2) The experimental results do clearly indicate that the proposed approach is better able to handle sparse and non-Markovian rewards than step-based RL algorithms and leads to more energy efficient policies as a result across a number of standard simulation environments.

(3) The method and experimental results are very clearly and thoroughly described and easy to understand.

Weaknesses:

(1) My interpretation of this paper was mainly that choosing a more appropriate action space (in this case ProMPs) can lead to more efficient RL. As a result, while comparison to step-based RL algorithms is good, I believe that these comparisons alone are insufficient since they have access to a much less expressive action space than the presented algorithm. To this end, the paper would be significantly strengthened by adding comparisons to hierarchical RL algorithms with similarly parameterized primitives or comparisons where the choice of motion primitive is changed (ie. something different from ProMP).

(2) While I believe that the proposed approach would be useful for practical physical robotic experiments, it would significantly strengthen the paper to evaluate it on physical hardware. 

(3) [Minor]: It would be helpful to discuss connections between the proposed approach and ideas from the bandits and contextual bandits community. These papers may be of particular interest: https://arxiv.org/abs/1807.09809, https://arxiv.org/pdf/2206.08921.pdf
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I am positive on this paper and like the idea of essentially doing 1-step RL with classical motion primitives for robotic learning. The primary weaknesses of the paper are (1) the lack of comparison to hierarchical RL methods, which would be much fairer and interesting than comparison to purely step-based RL approaches and (2) the lack of physical experiments. Otherwise the contribution and experiments  are very clearly and thoroughly presented, so I am learning towards an accept decision at this time. 
issues: (1) Please add comparisons to (a) hierarchical RL algorithms with similarly parameterized primitives or (b) comparisons where the choice of motion primitive is changed (ie. something different from ProMP) to (a) provide a fairer algorithmic comparison to prior approaches and (b) study the impact of the choice of motion primitive on experimental results

(2) The paper would be much stronger if it is evaluated on physical hardware. RL training need not be performed on physical hardware, but transferring a learned policy from simulation to the real world would be beneficial in making a better case for the relevance of the proposed approach to robotic applications.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a novel method for episode-based reinforcement learning (RL), BBRL-TRPL, that uses trust region projection layers (TRPLs) to learn motion parameters, and uses Probabilistic Motion Primitives (ProMP) to generate motor commands. Compared to step-based RL algorithms, BBRL-TRPL shows better performance when the reward is sparse and non-Markovian, and learns more energy-efficient trajectories.
strengths_and_weaknesses: 
**Strengths:**

Overall, I find the writing to be clear and easy to follow. I do have a few suggestions on how to improve presentation, but the paper is overall easy to read. Moreover, the authors did thorough evaluation of their method against existing baselines with detailed analysis.

**Weakness:**

My major concern for the paper is that it misses an entire family of papers that use evolutionary strategies (ES) for reinforcement learning tasks, such as CMAES [1] and ARS [2]. The authors mentioned that these algorithms may be limited to "linear adaptations" (Line 43). It would be great if the authors could clarify on what "linear" means here, since these algorithms have been used to find the parameters of neural network policies. Moreover, these algorithms are known to work well across a wide variety of reinforcement learning tasks. Therefore, the authors should at least compare their method some ES baselines. A simple ES baseline could be a single policy that directly takes in the context and the robot states ($c$ and $s_t$ in Fig.1) and outputs motor commands.

Moreover, the major benefits of the proposed algorithm, such as better performance at sparse or non-Markovian rewards and structured exploration, have been known as the benefits of black-box algorithms _in general_ [3]. All these algorithms adapt the policy based on the total _return_ of the entire episode, and are not particularly sensitive to the shape of reward functions. Moreover, they all explore in the parameter space, and outputs smooth actions during training. Therefore, I find that the results of the proposed method does not improve much from existing methods.

[1] The CMA Evolution Strategy: A Tutorial [link](https://arxiv.org/abs/1604.00772)

[2] Simple random search provides a competitive approach to reinforcement learning. [link](https://arxiv.org/abs/1803.07055)

[3] OpenAI Blogpost on ES vs RL. [link](https://openai.com/blog/evolution-strategies/#:~:text=Injecting%20noise%20in%20the%20parameters.&text=However%2C%20RL%20injects%20noise%20in,guess%20and%20check%E2%80%9D%20on%20parameters.)

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Updating to weak reject after reading the author's responses. Also reducing confidence from 4 to 3 since I'm not very sure if I'm under-assessing the "contextual" part of the paper.

======================Original Content=======================

While the paper is well-written, I cannot recommend acceptance of this paper, given that the major result does not show significant difference from existing methods.
issues: First, I would like the authors to conduct additional results to benchmark their algorithms against ES baselines (see Weakness sections for more details). In addition, I have a few minor comments on writing:
* Separate Related Works and Background sections.
I found that Section 2 can be too long right now, and it can be confusing to read related works and the priliminary stuff (TRPL, RL) in the same section. It would be nicer to have a dedicated preliminary section for the problem setup.

* Equation in section 3.1: looks like the definition of the cost function is a bit off? The $\arg\max$ should appera in policy optimization, but not in the definition of $J$?
review_title: null

Meta-review: metareview: Scores: xpuk: Weak Accept, 3iLp: Weak Accept, FPE5: Weak Accept, Rejd: Weak Reject

Quality: The paper is of high quality providing new theory and extensive simulation experiments.

Clarity: The clarity of the paper has been improved in the revision. The additional baselines provide further support for the model. 

Originality: The proposed algorithm is interesting and novel.

Significance: The paper provides a solid contribution to robot learning by introducing a stable deep episode-based RL algorithm.

Pros: The paper provides
- a novel episode-based RL algorithm which incorporates movement primitives in the RL framework.
- extensive simulations and baseline comparisons showing robust performance better or comparable to SoTAs.
- the ability to incorporate structural prior in policy and the capability to address sparse reward and non-markovian tasks.

Cons: The major weaknesses of the paper are 
-  the lack of physical experiments 
-  a sim-2-real assessment
-  a lack of comparisons to other RL algorithms with more expressive action spaces than simply atomic ones.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)