Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a novel RL method that learns an adaptive curriculum to minimize the number of resets needed in episodic RL. This is evaluated against competing algorithms on manipulation tasks.
main_review: Minimizing the number of resets needed could be useful in many real-world RL applications. The approach is interesting and the experimental results look very promising compared to competing approaches.

My main concern is with the presentation. The paper is mostly well-written, but there appears to be many complicated design choices in there with subtle theoretical implications. I'm only superficially familiar with the related work in this niche of RL, but the general idea and mathematical formalization could be clearer. I had to read it several times.

Some comments:
- If the focus is *reducing* resets (see main contribution on L54), it was a bit confusing why the main baseline wasn't the number of resets needed vs. episodic RL? Instead you compare against "episodic" RL without resets, which is also a bit counterintuitive.
- It is not obvious to me why RL without resets performs so poorly in Fig.2. You claim "This is because these methods rely on the ability to sample the initial state distribution arbitrarily", but this does not seem entirely true as RL can be non-episodic (and many "episodic" applications just use approximate infinite horizons via e.g. GAE). Could you comment on why this happens? Is it perhaps because your learned reversion to the initial state distribution yields better exploration somehow (e.g. if items otherwise get pushed out to the corners with time), or could it just be that you are using HER?
- Likewise in the formal problem definition in (3), it's a bit confusing to use a finite horizon formulation when considering an infinite horizon setting in Fig.2 and calling it "persistent" RL. It really seems to me that this is an inifinite horizon problem. It's not that you are minimizing resets, you want to maximize reward over an infinite horizon without resets at all, right? It would help to at least clarify this.
- The abstract and intro also seems to equate all RL with episodic RL. Again, careful with terminology and notation.

Minor details:

L66: "can be up to 30% more sample-efficient" - needlessly vague, were on average x% more sample-efficient in our experiments perhaps?

L70: manually design*ed*

L246 & 252: Conenct reference [2] to HER.	

L252: "... and *the* goal ..."

L292: "the the"

----


After rebuttal:
I thank the authors for their clear response. I share some of the other reviewers' concerns that this paper could benefit from a more formal treatment, but the authors' clarifications were helpful. I will maintain my score of 6.
limitations_and_societal_impact: Superficial but adequate.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a curriculum approach for persistent goal conditioned reinforcement learning, that is, for learning how to train continuously and then evaluate in episodic fashion. The main idea is, in each iteration, to try to keep the value of moving from a selected state to the goal above a threshold while selecting the state (where the agent tries to move before moving to the goal) as close as possible to the initial state distribution. The proposed approach outperforms comparison methods in three different simulated robotic benchmark tasks.
main_review: According to my knowledge persistent learning is a not so well researched problem setting and thus the proposed approach is novel. The robotics motivation is definitely valid. Related work appears to be adequately cited.

The submission appears technically sound. The experimental results show that in three different simulated robotic goal-conditioned tasks the proposed algorithm outperforms baselines.

The proposed algorithm is based on generating a state that is as close as possible to the initial state distribution while maintaining predicted value over a threshold. The goal generator then uses that state as the current goal until reaching that state and going towards the goal of the actual task. Distance is measured using the number of time steps from the state to the initial state from existing data.

An interesting question is how the discount factor influences the behavior of the algorithm. Compared to continuous learning, since the agent always goes to the same goal and the same task repeats, here long term planning and thus the discount factor are not as important?

The paper makes the assumption of a reversible environment but this is adequately discussed.

Computation time should be reported in the paper since the approach does computations that the comparison methods do not.

The paper is well written and organized. There are some typos that should be corrected. Examples:

Line 79: "a task-graph uses the current state to decides" -> "a task-graph uses the current state to decide"
Line 292: "the the" -> "the"
Line 370: "Thus, VaPRL can generates" -> "Thus, VaPRL generates"

The results are important for real robotic reinforcement learning. Training without resets is critical in many robotic tasks. The proposed approach appears efficient which is validated in three benchmark tasks.

Update:

I was satisfied with the rebuttal and will keep my score.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces the definition of Persistent Reinforcement Learning (PRL), and an algorithm to solve a goal conditioning version of PRL - Value-Accelerated PRL (VaPRL) - based on two ideas: separate the training and evaluation MDPs; and build a curriculum of (easier to solve) MDPs. 

The main idea behind the paper is to solve episodic RL tasks by minimizing the number of resets to the initial state, which is particularly important in real-work RL. This is achieved by using a generator of surrogate goals instead of solving directly for the initial task goal, and by using an initial state distribution different from the evaluation MDP. This initial state distribution is at the beginning of training closer to the goal, and during training it is expanded such that it approaches the initial state distribution of the evaluation MDP.

The experimental results empirically show the claimed benefits, and are compared with other algorithms that do not use any curriculum or have access to resetting to the initial state distribution.
main_review: The motivation for this work is very important for real-world reinforcement learning, especially in robotic tasks where resetting to an initial state distribution includes some form of human intervention, either by manually resetting the system, or by engineering a solution to do it automatically.

The paper is well written, organized, easy to follow and read.

Although I like this work, my opinion is that it lacks a more theoretical treatment of the problem and that some claims can be difficult to place in practice in a real system. Nevertheless, I see a good potential for this idea, but I think it is better suited to a robotics conference.

The following are a couple of comments and questions.

Method
----------
A central idea of the paper is to target the sparse reward episodic MDP setting for real scenarios, where the solution of the problem is partly known to humans, e.g., picking up an object. To solve such tasks, the authors claim that placing the agent near the goal is beneficial. While I agree with this statement, there are two concerns. First, for many tasks it is difficult to define what “close to the goal” really means. Second, there are tasks where placing the robot close to the goal is extremely difficult. Take the example of a robot hitting a hockey puck. The reward in this task is sparse and equal to placing the puck in a radius of a given location. One can place the puck near the goal, but if the hockey stick cannot reach it, then it will not be able to solve the task. 
“The objective of this goal generator is to recover the optimal policy with respect to J_E efficiently training in an environment with H_T >> H_E”.
If the training environment is different from the evaluation environment, how do you guarantee that the return is maximized in the evaluation environment? From my understanding, the authors define a proxy based on the curriculum training, but unfortunately there are no theoretical guarantees that training on the training MDP will lead to an optimal performance in the evaluation MDP.
By optimizing the policy locally, i.e., from the initial state given by the curriculum to a generated goal, is the trajectory generated by the global policy optimal? Or is it piecewise optimal? 
A big focus of the paper is on constructing the curriculum of starting states. Even though it is a good idea, it is in my opinion done in a very heuristic way, and without any guarantees that following this curriculum improves the returns in the evaluation environment.
In the main paper, for the constraint V(s,g) >= epsilon, the authors defined it as belonging to the [0, 1] interval, but in the experiment for hand manipulation, epsilon is out of this range (-300). Could you comment on this?  How to choose epsilon? Is it just a hyperparameter? Can it happen that a “bad” epsilon leads to an underperforming policy?
Goal relabeling is important to reduce the space of possible curriculum goals, and it is solved by sampling N goals at random. My question is again if this choice ensures that we obtain an optimal policy with respect to the evaluation MDP?


Experiments
-----------------
The experiments are well constructed and show the potential of the algorithm in sparse reward environments.
While I understand that NeurIPS is not a robotics conference, it would be good to see how this method applies in reality, especially how easy it is to place the agent near the goal state for a more intricate task.


Citations
------------
The authors seem to be citing a reasonable amount of work from a specific group.
Even though this work is not set in the hierarchical RL framework, it has also some connections to the idea of skill chaining, which although not directly matching this paper it could also be taken into consideration - Konidaris, G. et al, Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining, NIPS, 2009

Typos
Eq. (4), the state s is missing under the argmin

limitations_and_societal_impact: Not applicable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on ‘reset-free’ RL, i.e., the aim to eliminate the need for RL agents to be reset to one of a number of states from a start state distribution. The proposed a curriculum inspired approach, where the agent resets to progressively harder starting states, that are further away from the goal. They first formally define the persistent reinforcement learning problem. Then they specify a new method to solve this problem (VaPRL), which indeed outperforms baselines in three simulation experiments. 
main_review: Strong:
* Good formal definition of the type of problem. Although it is close to continual learning, I do agree with the authors that the setting is slightly different, and it is useful to identify this.
* The method is clear and works better than the baselines. 

Comments:
* Does the epsilon specification scale with funtion approximation? Then, it may become really hard to trust the epsilon estimate.
* It is intuitively a bit strange that you generate the curriculum by moving start states backwards from the goal, but to find out which state is a good next candidate as start state (using the distance function), you already use V(s,s_0), i.e., the value to get there from the start state. This is kind of the thing you wanted to eliminate with the curriculum in the first place, i.e., only focus on the easier last part first. 	
* Sec 4.2: I had slight issues understanding what’s going on here. First, how do you know all the elements in the goal space (distribution), are they explicitly given? Second, you start the section about C(g), but then you only relabel the goals right, not the start states? (and why not?)
* Algorithm 1: I would comment again that the switching happens within G(s,g_p), this confused me at first.
* I have some issues with the fact that you give demonstration trajectories to guide initial exploration (L289-305). You wanted to eliminate manual intervention, yet now you start from good trajectories. It is hard for me to judge how optimal these trajectories are, and thereby judge to what extend you are only inpainting/working backwards along an already optimal path with your method.
* There is no discussion at all, which I think is a problem. You should reflect a bit on your method, identify its potential problems and pitfalls, what did you encounter during experimentation. 

Conclusion:
This is a decent paper. The problem is relevant, notation and methods are clear, and results are good. My main issue is the use of expert demonstration trajectories to warm-start the algorithm, which I find really hard to judge because I do not know the quality of these demonstration trajectories. They may have a major impact, but the authors do not report on this ablation. A bit more discussion would be helpful al well, but in general this is a good paper with clear contribution, good structure and writing, and good results. 
limitations_and_societal_impact: The authors state there are no explicit negative potential impacts of this work (beyond the general ones of RL), I agree
review_title: null

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The reviewers did not reach a consensus. In particular, the lack of a theoretical analysis together with some doubts about its applicability to realistic scenarios have been considered as significant limitations.
Nonetheless, the majority of the reviewers were satisfied with the answers provided by the authors and think that the approach proposed in the paper is interesting and promising.
Overall, I think that, despite its limits, the paper provides a nice contribution and I propose to accept it.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)