Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper solves the problem of how to do autonomous resets, which is an important problem in real world RL. The method is novel, the explanation is clear, and has good experimental results.
 
Pros:
1. The approach is simple, solves a task of practical importance, and performs well in the experiments. 
2. The experimental section performs good ablation studies wrt fewer reset thresholds, reset attempts, use of ensembles.

Cons:
1. The method is evaluated only for 3 tasks, which are all in simulation, and on no real world tasks. Additional tasks could be useful, especially for qualitative analysis of the learned reset policies.
2. It seems that while the method does reduce hard resets, it would be more convincing if it can solve tasks which a model without a reset policy couldnt. Right now, the methods without the reset policy perform about equally well on final reward.
3. The method wont be applicable to RL environments where we will need to take multiple non-invertible actions to achieve the goal (an analogy would be multiple levels in a game). In such situations, one might want to use the reset policy to go back to intermediate “start” states from where we can continue again, rather than the original start state always.

Conclusion/Significance: The approach is a step in the right direction, and further refinements can make it a significant contribution to robotics work.

Revision: Thanks to the authors for addressing the issues I raised, I revise my review to 7
review_title: Simple, practical approach for autonomous resets; good experimental results and ablation studies, but no real world tasks.

final_score: rating: 5: Marginally below acceptance threshold
review_content: (This delayed review is based on the deadline version of the paper.)

This paper proposes to learn by RL a reset policy at the same time that we learn the forward policy, and use the learned reset Q-function to predict and avoid actions that would prevent reset — an indication that they are "unsafe" in some sense.

This idea (both parts) is interesting and potentially very useful, particularly in physical domains where reset is expensive and exploration is risky. While I'm sure the community can benefit from ideas of this kind, it really needs clearer presentations of such ideas. I can appreciate the very intuitive and colloquial style of the paper, however the discussion of the core idea would benefit from some rigor and formal definitions.

Examples of intuitive language that could be hiding the necessary complexities of a more formal treatment:

1. In the penultimate paragraph of Section 1, actions are described as "reversible", while a stochastic environment may be lacking such a notion altogether (i.e. there's no clear inverse if state transitions are not deterministic functions).

2. It's not clear whether the authors suggest that the ability to reset is a good notion of safety, or just a proxy to such a notion. This should be made more explicit, making it clearer what this proxy misses: states where the learned reset policy fails (whether due to limited controllability or errors in the policy), that are nonetheless safe.

3. In the last paragraph of Section 3, a reset policy is defined as reaching p_0 from *any* state. This is a very strong requirement, which isn't even satisfiable in most domains, and indeed the reset policies learned in the rest of the paper don't satisfy it.

4. What are p_0 and r_r in the experiments? What is the relation between S_{reset} and p_0? Is there a discount factor?

5. In the first paragraph of Section 4.1, states are described as "irreversible" or "irrecoverable". Again, in a stochastic environment a more nuanced notion is needed, as there may be policies that take a long time to reset from some states, but do so eventually.

6. A definition of a "hard" reset would make the paper clearer.

7. After (1), states are described as "allowed". Again, preventing actions that are likely to hinder reset cannot completely prevent any given state in a stochastic environment. It also seems that (2) describes states where some allowed action can be taken, rather than states reachable by some allowed action. For both reasons, Algorithm 1 does not prevent reaching states outside S*, so what is the point of that definition?

8. The paper is not explicit about the learning dynamics of the reset policy. It should include a figure showing the learning curve of this policy (or some other visualization), and explain how the reset policy can ever gain experience and learn to reset from states that it initially avoids as unsafe.

9. Algorithm 1 is unclear on how a failed reset is identified, and what happens in such case — do we run another forward episode? Another reset episode?
review_title: Interesting idea

final_score: rating: 6: Marginally above acceptance threshold
review_content: If one is committed to doing value-function or policy-based RL for an episodic task on a real physical system, then one has to come up with a way of resetting the domain for new trials.  This paper proposes a good way of doing this:  learn a policy for resetting at the same time as learning a policy for solving the problem.  As a side effect, the Q values associated with the reset policy can be used to predict when the system is about to enter an unrecoverable state and "forbid" the action.

It is, of course, necessary that the domain be, in fact, reversible  (or, at least, that it be possible to reach a starting state from at least one goal state--and it's better if that goal state is not significantly harder to reach than other goal states.

There were a couple of places in the paper that seemed to be to be not strictly technically correct.

It says that the reset policy is designed to achieve a distribution of final states that is equivalent to a starting distribution on the problem.  This is technically fairly difficult, as a problem, and I don't think it can be achieved through standard RL methods.   Later, it is clearer that there is a set of possible start states and they are all treated as goal states from the perspective of the reset policy.   That is a start set, not a distribution.  And, there's no particular reason to think that the reset policy will not, for example, always end up returning to a particular state.

Another point is that training a set of Q functions from different starting states generates some kind of an ensemble, but I don't think you can guarantee much about what sort of a distribution on values it will really represent.   Q learning + function approximation can go wrong in a variety of ways, and so some of these values might be really gross over or under estimates of what can be achieved even by the policies associated with those values. 

A final, higher-level, methodological concern is that, it seems to me, as the domains become more complex, rather than trying to learn two (or more) policies, it might be more effective to take a model-based approach, learn one model, and do reasoning to decide how to return home (and even to select from a distribution of start states) and/or to decide if a step is likely to remove the robot from the "resettable" space.

All this aside, this seems like a fairly small but well considered and executed piece of work.  I'm rating it as marginally above threshold, but I indeed find it very close to the threshold.
review_title: A strategy for learning to self-reset in episodic tasks as well as to avoid some types of bad failure

final_score: rating: 7: Good paper, accept
review_content: This paper proposes the idea of having an agent learning a policy that resets the agent's state to one of the states drawn from the distribution of starting states. The agent learns such policy while also learning how to solve the actual task. This approach generates more autonomous agents that require fewer human interventions in the learning process. This is a very elegant and general idea, where the value function learned in the reset task also encodes some measure of safety in the environment.

All that being said, I gave this paper a score of 6 because two aspects that seem fundamental to me are not clear in the paper. If clarified, I'd happily increase my score.

1) *Defining state visitation/equality in the function approximation setting:* The main idea behind the proposed algorithm is to ensure that "when the reset policy is executed from any state, the distribution over final states matches the initial state distribution p_0". This is formally described, for example, in line 13 of Algorithm 1.
The authors "define a set of safe states S_{reset} \subseteq S, and say that we are in an irreversible state if the set of states visited by the reset policy over the past N episodes is disjoint from S_{reset}." However, it is not clear to me how one can uniquely identify a state in the function approximation case. Obviously, it is straightforward to apply such definition in the tabular case, where counting state visitation is easy. However, how do we count state visitation in continuous domains? Did the authors manually define the range of each joint/torque/angle that characterizes the start state? In a control task from pixels, for example, would the exact configuration of pixels seen at the beginning be the start state? Defining state visitation in the function approximation setting is not trivial and it seems to me the authors just glossed over it, despite being essential to your work.

2) *Experimental design for Figure 5*: This setup is not clear to me at all and in fact, my first reaction is to say it is wrong. An episodic task is generally defined as: the agent starts in a state drawn from the distribution of starting states and at the moment it reaches the goal state, the task is reset and the agent starts again. It doesn't seem to be what the authors did, is that right? The sentence: "our method learns to solve this task by automatically resetting the environment after each episode, so the forward policy can practice catching the ball when initialized below the cup" is confusion. When is the task reset to the "status quo" approach? Also, let's say an agent takes 50 time steps to reach the goal and then it decides to do a soft-reset. Are the time steps it is spending on its soft-reset being taken into account when generating the reported results?


Some other minor points are:

- The authors should standardize their use of citations in the paper. Sometimes there are way too many parentheses in a reference. For example: "manual resets are necessary when the robot or environment breaks (e.g. Gandhi et al. (2017))", or "Our methods can also be used directly with any other Q-learning methods ((Watkins & Dayan, 1992; Mnih et al., 2013; Gu et al., 2017; Amos et al., 2016; Metz et al., 2017))"

- There is a whole line of work in safe RL that is not acknowledged in the related work section. Representative papers are:
    [1] Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh: High-Confidence Off-Policy Evaluation. AAAI 2015: 3000-3006
    [2] Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh: High Confidence Policy Improvement. ICML 2015: 2380-2388

- In the Preliminaries Section the next state is said to be drawn from s_{t+1} ~ P(s'| s, a). However, this hides the fact the next state is dependent on the environment dynamics and on the policy being followed. I think it would be clearer if written: s_{t+1} ~ P(s'| s, \pi(a|s)).

- It seems to me that, in Algorithm 1, the name 'Act' is misleading. Shouldn't it be 'ChooseAction' or 'EpsilonGreedy'? If I understand correctly, the function 'Act' just returns the action to be executed, while the function 'Step' is the one that actually executes the action.

- It is absolutely essential to depict the confidence intervals in the plots in Figure 3. Ideally we should have confidence intervals in all the plots in the paper.
review_title: Great idea but the write up needs to be made clearer

Meta-review: This paper is an easy accept -- three reviewers have above threshold scores, while one reviewer is slightly below threshold, but based on the submitted manuscript.  It appears that the paper has substantially improved based on reviewer comments.

Pros:

All reviews had positive sentiment: "very elegant and general idea" (Reviewer4); "idea is interesting and potentially very useful" (Reviewer2); "method is novel, the explanation is clear, and has good experimental results" (Reviewer3); "a good way to learn a policy for resetting while learning a policy for solving the problem.  Seems like a fairly small but well considered and executed piece of work." (Reviewer1)

Cons:

One reviewer found that testing in only three artificial tasks was a limitation.

The initial reviews noted several issues where clarification of the text and/or figures was needed.  There were also a bunch of statements where the reviewers questioned the technical correctness / accuracy of the discussion.  Most of these points appear to have been adequately addressed in the revised manuscript.
Decision: Accept (Poster)