Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a new method for doing reverse curriculum training for reinforcement learning tasks with deterministic dynamics, a desired goal state at which reward is received, and the ability to teleport to any state. This covers a number of important cases of interest, including all simulated domains, and a number of robotics applications. The training proceeds in phases, where in each phase the initial starting set of states is expanded. The initial set of states used is close to the desired state goal. Each phase is initiated when 80% of the states in the current phase can reach the goal. Once the initial set of start states overlaps with the desired initial set of states for the task, training can terminate. During the training in a single phase, the algorithm uses a shaping reward (the tendency) which is based on a binary classifier that predicts if it will be possible to reach the goal from this state. This reward is combined in a hybrid reward signal. The authors suggest the use of a small number of checkpoints to guide the backwards state expansion to improve the search efficiency. Results are presented on several domains: maze, Super Mario, and Mujoco domains. 

The topic of doing more sample efficient training is important and interesting, and the subset of settings the authors consider is still a good set. 

The paper was clearly written though some details were relegated to the appendix which would’ve been useful to see in the main text.

I’m not yet convinced about this method for the desired setting in terms of significance and quality.

An alternative to using tendency shaping reward would be (during phase expansion) make the new “goal” states any of the states in the previous phase of initial states P_{i} that did reach the goal. This should greatly reduce the decision making horizon needed in each  phase. Since the domain is deterministic, as soon as one can reach one of those states, we have a path to the goal. If we care about the number of steps to reach the goal (vs finding any path), then each of the states in P_{i} for which a successful path can be achieved to the goal can also be labeled by the cost / number of time steps to reach the goal. This should decompose the problem into a series of smaller problems. Perhaps I’m missing something-- could the authors please address this suggestion and/or explain why this wouldn’t be beneficial?

The authors currently use checkpoints to help guide the search towards the true task desired set of initial states. If those are lacking, it seems like the generation of the new P_{i+1} could be biased towards that desired set of states. One approach could be to randomly roll out from the start state and then bias P_{i+1} towards any states close to states along such trajectories. In general one could imagine a situation in which one both does forward learning/planning from the task start state and backwards learning from the goal state to obtain a significant benefit, similar to ideas that have been used in robot motion planning.

Why learn from pixels for the robot domains considered? Here it would be nice to compare to some robotics approaches. With the action space of the robot and motion planning, it seems like this problem could be tackled using existing techniques. It is interesting to have a method that can be used with pixels, but in cases where there are other approaches, it would be useful to compare to them.

Small point
D.2 Why not compare to GAIL instead? 

review_title: interesting direction, questions about the proposed approach

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors extend the approach proposed in the  "Reverse Curriculum Learning for Reinforcement Learning" paper by adding a discriminator that gives a bonus reward to a state based on how likely it thinks the current policy is to reach the goal from said state. The discriminator is a potentially interesting mechanism to approximate multi-step backups in sparse-reward environments. 

The approach of this paper seems severely severely limited by the assumptions made by the authors, mainly assuming a deterministic environment, known goal states and the ability to sample anywhere in the state space. Some of these assumptions may be reasonable in domains such as robotics, but they seem very restrictive in the domains like the games considered in the paper.


Additional Comments:

-The authors demonstrate some benefits of using Tendency rewards, but made little attempt to explain why it leads to accelerated learning. Results are pure performance results.

-The authors should probably structure the tendency reward as potential based instead of using the Gaussian kernel hack they introduce in section 4.2

- Presentation: There are several mistakes and formatting issues in References

- Assumption 2 transformations -> transitions?

-Need to add assumption 3: advance knowledge of goal state

- the use of gamma as  a scale factor in equation 2 is confusion, it was already introduced as the discount factor ( which is default notation in RL). It also isn't clear what the notation r_f denotes (is it the same as r^f in appendix?).

-It is nice to see that the authors compare their method with alternative approaches. Unfortunately, the proposed method does not seem to offer many benefits. 

review_title: Interesting idea, but approach seems limited. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new method for reverse curriculum generation by gradually reseting the environment in phases and classifying states that tend to lead to success. It additionally proposes a mechanism for learning from human-provided "key states".

The ideas in this paper are quite nice, but the paper has significant issues with regard to clarity and applicability to real-world problems:
First, it is unclear is the proposed method requires access only high-dimensional observations (e.g. images) during training or if it additionally requires low-dimensional states (e.g. sufficient information to reset the environment). In most compelling problems settings where a low-dimensional representation that sufficiently explains the current state of the world is available during training, then it is also likely that one can write down a nicely shaped reward function using that state information during training, in which case, it makes sense to use such a reward function. This paper seems to require access to low-dimensional states, and specifically considers the sparse-reward setting, which seems contrived.
Second, the paper states that the assumption "when resetting, the agent can be reset to any state" can be satisfied in problems such as real-world robotic manipulation. This is not correct. If the robot could autonomously reset to any state, then we would have largely solved robotic manipulation. Further, it is not always realistic to assume access to low-dimensional state information during training on a real robotic system (e.g. knowing the poses of all of the objects in the world).
Third, the experiments section lacks crucial information needed to understand the experiments. What is the state, observation, and action space for each problem setting? What is the reward function for each problem setting? What reinforcement learning algorithm is used in combination with the curriculum and tendency rewards? Are the states and actions continuous or discrete? Without this information, it is difficult to judge the merit of the experimental setting.
Fourth, the proposed method seems to lack motivation, making the proposed scheme seem a bit ad hoc. Could each of the components be motivated further through more discussion and/or ablative studies?
Finally, the main text of the paper is substantially longer than the recommended page limit. It should be shortened by making the writing more concise.

Beyond my feedback on clarity and significance, here are further pieces of feedback with regard to the technical content, experiments, and related work:
I'm wondering -- can the reward shaping in Equation 2 be made to satisfy the property of not affecting the final policy? (see Ng et al. '09) If so, such a reward shaping would make the method even more appealing.
How do the experiments in section 5.4 compare to prior methods and ablations? Without such a comparison, it is impossible to judge the performance of the proposed method and the level of difficulty of these tasks. At the very least, the paper should compare the performance of the proposed method to the performance a random policy.

The paper is missing some highly relevant references. First, how does the proposed method compare to hindsight experience replay? [1] Second, learning from keyframes (rather than demonstrations) has been explored in the past [1]. It would be preferable to use the standard terminology of "keyframe".

[1] Andrychowicz et al. Hindsight Experience Replay. 2017
[2] Akgun et al. Keyframe-based Learning from Demonstration. 2012

In summary, I think this paper has a number of promising ideas and experimental results, but given the significant issues in clarity and significance to real world problems, I don't think that the current version of this paper is suitable for publication in ICLR.

More minor feedback on clarity and correctness:
- Abstract: "Deep RL algorithms have proven successful in a vast variety of domains" -- This is an overstatement.
- The introduction should be more clear with regard to the assumptions. In particular, it would be helpful to see discussion of requiring human-provided keyframes. As is, it is unclear what is meant by "checkpoint scheme", which is not commonly used terminology.
- "This kind of spare reward, goal-oriented tasks are considered the most difficult challenges" -- This is also an overstatement. Long-horizon tasks and high-dimensional observations are also very difficult. Also, the sentence is not grammatically correct.
- "That is, environment" -> "That is, the environment"
- In the last paragraph of the intro, it would be helpful to more clearly state what the experiments can accomplish. Can they handle raw pixel inputs?
- "diverse domains" -> "diverse simulated domains"
- "a robotic grasping task" -> "a simulated robotic grasping task"
- There are a number of issues and errors in citations, e.g. missing the year, including the first name, incorrect reference
- Assumption 1: \mathcal{P} has not yet been defined.
- The last two paragraphs of section 3.2 are very difficult to understand without reading the method yet
- "conventional RL solver tend" -> "conventional RL tend", also should mention sparse reward in this sentence.
- Algorithm 1 and Figure 1 are not referenced in the text anywhere, and should be
- The text in Figure 1 and Figure 3 is extremely small
- The text in Figure 3 is extremely small



review_title: Review: Concerns with regard to clarity and real-world applicability

Meta-review: The paper proposes an extension to the reverse curriculum RL approach which uses a discriminator to label states as being on a goal trajectory or off the goal trajectory. The paper is well-written, with good empirical results on a number of task domains. However, the method relies on a number of assumptions on the ability of the agent to reset itself and the environment which are unrealistic and limiting, and beg the question as to why use the given method at all if this capability is assumed to exist. Overall, the method lacks significance and quality, and the motivation is not clear enough.

Decision: Reject