Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to learn to reach goals in an RL environment. The method is based on principles of imitation learning. For instance, beginning with an arbitrary policy that samples a sequence of state-action pairs, in the next iteration, the algorithm treats the previous policy as an expert by relabeling its ending state as a goal. The paper shows that the method is theoretically sound and effective empirically for goal-achieving tasks. 

The paper is relatively clear and experiments are okay. I would then recommend it is on the positive side of the borderline.

Comments:
* The method is interesting but is still an "RL" method. So it is really learning to reach the goal via "RL". Note that in the method, the algorithm is not doing effective exploration but just randomly explore until you collect sufficient data to solve for a new goal. 
* If you formulate the problem better, you can see that it actually has a reward: add an initial state s0; for each g sampled from p(g), transition s0 to an MDP with goal g. You can now do the usual RL algorithm in this new MDP. I would think you can also do model-based learning -- give the model a good representation and then use the policies to learn the dynamics. It may worth to compare your algorithm with these natural baselines.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper claims to do imitation learning without expert demonstration using trajectories that are generated by suboptimal policies from other tasks.

The point of having an expert demonstrator is to help narrow the search for an optimal policy.  By taking the expert demonstration knowledge out of learning, to me, this is not retaining the benefit of imitation learning.  Thus, the paper is not about imitation learning, but rather about an optimization method that reuses data generated from multiple tasks.   Reusing trajectory data generated from multiple tasks to learn a policy of another task is not a novel idea.  If we could save all of the data regardless of whether if an optimal policy generates them or not, why not use them?  Less useful data may still contain useful information.  The better question is how to use them to learn policy efficiently.  If the motivation is to use trajectories from suboptimal policies from other tasks without expert knowledge, then I fail to see the motivation and the novelty of this paper. 

The paper claims that the methodology self-supervises each action taken, judging how good it is for reaching a goal in the future without learning Q-values.  However, this was not realized.  The methodology gathers all trajectories that reach a goal into a set, and use behaviour cloning on the data of the set to learn a policy.  The sampled trajectories in the set could be suboptimal for reaching a goal, and there’s little evidence that optimizing J_GCSL(\pi) will learn an optimal policy based on these data.  Optimizing objective J_GCSL(\pi) also does not take the long term effect of actions into account.  The gathering of trajectories and identifying the trajectory as goal-reaching is already a costly step, where no learning happens.  RL, on the other hand, would gather the data incrementally, learn, and act right away.  Also, it seems that the algorithm would require human knowledge to discern a trajectory as goal-reaching or not, which is contrary to self-supervision.  
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This work presents the goal-conditioned supervised learning algorithm (GCSL), which learns goal conditioned policies using only behavioral-cloning of the agent's own actions.  The intuition behind the algorithm is the goal of an observed trajectory can be identified after the fact, by simply looking at the states reached during that trajectory.  GCSL treats each executed action as a sample from the expert policy conditioned on each of the states reached after that action is taken.  Given a distribution over goal states, GCSL alternates between executing its current goal-conditioned policy on randomly selected goals, and learning to imitate the generated actions conditioned on the states they actually reached.  Experimental results demonstrate superior performance against a base (non-goal conditioned) RL algorithm (TRPO), and against another approach to learning goal-conditioned polices (TD3-HER), on a relatively diverse set of control problems.

A major issue is that the proof of the main theoretical result appears to be wrong.  As there don't appear to be any constraints placed on the policy pi_old, it would seem that the surrogate loss would collapse to 0 for any policy pi if pi_old is such that the target goal is never reached (the probability of any trajectory t reaching g is 0 under pi_old(t|g)).  It seems to be the case that the quality of the GCSL loss depends on the relationship between pi_old and the goal distribution p(g).  The fact that the theoretical results are incorrect does not mean that the algorithm, or the general approach do not have value, but it does highlight the fact that this approach may only be effective for a specific class of problems similar to the experimental domains.

While not a flaw in the work itself, it should be made clear in the text that the notion of optimality for the learning tasks considered in this work (i.e. achieving the goal by the end of episode), avoids one of the apparent limitations of the algorithm.  A randomly generated trajectory is itself optimal for any state that it reaches, if we define optimality as simply reaching a state.  Such a trajectory may not be the most efficient way of reaching that state however, so the relabelling process would seem to be prone to learning policies that achieve the conditioned goals, but not doing so in an efficient manner.  It isn't clear how well this approach would work for tasks where the efficiency, in terms of the time required to reach the objective, is a key part of the evaluation.  Again, this is not a flaw in the work itself, and it is possible that the algorithm will be effective in such tasks, perhaps because the likelihood of an action resulting in a given state is higher if that action brings us closer to this state.  It might be useful to conduct some additional experiments where evaluation is based on the time required to solve a task, rather than just the accuracy of the final state.
review_title: Official Blind Review #1

Meta-review: The authors present an algorithm that utilizes ideas from imitation learning to improve on goal-conditioned policy learning methods that rely on RL, such as hindsight experience replay.  Several issues of clarity and the correctness of the main theoretical result were addressed during the rebuttal period in way that satisfied the reviewers with respect to their concerns in these areas.  However, after discussion, the reviewers still felt that there were some fundamental issues with the paper, namely that the applicability of this method to more general RL problems (complex reward functions rather than signle state goals, time ) is unclear.  The basic idea seems interesting, but it needs further development, and non-trivial modifications, to be broadly applicable as an approach to problems that RL is typically used on.  Thus, I recommend rejection of the paper at this time.
Decision: Reject