Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
[Summary]
This paper proposes POLISH, an imitation learning algorithm that provides a balance between Behavioral Cloning (BC) and DAgger. The algorithm reduces the mismatch between the target policy and an expert policy on states obtained from starting at the target policy's state distribution and following the expert policy for a time segment of t steps. The claim is that a suitable t will keep the training states close to the target policy's state distribution and avoid the compounding errors that arise when the agent drifts away from its training distribution. The paper also explores the possibility of policy optimization by replacing the pre-defined expert policy in POLISH with a policy derived from Monte Carlo Tree Search. Theoretical and empirical analyses in the paper studies the effect of t and MCTS planning in POLISH on policy improvement.

[Decision]
A clear study of the dimension between BC and DAgger is a useful contribution to the literature and an algorithm that effectively solves the distributional shift problem in BC will be of high practical value. However, the results in this paper do not support the claim that a reasonable time segment length in POLISH alleviates this problem. The theory shows a bound on the performance of the target policy that varies with t, but it is not clear if a suitable t is better than the two extremes, i.e., BC and DAgger. The experiments section is limited and, on two out of the three tasks, there is no considerable difference between the performance of POLISH, BC, and DAgger. I am leaning towards rejecting this paper.

[Explanation]
In Section 5, Theorem 1 shows the effect of t, the length of time segments, on the performance and the target policy by providing a bound. If this theorem is motivating a middle ground between BC and DAgger, it needs to show that a value of t other than the extremes will maximize the bound. The bound in the paper consists of a positive term and a negative term, both of which grow with t. It is then concluded that a balance point will maximize the overall bound. I do not see how it follows that this balance point is a middle ground and not an extreme value. If, for example, the negative term grows faster than the positive term, then DAgger (t=1) will be have the best performance according to this theorem.

It is not clear how the bound in Theorem 1 is comparing the performance of algorithms with different values of t. For a fixed policy, one can obtain different bounds by choosing different values of t. These bounds will have different values of \epsilon. The state distribution for \epsilon is the distribution of states visited in a limited time segment (which depends on the target policy, the expert policy, and the segment length). Using \epsilon (or \epsilon_i) in the bound drops the relationship between \epsilon and the length of time segments, and hides the fact that different algorithms are minimizing different errors. 

The equality in Eq 2 is not obvious to me. J(\pi) is and expectation under the discounted visit distribution. For example, if gamma is small, then the states in the start state distribution will have a higher weight in J(\pi) while the right-hand side sums over the expected performance in all time segments equally. I believe the later terms in the sum should also be discounted.

Does changing t also affect the MCTS step and the expert policy obtained from it? If so, it is possible that a suitable t will result in better performance because the MCTS step finds a stronger expert policy, and not because the imitation learning step better reduces the error.

Does the MCTS step use the perfect simulator or a learned model in the experiments? If POLISH, unlike PPO, has access to the MDP, the comparison of these two methods is not fair.

In Fig 2 (a) and (c), the curves for t=1, t=32, and t=1000 overlap through most of the training process. This is not conclusive evidence that a sweet spot for t results in better performance. An experiment on a simpler setting with more runs may elucidate the effect of t on the performance.

In 6.3, what does the reward improvement after running MCTS with the current policy precisely mean, and how does this correspond to the first term in the bound, i.e., the sum of the expected performance of \pi_* over time segments?


[Minor comments]
- I suggest adding the process of obtaining an expert policy through MCTS to Algorithm 1. It is hard to understand the process without a clear step-by-step description.
- How is t=32 chosen for the experiments in Fig 2?
-------------------
After rebuttal: I have read the authors' response and the other reviews. The rebuttal addresses my questions and concerns about clarity of presentation. However, I am still not convinced by the evidence in this paper. On two out of the three environments, the performance of the three values of t is not much different through the training and this is not because of the advantage over PPO. On Ant, for example, the curves for t=32 and t=1000 are not even one std apart through most of the training.

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper was confusing and difficult to read.  I think it is trying to devise a methodology that improves upon a given expert policy, but I am not confident whether if this is the objective.  The main contribution of the paper is the algorithm POLISH on page 5.  However, it was unclear how MCTS with UCT is used with the expert policy in the algorithm.  The first mention of the 0-1 loss objective is in section 4.1, and Algorithm 1 on page 5 claims to minimize this loss on line 15.  However, in the experiment, the loss function is then switched to another L(D, \pi) = D_{KL}(\pi || \pi*) + H(\pi) + Lv(D, \pi).  Why the switch?  And what is the definition of the third term?  By examining the experimental results, Algorithm 1 seems to outperform PPO.  Since the expert policy is known and given to Algorithm 1, I speculate Algorithm 1 is only replicating the expert policy, which would have outperformed PPO that has to learn from scratch.  Thus, the comparison does not seem fair.  It would be interesting to see how POLISH compares with the expert policy.

Other comments:

* Page 1 in Introduction: “However, these models suffer from the problem that even small difference between the learned policy and the expert behavior can lead to a snow-balling effect, where the state distribution diverges to a place where the behaviour of the policy is now meaningless since it was not trained that part of space”.  Do you mean that the algorithm diverges instead of the state distribution diverges?   The agent may incur errors in a space that has not been observed, but it should be able to learn eventually.  So, what is causing divergence in states that have not yet encountered?  

* Page 4 in The POLISH Algorithm Main Algorithm: the 0-1 loss function is defined to be “L(D, \pi) = 1/|D| \sum_{s,a*}\in D (I(\pi(s) \neq a*)), where a* is the expert policy’s selected action.  I think it makes more sense to write “L(D, \pi)" as 1/|D| \sum_{s,a*}\in D I ( a \neq a* : a ~ \pi(s)).  Also, in RL, we don't say that the policy receives a reward, but rather the agent receives a reward. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes POLISH, a reinforcement learning learning algorithm based on imitating partial trajectories produced by an MCTS procedure. The intuition behind this idea is that behavioral cloning suffers from distribution shift over time, and using MCTS allows imitation learning to be done on states closer to the policy's state distribution, which the authors justify using techniques similar to DAgger. The authors evaluate this method on continuous OpenAI Gym tasks, and show that it consistently beats a PPO baseline.

Overall, my decision for this paper errs on the side of reject. This primarily comes from the fact that the writing is unclear to me, and this algorithm needs both access to an expert and a reward function, which is a setting that I'm not sure is very applicable in practice. Additionally, the experimental results seem fairly weak. In the imitation learning setting, there appears to be little difference between behavioral cloning, DAgger, and an intermediate segment length. In the reinforcement learning setting, the PPO baseline seems to be unfair, which I detail below. 

The writing is confusing to me as it mixes two seemingly distinct problem settings (imitation learning and reinforcement learning) and interferes with my full understanding of the motivation of the paper. My current guess is that this paper is primarily aimed towards policy optimization in a reinforcement learning setting. The algorithm can start from scratch with a random initial policy, and optimize the policy to maximize total returns. However, much of the paper is written as if the setting were imitation learning, where expert advice is available. If this paper is primarily aimed at imitation learning paper, I am unsure of the advantage of using this method over DAgger. My hypothesis is that the primary use-case of POLISH over DAgger is when the provided expert is suboptimal, and using MCTS allows the policy to improve beyond the expert. However, this point is not provided in the paper.

For the experiments, I'm not sure that PPO is a directly comparable baseline. 
1) Were queries to the simulator used during MCTS accounted for when measuring sample complexity? (Figure 2). Being able to query the environment without cost is a significant advantage to POLISH in terms of sample complexity. 
2) Additionally, it was stated that POLISH had access to a pre-trained policy, which is additional information that PPO cannot exploit. A reasonable comparison could be to initialize the PPO agent from that pre-trained policy, or to not give POLISH access to the pre-trained policy.

For related work, I would argue that an important class of algorithms to mention are RL methods based on imitating some sort of policy improvement procedure. This includes work such as (not exhaustive) self-imitation learning (Oh 2019), the cross-entropy method, guided policy search (Levine 14), reward-weighted regression (Peters 07) and UREX (Nachum 17).
review_title: Official Blind Review #1

Meta-review: Thanks for your detailed responses to the reviewers, which helped us a lot to better understand your paper.
However, given that the current manuscript still contains many unclear parts, we decided not to accept the paper. We hope that the reviewers' comments help you improve your paper for potential future submission.
Decision: Reject