Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a method for using an ensemble of deep reinforcement learning policies, where members of the ensemble are periodically updated to imitate the most promising member of the ensemble. Thus learning proceeds by performing off policy reinforcement learning updates for each individual policy, as well as some supervised learning for inter-policy imitation learning.

I start by what I view as the positive aspects about the paper:
1- The algorithm is quite simple (to understand and to implement).
2- Experimental results are performed on a variety of domains, and more importantly, each experiment is motivated by a question.

That said, I have some concerns about this paper which I list below:

1- Perhaps my biggest concern is that the approach is not motivated from a theory stand point. There has been interesting results in Osband's work [Osband, 2016] (and references therein) for randomized value functions which can serve as a foundation for this work. That said, a) Osband's results, at least immediately, are related to value-function based methods, as opposed to policy gradient b) the KL update which one could argue is the main and only significant contribution of the paper, is not justified by Osband or any other prior work c) there is not anything that this paper adds to the literature to better justify diversity through randomization and/or imitation learning based on the best member of the ensemble.

2- I have found various claims in the paper which are unclear, scientifically not true, or sometimes even contradicting. In Introduction, for example, the authors mention that the agent sometimes gets into a sub-optimal policy and may require a large number of interactions before escaping the sub optimal policy. How does gathering more data help to improve the policy? Either we are in a local maximum, which if we are doing gradient ascent, there is really not much we could do, or that we are in a saddle point, which we can escape by adding some noise to the gradient. [Jin,2017]

3- In section 4.3 the authors talk about on-policy methods requiring importance sampling (IS) ratios. To the best of my knowledge, IS is only used for off-policy learning. Can the authors provide a link to an on-policy method that does IS?

4- Again in section 4.3 authors claim and I quote "Using off-policy methods, all the policies in the ensemble can easily be updated, since off-policy update methods can perform updates from any \tau". But later on in Section 5.3 authors claim that "off-policy actor-critic methods (e.g. SAC) cannot fully utilize the other agent's or past experience." So which statement is true?

5- Again, the KL update is interesting, but is it even surprising that the KL update is necessary for an ensemble of policies updates using policy gradients? In the absence of this KL update, which the authors characterize as the method that Osband proposed, the policies could generally be arbitrarily far from one another. This means that each policy needs to perform policy evaluation using trajectories that are coming from other policies who in principle can be radically different than the policy we want to update. This means that updates will be quite "off-policy" which we know can really degrade the quality of the estimated gradient. This is perhaps why even choosing a random policy to update towards is providing "some" improvement. I think this is the real insight, but it is not really discussed at all in the paper.

6- On the same note, I do not think that one can say Osband's method is the same as CIKD but only without the KL update. Most notably, Osband's work was presented for value-function-based methods like DQN. These methods work fundamentally different than policy gradient methods, which rely on (near) on-policy updates to perform good policy improvements. In that sense, the presented results make sense, but I disagree with the framing of the results and how they are presented here.

7- In section 5.3, when the authors utilize more policy updates to have a fair comparison, are they retuning hyper parameters? Surely they need to do that, at least for hyper-parameters that are known to be super important such as the step size.

8- Overall I liked section 5.5 that is trying to dissect causes for improvement. However, it seems like that the "dominant agent" hypothesis has been rejected hastily, unless I misunderstood the experiment. The authors show that the notion of best is spread across different agents. But of course this will be the case in light of the KL update, since the policies are getting closer to one another. Can you redo the experiment in the absence of the KL update?

9- Have the authors thought about any connection between this and genetic algorithms? In genetic algorithms, the idea is the next set of candidates are chosen based on the most promising candidates in the current iteration. CIKD seems like a soft implementation of this idea.

In light of the comments above, I am voting for weak rejection, though as I said before, I do see some interesting things in this paper. I encourage the authors to think about CIKD from a theoretical lens in the future.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes an RL training procedure that maintains an ensemble of k policies and periodically pushes all the policies to be closer to the best performing one. The formulation, experiments and analysis are very clear and show a mild improvement over using the same underlying RL algorithm without the imitation part. The idea is close to many other proposed in the literature, but to my knowledge it is the first time this exact procedure is studied in detail.

The first piece of their approach is an off-policy RL algorithm. In their case, they use SAC. The second piece is adding an ensemble of policies (3 in their case), and randomly selecting one of them every time a rollout is collected, and using the collected rollout to update all the policies. This effectively implies 3 times more overall gradient updates compared to SAC. They call this ablation SAC-ensemble. Interestingly they only use the most recently collected trajectories to update all policies, and despite storing the rollouts in a replay buffer, they seem to only use the stored transitions for the imitation part described below. Some of their experimental results uses extra gradient steps, although it’s not clear if those gradient steps are also only on the last rollout collected, or on transitions sampled from the replay buffer as it is typical in off-policy RL methods. In general, I think the work could improve with more details about how much the policy training could improve by increasing the number of gradient steps on the full replay buffer.

The final piece of their method is selecting the best performing policy (or “teacher”) of the ensemble based on the recent experience, and update all other policies by executing some gradient steps on the KL divergence between them and the current “teacher”. They also try an experiment where the “teacher” is selected randomly, and it does surprisingly well in my opinion (specially realizing that the “HalfCheetah” experiments seem to not have all seeds run to convergence, please report the full results). I suspect that most of the benefit of their method comes from randomly perturbing the parameters of the policies in the ensemble. More thorough and careful experimentation needs to be carried out to investigate this direction. This is in fact not very surprising given the results of Evolutionary Strategy methods, or Population-based training (even if usually used for hyper-parameters adaptation).

Furthermore, the authors only run the environments for 1M steps, whereas in previous works some environments are shown to get higher return after more training steps. I would also encourage the authors to report the results in all the standard MuJoCo benchmarks for the ablations (even if it’s in the appendix) to better asses their claims.

Overall, this is a very well presented work, although it lacks some novelty and a few more thorough experiments to fully understand the improvements they show. I think this idea is worth sharing with the community, and I recommend a weak accept.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary:
This paper proposed an ensemble method (CIKD) that train multiple agents and
use knowledge distillation to transfer knowledge from the current best agent to
sub-optimal agents periodically.  According to the reported results, CIKD is a
simple yet effective approach to improve sample-efficiency and final performance.  
The experimental results are sufficient, and the ablation studies are conducted thoroughly. It is shown that both selecting the best agent and using KD to
transfer knowledge are effective comparing to other naive alternatives. 


I recommend the acceptance of this paper. 

The paper proposed a novel approach (CIKD) to improve the sample-efficiency of the state-of-the-art. The proposed ensemble approach is aligned with our intuition, and it is effective. The authors proposed to train several agents at the same time and randomly select one of
the agents as a behavior policy during each rollout. Then the collected trajectory is used to update the policy of all agents. Meanwhile,
they keep tracking the performance of each agent and use the current best agent to conduct knowledge distillation to other agents periodically. 

This paper first conducts experiments to show when consolidating
the SAC with CIKD, both of the final performance and sample-efficiency can be improved. Then a set of ablation studies verified the best agent selection strategy, and the knowledge distillation
strategy is necessary for the ensemble method. 


Investigation on the reasons for improvement:
Though extensive ablation studies have shown the effectiveness
of each component of CIKD. It is still not clear why this approach
can be effective. 
Intuitively, it is possible that the exploration from a set of agents would outperform
a single agent. The measure of exploration efficiency could help in explaining the results. Furthermore, better exploration not necessarily
leads to better performance and sample-efficiency. Does knowledge distillation serve as a better alternative to exploit existing data? 

Model/algorithm agnostic
The proposed method is more convenient to be applied with off-policy approach when the policy is in the form of softmax. Is it also applicable
to other approaches? 

Experiments:
How do you determine when to stop the KD process? As mentioned in section 5.5, if we conduce KD fully, all students would be just imitating
the teacher's behavior. It seems the key is to tune a good termination
threshold for each task? Are there any guidelines to set up this threshold?
Do you have some automatic way to terminate the KD procedure?


Minor:
L1, P5, "how to CIKD improves the sample efficiency" 


review_title: Official Blind Review #3

Meta-review: The paper introduces an ensemble of RL agents that share knowledge amongst themselves. Because there are no theoretical results, the experiments have to carry the paper.  The reviewers had rather different views on the significance of these experiments and whether they are sufficient to convincingly validate the learning framework introduced. Overall, because of the high bar for ICLR acceptance, this paper falls just below the threshold. 

Decision: Reject