Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Overall, this is a very interesting paper and, I think, would make a great addition to ICLR.  I find the ideas discussed in the paper to be stimulating and important to the community.

The authors are addressing a frequently overlooked and under-discussed aspect of RL: namely the instability of training that occurs when a Q-function becomes decreasingly familiar with areas of a state space while it focuses on a particular "trajectory" or solution to a task.

The authors add a measurement of uncertainty which is a simple heuristic.  It does require a couple additional hyperparameters.  One of which is not investigated in the experiments.

The experiments are ample and the approach is compared to a baseline and an alternative approach.  The experiments are informative and show the superiority of the proposed approach.

Comments:

The paper initially left me with the impression that the members were trained separately but Algorithm 1 looks like they are trained together.  This would have a significant impact on the number of training time steps required for training.  Please clarify.

The literature review is good but must add two relevant works.  One is MMRL by Doya et al.  I think their notion of a forward model is similar in spirit so it should be discussed.  Also, see the dissertation by DL Elliott entitled: THE WISDOM OF THE CROWD: RELIABLE DEEP REINFORCEMENT LEARNING THROUGH ENSEMBLES OF Q-FUNCTIONS.

Section 3.3 didn't elucidate the topic for me.  It could be removed.  I think the description of the algorithm is sufficient.  Could be replaced by additional analysis of the uncertainty parameter from the experiments.  Also, what is the importance of the variable L in (5)?

What does i denote in (6)?  I assume it's the index of the ensemble member.  Also, mention how the uncertainty values are initialized here.

I feel that the paper is never really solidified in the mind until seeing (7).

Your experiments indicate that the uncertainty value causes the "preferred" ensemble member to switch during the game of breakout.  That makes a lot of sense given the cyclical, for lack of a better term, nature of the game.  How about for tasks like the bipedal walker or the cart-pole task?  Would you expect, when training is completed, for a single member to dominant from the start position to the end of the trial?  If so, that's a limitation of the approach that doesn't diminish the contribution in my opinion.

Would be interesting to see how the number of ensemble members used during a trial changes during training.  Does it increase/decrease?  Would love to see more analysis of this.  AGAIN, I don't think it's a negative thing if all are used but, eventually, one dominates.

How would using simpler Q-function approximators change the results?  Would it force them to decompose the task?
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: 1.	The paper suggests an ensemble method that leads to better performance over the execution of a single agent. The final (behavior) agent is a weighted average (or majority vote) of the members, where weights are determined by the accumulated TD error of the agent in the episode so far. The TD error serves as a confidence measure of the ensemble members in their predictions.
2.	Assume that one of the Q functions, Q_i is Q*. I.e., the optimal Q function. Let Q_ii = Q_i + B. I.e., a biased version of the optimal Q function. We know that both Q_i and Q_ii induce the same (optimal) policy. However, while delta_t^i is zero everywhere (the bellman error is zero for Q*), delta_t^ii is not zero (because of the discount factor). If B is large enough then delta_t^ii will be arbitrarily large to the point where the TDW algorithm will completely overlook Q_ii.
3.	Fear of uncertainty: in stochastic settings the TD error will almost surely won’t be zero. How does the method work in the presence of stochasticity?
4.	In my understanding, TDW favors policies that go to deterministic parts of the state space (where the TD error can be arbitrarily small), over policies that go to uncertain parts of the state space (where the TD error will never be zero). However, it is not difficult to construct an example where the optimal policy is to go to the uncertain parts of the state space. Therefore, TDW favors determinism over uncertainty. However, determinism is not necessarily linked to better performance.
5.	The paper links between certainty (as reflected in the TD error) and performance. If the performance criterion would be risk-sensitive, e.g. CVAR, then I could agree more with the claim (maybe then performance is linked with certainty). However, certainty and performance do not go together, at least not when talking about the expected return criteria. 
6.	Out of curiosity: since the method requires calculating the max over Q. How would it work in a continuous action space?
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper addresses the problem of combining separately learned action-value functions into a single, no-longer-learning, action selection algorithm. The result is not necessarily a combined policy, strictly speaking, because it need not take the same action (or produce the same action selection probabilities) each time it is in the same state. Such “ensemble methods” have been a big topic in supervised learning, where ensemble methods are often effective, but less so in reinforcement learning, where the evidence in support of the idea is, perhaps, weak. This paper just assumes it is a good idea, while citing prior work that i don’t think really shows that it is. A new method is proposed, called TDW, based on weighting the learner’s action values according to their recent squared temporal-difference errors. This is compared with simple ensemble method that have no memory other than their action values, on two gridworld problems and six Atari games. Comparison is also made with single (non-ensemble) methods that are given as much training data as one of the members of the ensemble. The results are consistent with TDW being better than the simpler ensemble methods on both gridworlds and Atari games, and with both ensemble methods being better than the single methods on Atari games but worse than the single methods on the gridworld problems. 

Does this work make a contribution to the field of reinforcement learning? A lot depends on whether you think the problem of combining separate learners is an important problem. I am skeptical. First, this problem requires one to be able to identify separate learners that are operating in independent copies of the identical environment, and whose states and actions can be mapped one-to-one onto each another. (This is also a requirement of asynchronous methods, so this criticism applies to them as well.) Of course it is easy for us to arrange these things with our simulated test environments, and there may even be _a few_ real problems like this, but most of the time this is unrealistic. If a method requires this, then it is not a general method. Second, and I suppose relatedly, this new problem bears little relation to the original problem: Does our field benefit from the introduction of new problems and solution methods that apply only in special cases? Or are these special cases primarily distractions from the general case and thus ways to avoid coming to grips with the real problem?

Different researchers might answer these rhetorical questions in different ways. It is a judgement call, but they reflect a real concern. A greater problem is that this work does not do a good job addressing the problem of ensembles. First, the introduction glosses over the question of whether ensembles are a good idea; it kinda suggests that their effectiveness has already been established, but does not commit itself to that statement; it does not make a clear claim that might be true or false about that prior work and that would validly support interest in this area. Second, this paper contains the wrong sort of comparisons of ensemble methods with single (non-ensemble) methods. It compares an ensemble of 10 learners with a single learner who only has as much data as one of the 10 learners in the ensemble. Thus, each single learner has only one-tenth as much data as has gone into the ensemble. Surely this is not the right comparison. It is a real problem that an early proponent of combining learners makes an unfair comparison with non-combined learners. It sets a poor standard and example for those who come later. The third way in which this work on ensemble RL agents is not very good it that its ensemble methods are not well suited to the task. That is, if one was going to combine learners, then there are many interesting natural ways to do this, and these are not done. It would be natural for each learner to keep track of how much experience or confidence it has in each part of the state space. The TDW algorithm can be seen as one way of doing this, but it is just one rather idiosyncratic way; there are many more natural ways. If we are going to explore this new problem, and new algorithms for solving it, then one really ought to do better than propose only methods that retain only their action-value function and nothing else that might facilitate the subsequent combination. In these three ways I feel this paper is not a good example of exploring the new problem of ensemble learning, even if you think that problem is a worthy one.

review_title: Official Blind Review #1

Meta-review: The paper proposes a method to combine the decision of an ensemble of RL agents. It uses an uncertainty measure based on the TD error, and suggests a weighted average or weighted voting mechanism to combine their policy or value functions to come up with a joint decision.
The reviewers raised several concerns, including whether the method works in the stochastic setting, whether it favours deterministic parts of the state space, its sensitivity to bias, and unfair comparison to a single agent setting.
There is also a relevant PhD dissertation (Elliot, 2017), which the authors surprisingly refused to discuss and cite because apparently it was not published at any conference. A PhD dissertation is a citable reference, if it is relevant. If it is, a good scholarship requires proper citation.

Overall, even though the proposed method might potentially be useful, it requires further investigations. Two out of three reviewers are not positive about the paper in its current form. Therefore, I cannot recommend acceptance at this stage.

Elliott, Daniel L., The Wisdom of the crowd : reliable deep reinforcement learning through ensembles of Q-functions, PhD Dissertation, Colorado State University, 2017
Decision: Reject