The wisdom of the crowd: reliable deep reinforcement learning through ensembles of Q-functions

Abstract: Reinforcement learning agents learn by exploring the environment and then exploiting what they have learned.
      This frees the human trainers from having to know the preferred action or intrinsic value of each encountered state.
      The cost of this freedom is reinforcement learning is slower and more unstable than supervised learning.
      We explore the possibility that ensemble methods can remedy these shortcomings and do so by investigating a novel technique which harnesses the wisdom of the crowds by bagging Q-function approximator estimates.
      
      Our results show that this proposed approach improves all three tasks and reinforcement learning approaches attempted.
      We are able to demonstrate that this is a direct result of the increased stability of the action portion of the state-action-value function used by Q-learning to select actions and by policy gradient methods to train the policy.
      

Reviews and author responses:

This paper proposes a cute idea as suggesting ensembles of Q-function approximations rather than a singular DQN. 

However, at the core of it, this boils down to previously studied methods in the literature, one of which also is not cited here: 

@inproceedings{osband2016deep,
  title={Deep exploration via bootstrapped DQN},
  author={Osband, Ian and Blundell, Charles and Pritzel, Alexander and Van Roy, Benjamin},
  booktitle={Advances in neural information processing systems},
  pages={4026--4034},
  year={2016}
}

Experiments provided in this paper compares with only the weak baseline of single DQN, however, it fails to compare other similar ideas in the literature such as the above paper. Hence, this paper lacks enough novelty for publication, and it is not clear from the experiments that the specific method proposed in this paper is better than others in the SOTA. 

This paper proposes the deep reinforcement learning with ensembles of Q-functions. Its main idea is updating multiple Q-functions, instead of one, with independently sampled experience replay memory, then take the action selected by the ensemble. Experimental results demonstrate that the proposed method can achieve better performance than non-ensemble one under the same training steps, and the decision space can also be stabilized.

This paper is well-written. The main ideas and claims are clearly expressed. Using ensembles of Q-function can naturally reduce the variance of decisions, so it can speed up the training procedure for certain tasks. This idea is simple and works well. The main contribution is it provides a way to reduce the number of interactions with the environment. My main concern about the paper is the time cost. Since the method requires updating multiple Q-functions, it may cost much more time for each RL time step, so I’m not sure whether the ensemble method can outperform the non-ensemble one within the same time period. This problem is important for practical usage. However, the authors didn’t show these results in the paper.

Minor things:
+The main idea is described too sketchily. I think more examples, such as in section 8.1, should be put in the main text.
+Page6 Line2, duplicated ‘the’.


This paper introduces an ensemble version of Deep RL by bagging Q-function approximation estimates. In the experiments, the performance of the proposed work is compared to the baseline, single DQN. In spite of the contribution, this paper has a critical issue. 

It has been extensively studied in the literature that ensemble DQN could lead to better performance than a single DQN. See the seminal work by Osband et al. (2016). The authors did not cite this paper, not to say a long list of recent works who have cited this seminal work. This indicates that the authors fail to conduct a serious literature review. In addition, more comprehensive experiments are required to compare the proposed work with the state-of-the-art ensemble DQN methods.

Osband et al. (2016), Deep Exploration via Bootstrapped DQN. NIPS.




Meta-review: The paper suggests using an ensemble of Q functions for Q-learning. This idea is related to bootstrapped DQN and more recent work on distributional RL and quantile regression in RL. Given the similarity, a comparison against these approaches (or a subset of those) is necessary. The experiments are limited to very simple environment (e.g. swing-up and cart-pole). The paper in its current form does not pass the bar for acceptance at ICLR.

Decision: rejected-papers