# Distributional Reinforcement Learning with Ensembles
## 3 Learning with Ensembles  
### Ensembles  
Ensemble methods have been widely used in both supervised learning and reinforcement learning. In supervised learning, this can involve bootstrap aggregating predictors for better accuracy when given unstable processes such as neural networks or using \"expert\" opinion mixtures for better estimators [18, 19]. A simple example that demonstrates the possible benefits of aggregation is the following average pool of \(k\) regression models: Given a sample to predict, assume that the models draw prediction errors \(\varepsilon_{i}\), \(i=1,\ldots,k\) from a zero-mean multivariate normal distribution with \(\mathbb{E}[\varepsilon_{i}^{2}]=\sigma^{2}\) and correlations \(\rho_{ij}=\rho\). Then, the error made by averaging their predictions is \(\varepsilon\coloneqq(1/k)\sum_{i=1}^{k}\varepsilon_{i}\) with  
\[\mathbb{E}[\varepsilon^{2}]=\left(1+\rho(k-1)\right)\frac{\sigma^{2}}{k}.\]  
It follows that the mean squared error goes to \(\sigma^{2}/k\) as \(\rho\to 0\), whereas we get \(\sigma^{2}\) and no benefit when the errors are perfectly correlated.  
Under the assumption of independently trained agents, we have a reinforcement learning variant of the average pool in the following definition.  
**Definition 3**.: _Given an ensemble of \(k\) agents, let \(\widehat{Q}^{(i)}\) denote the \(Q\)-value function estimate of agent \(i\), and let \(\widehat{Q}\coloneqq(1/k)\sum_{i=1}^{k}\widehat{Q}^{(i)}\) denote the mean function. Then, the average joint policy \(\overline{\pi}\) selects actions according to:_  
\[a^{*}=\operatorname*{arg\,max}_{a}\widehat{Q}(x,a)=\operatorname*{arg\,max}_{ a}\frac{1}{k}\sum_{i=1}^{k}\widehat{Q}^{(i)}(x,a).\]  
_at every \(x\in\mathcal{X}\)._  
Thus, \(\overline{\pi}\) represents an aggregation strategy where we consider the information provided by each agent as equally important. Moreover, by the linearity of expectations and in view of (3), if we have initial functions \(Q_{0}^{(i)}\) with \(n\)-step ensemble values \(Q_{n}\coloneqq(1/k)\sum_{i=1}^{k}Q_{n}^{(i)}\), then full updates \(Q_{n}^{(i)}\coloneqq\mathcal{T}^{*}Q_{n-1}^{(i)}\) of each agent will yield \(Q_{n}=\mathcal{T}^{*}Q_{n-1}\) for the ensemble. Assume further that learning is done with a single algorithm in separate environments. If we take \(\widehat{Q}^{(i)}(x,a)\) as estimates of \(Q_{n}^{(i)}(x,a)\) for some step \(n\), with errors \(\varepsilon_{i}\) distributed as multivariate Gaussian noise, then we should expect \(\widehat{Q}(x,a)\) to have a smaller expected error variance in its estimation of \(Q_{n}(x,a)\) similar to regression models. This implies more robust performance when given an unstable training process far from convergence, but it also implies diminishing improvements when the algorithm is close to converging to a unique policy.  
However, in real applications, and in particular with function approximation, there may be instances where the improved performance by \(\overline{\pi}\) does not vanish due to agents converging to distinct sub-optimal policies. An illustration of this phenomenon can be seen in Figure 1. It shows evaluations during learning in the LunarLander-v2 environment [20]. The single agents used CDRL on a 29 atom support. To approximate distributions, the agents used small neural networks with three encoding layers consisting of 16 units each. The architecture was purposely chosen to make it harder for the optimizer to converge to an optimal policy, possibly due to lack of capacity. At each evaluation point, the models were tested with \(\varepsilon=0.001\). The figure also includes evaluations of average joint policies of five agents having the same evaluation \(\varepsilon\). However, we can see that the joint information provided by an ensemble of five agents transcends individual capacity, indicating that some agents settle on distinct sub-optimal solutions.