# Distributional Reinforcement Learning with Ensembles
## 2 Background
### Distributional Reinforcement Learning
#### 2.2.3 Categorical Evaluation and Control  
In most real applications, the updates of (4) and (5) are either computationally infeasible or impossible to fully compute due to \(p\) being unknown. It follows that approximations are key to defining practical distributional algorithms. This could involve parametrization over some selected set of distributions along with projections onto these distributional subspaces. It could also involve stochastic approximations with sampled transitions and gradient updates with function approximation. A structure for algorithms making use of such approximations is Categorical Distributional Reinforcement Learning (CDRL). In what follows is a short summary of the CDRL procedure fundamental to single agent implementations in this paper. Let \(\mathbf{z}=\{\,z_{1},z_{2},\ldots,z_{K}\,\}\) be an ordered fixed set of equally-spaced real numbers such that \(z_{1}<z_{2}<\cdots<z_{K}\) with \(\Delta z\coloneqq z_{i+1}-z_{i}\). Let:  
\[\mathcal{P}=\left\{\sum_{i=1}^{K}p_{i}\delta_{z_{i}}:p_{1},\ldots,p_{K}\geq 0,\ \sum_{i=1}^{K}p_{i}=1\right\}\subset\mathcal{P}(\mathbb{R})\]be the subset of categorical distributions in \(\mathcal{P}(\mathbb{R})\) supported on \(\mathbf{z}\). We consider parameterized distributions by using \(\widehat{\mathcal{D}}=\mathcal{P}^{\mathcal{A}\times\mathcal{X}}\) as the collection of possible inputs and outputs of an algorithm. Moreover, for each \(\eta\in\widehat{\mathcal{D}}\), we have:  
\[Q_{\eta}(x,a)=\sum_{i=1}^{K}p_{i}(x,a)z_{i}.\]  
as its Q-value function. Given a subsequent treatment of our extension of CDRL, we first reproduce the steps of the general procedure in Algorithm 1 (see [10], Algorithm 1). ```
1. At each iteration step \(t\) and input \(\eta_{t}\in\widehat{\mathcal{D}}\), sample a transition \((x_{t},a_{t},r_{t},x_{t}^{\prime})\). 2. Select \(a^{*}\) to be either sampled from \(\pi(x_{t})\) in the evaluation setting or taken as \(a^{*}=\arg\max_{a}Q_{\eta_{t}}(x_{t}^{\prime},a)\) in the control setting. 3. Recall the Cramer projection \(\Pi_{\mathbf{z}}\) given in Definition 2, and put: \[\widehat{\eta}_{t}^{(x_{t},a_{t})}\coloneqq\Pi_{\mathbf{z}}\left(f_{r_{t}} \right)_{\#}\eta_{t}^{(x_{t}^{\prime},a^{*})}.\]
4. Take the next iterated function as some update \(\eta_{t+1}\) such that: \[\text{KL}\left(\widehat{\eta}_{t}^{(x_{t},a_{t})}\parallel\eta_{t+1}^{(x_{t}, a_{t})}\right)<\text{KL}\left(\widehat{\eta}_{t}^{(x_{t},a_{t})}\parallel\eta_{t}^{(x _{t},a_{t})}\right),\] where: \[\text{KL}(\mathbf{p}\parallel\mathbf{q})\coloneqq\sum_{i=1}^{K}p_{i}\log \left(\frac{p_{i}}{q_{i}}\right)\] denotes the Kullback-Leibler divergence. ```  
**Algorithm 1**Categorical Distributional Reinforcement Learning (CDRL)  
Consider first a finite MDP and a tabular setting. Define \(\widehat{\eta}_{t}^{(x,a)}\coloneqq\eta_{t}^{(x,a)}\) whenever \((x,a)\
eq(x_{t},a_{t})\). Then, by the convexity of \(-\log(z)\), it is readily verified that updates of the form:  
\[\eta_{t+1}=(1-\alpha_{t})\eta_{t}+\alpha_{t}\widehat{\eta}_{t}\qquad(\alpha_{ t}\in(0,1))\]  
satisfy Step 4. In fact, if there exists a unique policy \(\pi^{*}\) associated with the convergence of (3), then this update yields an almost sure convergence, with respect to the supremum-Cramer metric, to a distribution in \(\widehat{\mathcal{D}}\) with \(\pi^{*}\) as the greedy policy (with some additional assumptions on the stepsizes \(\alpha_{t}\) and sufficient support (see [10], Theorem 2, for details). In practice, we are often forced to use function approximation of the form:  
\[\eta^{(x,a)}=\phi(x,a;\boldsymbol{\theta}),\]  
where \(\phi\) is parameterized by some set of weights \(\boldsymbol{\theta}\).