# Distributional Reinforcement Learning with Ensembles
## 2 Background  
We considered agent-environment interactions. For each observed state, the agent selects an action, whereby the environment generates a reward and a next state. Following the framework of [10], we let \(\mathcal{X}\) and \(\mathcal{A}\) denote the sets of states and actions, respectively, and let \(p\colon\mathcal{X}\times\mathcal{A}\to\mathscr{P}(\mathbb{R}\times\mathcal{X})\) be a transition kernel that maps state-action pairs to joint distributions of immediate rewards and next states. Then, we can model this interaction by a Markov Decision Process (MDP) \((\mathcal{X},\mathcal{A},p,\gamma)\), where \(\gamma\in[0,1)\) is a discount factor of future rewards. Moreover, an agent can sample its actions through a stationary policy \(\pi\colon\mathcal{X}\to\mathscr{P}(\mathcal{A})\), which maps a current state to a distribution over available actions.  
Throughout the rest of this paper, we consider MDPs where \(\mathcal{X}\times\mathcal{A}\) is a countable state-action space. We denote by \(\mathcal{D}=\mathscr{P}(\mathbb{R})^{\mathcal{X}\times\mathcal{A}}\) the set of functions where \(\eta\in\mathcal{D}\) maps each state-action pair \((x,a)\) to a distribution \(\eta^{(x,a)}\in\mathscr{P}(\mathbb{R})\). Similarly, we put \(\mathcal{D}_{n}=\mathscr{P}_{n}(\mathbb{R})^{\mathcal{X}\times\mathcal{A}}\), where \(\mathscr{P}_{n}(\mathbb{R})\) is the set of probability distributions with finite \(n^{\text{th}}\)-moments. For a given \(\eta\in\mathcal{D}\), we let \(Q_{\eta}\colon\mathcal{X}\times\mathcal{A}\to\mathbb{R}\) denote the function that maps state-action pairs \(\{(x,a)\}\) to the corresponding first moments of \(\{\eta^{(x,a)}\}\), i.e.,  
\[Q_{\eta}(x,a)\coloneqq\int_{\mathbb{R}}z\;\eta^{(x,a)}(dz).\]  
To appreciate a subsequent summary of distributional reinforcement theory fully, we may also need to make the following definition explicit.  
**Definition 1**.: _For a Borel measurable function \(g\colon\mathbb{R}\to\mathbb{R}\) and \(\
u\in\mathscr{P}(\mathbb{R})\), we let \(g_{\#}\
u\) denote the push-forward measure defined by:_  
\[g_{\#}\
u(A)\coloneqq\
u\left(g^{-1}(A)\right)\]  
_on all Borel sets \(A\subseteq\mathbb{R}\). In particular, given \(r,\gamma\in\mathbb{R}\), we let \((f_{r,\gamma})_{\#}\
u\) be the push-forward measure where \(f_{r,\gamma}(x)\coloneqq r+\gamma x\)._  
Suppose further that we have a set \(\mathcal{P}\) of categorical distributions supported on a fixed set \(\mathbf{z}=\left\{\,z_{1},z_{2},\ldots,z_{K}\,\right\}\) of equally-spaced numbers. Then, the following projection operator minimizes the distance between any categorical distribution \(\
u=\sum_{i=1}^{n}p_{i}\delta_{y_{i}}\) and elements in \(\mathcal{P}\) with respect to the Cramer metric [9; 13].  
**Definition 2**.: _The Cramer projection \(\Pi_{\mathbf{z}}\) maps any Dirac measure \(\delta_{y}\) to a distribution in \(\mathcal{P}\) by:_  
\[\Pi_{\mathbf{z}}(\delta_{y})=\left\{\begin{array}{ll}\delta_{z_{1}}&y\leq z _{1},\\ \frac{z_{1}+y}{\Delta z}\delta_{z_{1}}+\frac{y-z_{1}}{\Delta z}\delta_{z_{1+1} }&z_{i}<y\leq z_{i+1},\\ \delta_{z_{K}}&y>z_{K}.\end{array}\right.\]_Moreover, the projection is defined to be linear over mixture distributions such that:_  
\[\Pi_{\mathbf{z}}\left(\sum_{i}p_{i}\delta_{y_{i}}\right)=\sum p_{i}\Pi_{\mathbf{z }}\left(\delta_{y_{i}}\right).\]