# Risk-Sensitive Policy with Distributional Reinforcement Learning
## 3 Materials and Methods
### Theoretical Background
#### 3.1.2 Distributional Reinforcement Learning  
In classical _Q-learning_ RL, the core idea is to model the _state-action value function_\(Q^{\pi}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) of a policy \(\pi\). This important quantity \(Q^{\pi}(s,a)\) represents the expected discounted sum of rewards obtained by executing an action \(a\in\mathcal{A}\) in a state \(s\in\mathcal{S}\) and then following a policy \(\pi\):  
\[Q^{\pi}(s,a)=\mathop{\mathbb{E}}_{s_{t},r_{t}}\left[\sum_{t=0}^{\infty}\gamma^ {t}r_{t}\right],\quad(s_{0},a_{0}):=(s,a),\ a_{t}=\pi(s_{t}). \tag{1}\]  
The key to the learning process is the _Bellman equation_[26] that the \(Q\) function satisfies:  
\[Q^{\pi}(s,a)=\mathop{\mathbb{E}}_{s^{\prime},r}\left[r+\gamma Q^{\pi}(s^{ \prime},\pi(s^{\prime}))\right]. \tag{2}\]  
In classical RL, the main objective is to determine an _optimal policy_\(\pi^{*}\), which can be defined based on the _optimal state-action value function_\(Q^{*}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\), as follows:  
\[Q^{*}(s,a)=\mathop{\mathbb{E}}_{s^{\prime},r}\left[r+\gamma\max_{a^{\prime} \in\mathcal{A}}Q^{*}(s^{\prime},a^{\prime})\right], \tag{3}\]  
\[\pi^{*}(s)\in\mathop{\mathrm{argmax}}_{a\in\mathcal{A}}\ Q^{*}(s,a). \tag{4}\]  
The optimal policy \(\pi^{*}\) maximises the expected return (discounted sum of rewards). This research work later presents an alternative objective criterion for optimality in a risk-sensitive RL setting. The distributional RL approach goes a step further by modelling the full probability distribution over returns instead of only its expectation [8], as illustrated in Figure 1. To this end, let the reward \(R(s,a)\) be a random variable distributed under \(p_{R}(\cdot|s,a)\), the _state-action value distribution_\(Z^{\pi}\in\mathcal{Z}\) of a policy \(\pi\) (also called _state-action return distribution function_) is a random variable, defined as follows:  
\[Z^{\pi}(s,a)\overset{D}{=}\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\,\quad(s_{0},a_{0}):=(s,a),\ a_{t}=\pi(s_{t}),\ s_{t+1} \sim p_{T}(\cdot|s_{t},a_{t})\, \tag{5}\]where \(A\overset{D}{=}B\) denotes the equality in probability distribution between the random variables \(A\) and \(B\). Therefore, the state-action value function \(Q^{\pi}\) is the expectation of the _random return_\(Z^{\pi}\). Equivalent to the expected case, there exists a _distributional Bellman equation_ that recursively describes the random return \(Z^{\pi}\) of interest:  
\[Z^{\pi}(s,a)\overset{D}{=}R(s,a)+\gamma P^{\pi}Z^{\pi}(s,a)\, \tag{6}\]  
\[P^{\pi}Z^{\pi}(s,a):\overset{D}{=}Z^{\pi}(s^{\prime},a^{\prime})\,\quad s^{ \prime}\sim p_{T}(\cdot|s,a),\ a^{\prime}=\pi(s^{\prime})\, \tag{7}\]  
where \(P^{\pi}:\mathcal{Z}\rightarrow\mathcal{Z}\) is the transition operator. To end this section, one can define the _distributional Bellman operator_\(\mathcal{T}^{\pi}:\mathcal{Z}\rightarrow\mathcal{Z}\), together with the _distributional Bellman optimality operator_\(\mathcal{T}^{*}:\mathcal{Z}\rightarrow\mathcal{Z}\) as follows:  
\[\mathcal{T}^{\pi}Z^{\pi}(s,a)\overset{D}{=}R(s,a)+\gamma P^{\pi}Z^{\pi}(s,a)\, \tag{8}\]  
\[\mathcal{T}^{*}Z^{*}(s,a)\overset{D}{=}R(s,a)+\gamma Z^{*}\big{(}s^{\prime}, \pi^{*}(s^{\prime})\big{)},\quad s^{\prime}\sim p_{T}(\cdot|s,a). \tag{9}\]  
A distributional RL algorithm may be characterised by two core features. Firstly, both the representation and parameterisation of the random return probability distribution have to be properly selected. Multiple solutions exist representing a unidimensional distribution: probability density function (PDF), cumulative distribution function (CDF), quantile function (QF). In practice, deep neural networks (DNNs) are generally used for the approximation of these particular functions. The second key feature relates to the probability metric adopted for comparing two distributions, such as the Kullback-Leibler (KL) divergence, the Cramer distance or the Wasserstein distance.