# Risk-Sensitive Policy with Distributional Reinforcement Learning
## 3 Materials and Methods
### Methodology
#### 3.2.4 Risk-Sensitive Distributional RL Algorithm  
In most applications, the motivation for choosing the distributional RL approach over the classical one is related to the improved expected performance that results from learning a richer representation of the environment. Despite having access to the full probability distribution of the random return, only the expectation is exploited to derive decision-making policies:  
\[\pi(s)\in\underset{a\in\mathcal{A}}{\operatorname{argmax}}\ \underbrace{\mathbb{E}[Z^{\pi}(s,a)]}_{ \mathcal{Q}^{\pi}(s,a)}. \tag{17}\]  
Nevertheless, as previously hinted, the random return \(Z^{\pi}\) also contains valuable information about the risk, which could be exploited to learn risk-sensitive decision-making and exploration policies. The present research work suggests that risk-sensitive distributional RL should be achieved by maximising the utility function \(U^{\pi}\), derived from \(Z^{\pi}\), instead of the expected return \(Q^{\pi}\) when selecting actions. This alternative operation would be performed during both exploration and exploitation. Even though maximising the utility function is not exactly equivalent to the optimisation of the objective criterion defined in Equation (11), it is a relevant step towards risk-sensitive RL. Consequently, a risk-sensitive policy \(\pi\) can be derived as follows:  
\[\pi(s)\in\underset{a\in\mathcal{A}}{\operatorname{argmax}}\ U^{\pi}(s,a). \tag{18}\]  
Throughout the learning phase, a classical Q-learning algorithm is expected to progressively converge towards the optimal value function \(Q^{*}\) that naturally arises from the optimal policy \(\pi^{*}\). In a similar way, the proposed risk-sensitive RL algorithm jointly learns the optimal policy \(\pi^{*}\) and the _optimal state-action risk-based utility function_\(U^{*}\). More formally, the latter two are mathematically defined as the following:  
\[U^{*}(s,a)=\alpha\ \mathbb{E}[Z^{*}(s,a)]\ +\ (1-\alpha)\ \mathcal{R}_{\rho}[Z^{*}(s,a )], \tag{19}\]  
\[Z^{*}(s,a)\overset{\mathcal{D}}{=}R(s,a)+\gamma Z^{*}\big{(}s^{\prime},\pi^{*} (s^{\prime})\big{)}, \tag{20}\]  
\[\pi^{*}(s)\in\underset{a\in\mathcal{A}}{\operatorname{argmax}}\ U^{*}(s,a). \tag{21}\]  
The novel methodology proposed by this research work to learn risk-sensitive decision-making policies based on the distributional RL approach is summarised as follows. Firstly, select any distributional RL algorithm that learns the full probability distribution of the random return \(Z^{\pi}\). Secondly, leave the learning process unchanged except for action selection, which involves the maximisation of the utility function \(U^{\pi}\) rather than the expected return \(Q^{\pi}\). This adaptation is the single change to the distributional RL algorithm, occurring at two different locations within the algorithm: _i._ the generation of new experiences by interacting with the environment, _ii._ the learning of the random return \(Z^{\pi}\) based on the distributional Bellman equation. However, this adaptation has no consequence on the random return \(Z^{\pi}\) learnt by the distributional RL algorithm, only on the actions derived from that probability distribution. Algorithm 1 details the proposed solution in a generic way, with the required modifications highlighted.