# Risk-Sensitive Policy with Distributional Reinforcement Learning
## 4 Results and Discussion
### Probability Distribution Visualisation  
A core advantage of the proposed solution is the improved interpretability of the resulting decision-making process. Indeed, the understanding and motivation of the decisions outputted by the learnt policy \(\pi\) is greatly facilitated by the access to the probability distributions of the random return jointly learnt. In addition, the analysis and comparison of the value, risk and utility functions (\(Q^{\pi}\), \(R^{\pi}\) and \(U^{\pi}\)) associated with different actions provide a valuable summary of the decision-making process, but also about the control problem itself. Such an analysis may be particularly important to correctly tune the risk trade-off parameter \(\alpha\) according to the user's risk aversion.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Benchmark Environment**} & \multicolumn{2}{c}{**DQN**} & \multicolumn{4}{c}{**RS-UMDQN-C**} \\ \cline{2-7}  & \(\mathbb{E}[S^{\pi}]\) & \(\mathcal{R}_{\rho}[S^{\pi}]\) & \(\mathbf{U}[S^{\pi}]\) & \(\mathbb{E}[S^{\pi}]\) & \(\mathcal{R}_{\rho}[S^{\pi}]\) & \(\mathbf{U}[S^{\pi}]\) \\ \hline Risky rewards & **0.3** & \(-1.246\) & \(-0.474\) & 0.1 & \(-\)**0.126** & \(-\)**0.013** \\ Risky transitions & **0.703** & 0.118 & 0.411 & 0.625 & **0.346** & **0.485** \\ Risky grid world & **0.347** & \(-1.03\) & \(-0.342\) & 0.333 & **0.018** & **0.175** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of the expectation \(\mathbb{E}[\cdot]\), the risk function \(\mathcal{R}_{\rho}[\cdot]\) and the utility function \(U[\cdot]\) of the cumulative reward \(S^{\pi}\) achieved by the decision-making policies \(\pi\) learnt by both the DQN and RS-UMDQN-C algorithms.  
Figure 5: Evolution of the risk-sensitivity performance indicator \(R_{s}\) (expected value of the random variable) achieved by the decision-making policies \(\pi\) learnt by both the DQN and RS-UMDQN-C algorithms during the training phase. (**a**) Risky rewards; (**b**) Risky transitions; (**c**) Risky grid world.  
As an illustration, Figure 6 demonstrates some random return probability distributions \(Z^{\pi}\) that are learnt by the RS-UMDQN-C algorithm. More precisely, a single relevant state is selected for analysis for each benchmark environment. The selection is based on the importance of the next decision in following a clear path, either maximising the expected outcome or mitigating the risk. Firstly, it can be observed that the risk-sensitive distributional RL algorithm manages to accurately learn the probability distributions of the random return, qualitatively, from a human perspective. In particular, the multimodality purposely designed to create risky situations appears to be well-preserved. Such a result is particularly encouraging, since this feature is essential to the success of the proposed methodology. Indeed, it ensures the accurate estimation of the risk, as defined in Section 3.2.2. This observation is in line with the findings of the research paper [27], introducing the UMDQN algorithm, and suggests that the solution introduced to achieve risk-sensitivity does not significantly alter the properties of the original distributional RL algorithm. Secondly, as previously explained, Figure 6 highlights the relevance of each function introduced (\(Q^{\pi}\), \(R^{\pi}\) and \(U^{\pi}\)) for making and motivating a decision. Their analysis truly contributes to the understanding of the potential trade-off between expected performance maximisation and risk mitigation for a given decision-making problem, as well as the extent to which different values of the important parameter \(\alpha\) lead to divergent policies.  
Figure 6: Visualisation of the random return probability distributions \(Z^{\pi}\) learnt by the RS-UMDQN-C algorithm for typical states of the benchmark environments, together with the value, risk and utility functions that are derived (\(Q^{\pi}\), \(R^{\pi}\) and \(U^{\pi}\)). (**a**) Risky rewards; (**b**) Risky transitions; (**c**) Risky grid world.