# Risk-Sensitive Policy with Distributional Reinforcement Learning
## 4 Results and Discussion  
### Decision-Making Policy Performance  
To begin with, the performance achieved by the decision-making policies \(\pi\) learnt has to be evaluated, both in terms of expected outcome and risk. For comparison purposes, the results obtained by the well-established DQN algorithm, a reference without any form of risk sensitivity, are presented alongside those of the newly introduced RS-UMDQN-C algorithm. It should be mentioned that these two RL algorithms achieve very similar results when risk sensitivity is disabled (\(\alpha=1\) for the RS-UMDQN-C algorithm), as expected. In the following, two analyses are presented. Firstly, the probability distribution of the cumulative reward of a policy \(\pi\), denoted \(S^{\pi}\in\mathcal{Z}\), is investigated. More precisely, the expectation \(\mathbb{E}[\cdot]\), the risk function \(\mathcal{R}_{\rho}[\cdot]\) and the utility function \(U[\cdot]\) of that random variable \(S^{\pi}\) are derived for each algorithm and compared. Secondly, this research work introduces a novel easy-to-interpret performance indicator \(R_{s}\in[-1,1]\) to evaluate the risk sensitivity of the decision-making policies that are learnt, by taking advantage of the simplicity of the benchmark environments presented in Section 3.3.1. In fact, this is made possible by the easy assessment, from a human perspective, of the relative riskiness of a path in the grid world environments being studied. If the optimal path in terms of risk is chosen (green arrows in Figure 4), a score \(R_{s}=+1\) is awarded. On the contrary, the riskier path, which is optimal in terms of expectation (orange arrows in Figure 4) yields a score of \(R_{s}=-1\). If no objective or trap areas are reached within the allowed time, a score of \(R_{s}=0\) is delivered. Consequently, the evolution of this performance indicator provides valuable information about the convergence of the RL algorithms towards the different possible paths, as well as about the stability of the learning process. Formally, let \(\tau=\{s_{t},a_{t}\}_{t\in[0,T]}\) with \(s_{t}\in\mathcal{S}\) and \(a_{t}\in\mathcal{A}\) be a trajectory defined over a time horizon \(T<10\) (ending with a terminal state, and subject to an upper bound), and let \(\tau_{+}\) and \(\tau_{-}\), respectively, be the sets of trajectories associated with the green and orange paths in Figure 4. Based on these definitions, the risk sensitivity \(R_{s}\) of a policy \(\pi\) is a random variable that can be assessed via Monte Carlo as the following:  
\[R_{s}(\pi)= \begin{cases}+1&\text{if $\pi$ produces trajectories $\{s_{t},a_{t}\}_{t\in[0,T]}\in\tau_{+}$ with $a_{t}=\pi(s_{t})$},\\ -1&\text{if $\pi$ produces trajectories $\{s_{t},a_{t}\}_{t\in[0,T]}\in\tau_{-}$ with $a_{t}=\pi(s_{t})$},\\ 0&\text{otherwise}.\end{cases}  \tag{22}\]  
The first results regarding policy performance are summarised in Table 2, which compares the decision-making policies learnt by the DQN and RS-UMDQN-C algorithms both in terms of expected outcome and risk. The second results regarding policy performance \begin{table}
\begin{tabular}{l l l} \hline \hline
**Hyperparameter** & **Symbol** & **Value** \\ \hline DNN structure & - & [128; 128] \\ Learning rate & \(L_{r}\) & \(10^{-4}\) \\ Deep learning optimiser epsilon & - & \(10^{-5}\) \\ Replay memory capacity & C & \(10^{4}\) \\ Batch size & N\({}_{e}\) & \(32\) \\ Target update frequency & \(N^{-}\) & \(10^{3}\) \\ Random return resolution & \(N_{z}\) & \(200\) \\ Random return lower bound & \(z_{\text{min}}\) & \(-2\) \\ Random return upper bound & \(z_{\text{max}}\) & \(+2\) \\ Exploration \(\epsilon\)-greedy initial value & - & \(1.0\) \\ Exploration \(\epsilon\)-greedy final value & - & \(0.01\) \\ Exploration \(\epsilon\)-greedy decay & - & \(10^{4}\) \\ Risk coefficient & \(\rho\) & \(10\%\) \\ Risk trade-off & \(\alpha\) & \(0.5\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Description of the main hyperparameters used in the experiments.