# Risk-Sensitive Policy with Distributional Reinforcement Learning
## 3 Materials and Methods
### Performance Assessment Methodology  
#### 3.3.1 Benchmark Environments  
This research work introduces some novel benchmark environments in order to assess the soundness of the proposed methodology to design risk-sensitive policies based on the distributional RL approach. These environments consist of three toy problems that are specifically designed to highlight the importance of considering the risk in a decision-making policy. More precisely, the control problems are built in such a way that the optimalpolicy will differ depending on whether the objective is to solely maximise the expected performance or to also mitigate the risk. This is achieved by including relevant stochasticity in both the state transition function \(p_{T}\) and reward function \(p_{R}\). Moreover, the benchmark environments are designed to be relatively simple in order to ease the analysis and understanding of the decision-making policies tha tare learnt. This simplicity also ensures the accessibility of the experiments, since distributional RL algorithms generally require a considerable amount of computing power. Figure 4 illustrates these three benchmark environments, and highlights the optimal paths to be learnt depending on the objective being pursued. For the sake of completeness, a thorough description of the underlying MDPs is provided in Appendix A, including the stochastic reward and transition functions. ```
Initialise the experience replay memory \(M\) of capacity \(C\). Initialise both the main and target DNN weights \(\theta=\theta^{-}\). for episode = 0 to \(N\)do for time step \(t=0\) to \(T\), or until episode termination do  Acquire the state \(s\) from the environment \(\mathcal{E}\). With probability \(\epsilon\), select a random action \(a\in\mathcal{A}\). Otherwise, select the action \(a=\operatorname*{argmax}_{a^{\prime}\in\mathcal{A}}U^{\pi}(s,a^{\prime};\theta)\). Execute action \(a\) in environment \(\mathcal{E}\) to get the next state \(s^{\prime}\) and the reward \(r\). Store the experience \(e=(s,a,r,s^{\prime})\) in \(M\). Randomly sample from \(M\) a minibatch of \(N_{e}\) experiences \(e_{i}=(s_{i},a_{i},r_{i},s_{i}^{{}^{\prime}})\). for\(i=0\) to \(N_{e}\)do  Distributional Bellman equation: \[Z^{\pi}(s_{i},a_{i};\theta)\overset{\mathcal{D}}{=}r_{i}+\gamma Z^{\pi}(s_{i+1 },\operatorname*{argmax}_{a_{i}^{\prime}\in\mathcal{A}}U^{\pi}(s_{i+1},a_{i}^ {\prime};\theta^{-});\theta^{-}).\] endfor  Compute the resulting loss \(\mathcal{L}(\theta)\), according to the probability metric selected. Update the main DNN parameters \(\theta\) using a deep learning optimiser with learning rate \(L_{r}\). Update the target DNN parameters \(\theta^{-}=\theta\) every \(N^{-}\) steps. Anneal the \(\epsilon\)-greedy exploration parameter \(\epsilon\). endfor endfor
```  
**Algorithm 1** Risk-sensitive distributional RL algorithm. The first benchmark environment is named _risky rewards_. It consists of a \(3\times 3\) grid world, within which an agent has to reach one of two objective areas that are equidistant from its fixed initial location. The difficulty of this control problem lies in the choice of  
Figure 4: Illustration of the benchmark environments introduced in this research work for the performance assessment of risk-sensitive decision-making policies. The optimal objective locations/paths in terms of risk mitigation and expected return maximisation are highlighted in green and orange, respectively. (**a**) Risky rewards; (**b**) Risky transitions; (**c**) Risky grid world. the objective area to target, because of the stochasticity present in the reward function. Reaching the first objective area yields a reward with a lower value in expectation and a limited deviation from that average. On the contrary, reaching the second objective location yields a reward that is higher in expectation, at the cost of an increased risk. The second benchmark environment that was studied is named _risky transitions_.