SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {C V a R} _ {\alpha} \big (V ^ {\pi} (s ^ {\prime}, \overline {{M}}) \mid s ^ {\prime} \sim \overline {{T}} (\cdot \mid s, a) \big) = V ^ {\pi} (\mu_ {0}, \overline {{M}}) - \frac {| K | \sqrt {\sigma_ {E} ^ {2} + \sigma_ {A} ^ {2}}}{\alpha \sqrt {2 \pi}} \exp^ {- \frac {1}{2} (\Phi^ {- 1} (\alpha)) ^ {2}}
$$
where  $\Phi^{-1}$  is the inverse of the standard normal CDF.
These results demonstrate that optimising for risk-aversion in the Bayesian MDP (Problem 4.1) results in risk-aversion to both aleatoric and epistemic uncertainty, as state-action pairs with either high epistemic or high aleatoric uncertainty are penalised. Therefore, Problem 4.1 favours choosing actions that have both low aleatoric and low epistemic uncertainty.
4.3 Approach
Our algorithm is inspired by the model-free online RL method for aleatoric risk only introduced by [65], but we have adapted it to the model-based offline RL setting. Following previous works on model-based offline RL, our algorithm makes use of a standard actor-critic off-policy RL algorithm for policy optimisation [58, 75, 76]. However, the key idea of our approach is that to achieve risk-aversion, we modify the distribution of trajectories sampled from the model ensemble. Following Equation 3, we first compute the adversarial perturbation to the transition distribution:
$$
\xi^ {*} = \underset {\xi \in \mathcal {B} _ {\rho} (\overline {{T}} (s, a, \cdot))} {\arg \min } \sum_ {s ^ {\prime}} \bar {T} (s, a, s ^ {\prime}) \cdot \xi \left(s ^ {\prime}\right) \cdot V ^ {\pi} \left(s ^ {\prime}, \bar {M}\right), \tag {7}
$$
where the value function  $V^{\pi}$  is estimated by the off-policy RL algorithm. When generating synthetic rollouts from the model, for each state-action pair  $(s,a)$  we sample successor states from the perturbed distribution  $\xi^{*}\overline{T}$ :
$$
s ^ {\prime} \sim \xi^ {*} (s ^ {\prime}) \cdot \bar {T} (s, a, s ^ {\prime}) \forall s ^ {\prime} \tag {8}
$$
This perturbed distribution increases the likelihood of transitions to low-value states. The transition data sampled from this perturbed distribution is added to the synthetic dataset,  $\widehat{\mathcal{D}}$ . This approach modifies the sampling distribution over successor states according to the risk-sensitive Bellman equation in Equation 3. Therefore, the value function learnt by the actor-critic RL algorithm under this modified sampling distribution is equal to the risk-sensitive value function. The policy is then optimised to maximise the risk-sensitive value function.
For the continuous problems we address, we cannot compute the adversarially modified transition distribution in Equation 7 exactly. Therefore, we use a sample average approximation (SAA) [62] to approximate the solution. Specifically, we approximate  $\overline{T}(s,a,\cdot)$  as a uniform distribution over  $m$  states drawn from  $\overline{T}(s,a,\cdot)$ . Then, we compute the optimal adversarial perturbation to this uniform distribution over successor states. Under standard regularity assumptions, the solution to the SAA
Algorithm 1 1R2R
1: Input: Offline dataset,  $\mathcal{D}$ ; Static risk measure,  $\rho$ ;  
2: Compute belief distribution over transition models,  $P(T \mid \mathcal{D})$ ;  
3: for iter = 1, ..., Niter :  
4: for rollout = 1, ..., Nrollout, generate synthetic rollouts:  
5: Sample initial state,  $s \in \mathcal{D}$ .  
6: for step = 1, ..., k :  
7: Sample action  $a$  according to current policy  $\pi(s)$ .  
8: Sample  $m$  successor states,  $\Psi = \{s_i'\}_{i=1}^m$  from  $\overline{T}(s, a, \cdot)$ .  
9: Consider  $\widehat{T}$ , a discrete approximation to  $\overline{T}$ :  $\widehat{T}(s, a, s') \approx \overline{T}(s, a, s') = \frac{1}{m}$ ,  $\forall s' \in \Psi$   
10: Compute adversarial perturbation to  $\widehat{T}$ :  
\[
\widehat{\xi}^{*} = \arg \min_{\xi \in \mathcal{B}_{\rho}(\widehat{T}(s, a, \cdot))} \sum_{s' \in \Psi} \widehat{T}(s, a, s') \cdot \xi(s') \cdot V^{\pi}(s', \overline{M})
\]  
11: Sample from adversarily modified distribution:  $s' \sim \widehat{\xi}^{*} \cdot \widehat{T}(s, a, \cdot)$   
12: Add  $(s, a, R(s, a), s')$  to the synthetic dataset,  $\widehat{\mathcal{D}}$ .  
13:  $s \gets s'$   
14: Agent update: Update  $\pi$  and  $V^{\pi}$  with an actor-critic algorithm, using samples from  $\mathcal{D} \cup \widehat{\mathcal{D}}$ .
converges to the exact solution as  $m \to \infty$  [62]. Details of how the perturbed distribution is computed for CVaR and the Wang risk measure are in Appendix B.
Algorithm Our approach is summarised in Algorithm 1 and illustrated in Figure 1. At each iteration, we generate  $N_{\mathrm{rollout}}$  truncated synthetic rollouts which are branched from an initial state sampled from the dataset (Lines 4-6). For a given  $(s,a)$  pair we sample a set  $\Psi$  of  $m$  candidate successor states from  $\overline{T}$  (Line 8). We then approximate the original transition distribution by  $\hat{T}$ , a uniform distribution over  $\Psi$ , in Line 9. Under this approximation, it is straightforward to compute the worst-case perturbed transition distribution for a given risk measure in Line 10 (see Appendix B). The final successor state  $s'$  is sampled from the perturbed distribution in Line 11, and the transition to this final successor state is added to  $\widehat{\mathcal{D}}$ .
After generating the synthetic rollouts, the policy and value function are updated using an off-policy actor-critic algorithm by sampling data from both the synthetic dataset and the real dataset (Line 14). Utilising samples from the real dataset means that Algorithm 1 does not precisely replicate the risk-sensitive Bellman equation (Equation 3). However, like [58, 75] we found that additionally utilising the real data improves performance (see Appendix D.4).
Implementation Details Following a number of previous works [29, 54, 76] we use a uniform distribution over an ensemble of neural networks to approximate  $P(T \mid \mathcal{D})$ . We also learn an ensemble of reward functions from the dataset, in addition to the transition model. Each model in the ensemble,  $T_{i}$ , outputs a Gaussian distribution over successor states and rewards:  $T_{i}(s', r \mid s, a) = \mathcal{N}(\mu_{i}(s, a), \Sigma_{i}(s, a))$ . For policy training we used soft-actor critic (SAC) [30] in Line 14. For policy improvement, we used the standard policy gradient that is utilised by SAC. We found this worked well in practice and meant that our approach requires minimal modifications to existing RL algorithms. Research on specialised policy gradients for risk-sensitive objectives can be found in [65, 12].
In our experiments, we seek to: (a) verify that 1R2R generates performant policies in deterministic environments by avoiding distributional shift, (b) investigate whether 1R2R generates risk-averse behaviour on stochastic domains, and (c) compare the performance of 1R2R against existing baselines on both stochastic and deterministic environments. To examine each of these questions, we evaluate our approach on the following domains. More information about the domains can be found in Appendix E.2, and code for the experiments can be found at github.com/marc-rigter/1R2R.
D4RL MuJoCo [21] There are three deterministic robot environments (HalfCheetah, Hopper, Walker2D), each with 4 datasets (Random, Medium, Medium-Replay, Medium-Expert).
Currency Exchange We adapt the Optimal Liquidation problem [2] to offline RL. The agent aims to convert 100 units of currency A to currency B, subject to a stochastic exchange rate, before a deadline. The dataset consists of experience from a random policy.
HIV Treatment We adapt this online RL domain [33, 20] to the offline setting. The 6-dimensional continuous state vector represents the concentrations of various cells and viruses. The actions correspond to the dosages of two drugs. Transitions are stochastic due to the efficacy of each drug varying stochastically at each step. The reward is determined by the health outcomes for the patient, and penalises side-effects due to the quantity of drugs prescribed. The dataset is constructed in the same way as the D4RL [21] Medium-Replay datasets.
Stochastic MuJoCo We modify the D4RL benchmarks by adding stochastic perturbation forces. We focus on Hopper and Walker2D as there is a risk that any episode may terminate early due to the robot falling over. We vary the magnitude of the perturbations between two different levels (Moderate-Noise, High-Noise). For each domain and noise level we construct Medium, Medium-Replay, and Medium-Expert datasets in the same fashion as D4RL.
Baselines We compare 1R2R against offline RL algorithms designed for risk-sensitivity (ORAAC [68] and CODAC [47]) in addition to performant risk-neutral model-based (RAMBO [58], COMBO [75], and MOPO [76]) and model-free (IQL [38], TD3+BC [22], CQL [40], and ATAC [11]) algorithms. Due to computational constraints, we only compare a subset of algorithms for Stochastic MuJoCo. Following [21], the results are normalised to approximately between 0 and 100.
Objectives and Hyperparameters A disadvantage of the dynamic risk perspective used by 1R2R is that it is unclear how to evaluate performance for dynamic risk. For this reason, we evaluate the algorithms using static risk measures. For D4RL, which is deterministic, we optimise the standard expected value objective. Following [68, 47], we use static  $\mathrm{CVaR}_{0.1}$  (CVaR at confidence level  $\alpha = 0.1$ ) as the optimisation objective for all other domains. This is the average total reward received on the worst  $10\%$  of runs. For ORAAC and CODAC, we set the optimisation objective to the desired objective for each domain. For all other baselines, we tune hyperparameters to obtain the best online performance for the desired objective on each dataset (see Appendix E.5). Likewise, we tune the hyperparameters for 1R2R (the rollout length and risk measure parameter) to achieve the best online performance as described in Appendix C.4.
Versions of 1R2R We test two versions of 1R2R which utilise different risk measures to re-weight the transition distribution to successor states:  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  and  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$ . These risk measures are defined formally in Appendix B. We chose CVaR because it is commonly used and easy to interpret. However, CVaR disregards the best possible outcomes, and this can lead to an unnecessary loss in performance [56]. Therefore, we also investigate using the Wang risk measure which is computed by reweighting the entire distribution over outcomes.
We present results for two ablations. Ablate risk is the same as 1R2R, except that the transition distribution of the rollouts is not modified, making this ablation equivalent to standard model-based RL. Ablate ensemble uses only a single model, and therefore the risk aversion is only with respect to the aleatoric uncertainty.
Results Presentation The results in Tables 1-3 are from evaluating the policies over the last 10 iterations of training, and averaged over 5 seeds. The highlighted numbers indicate results within  $10\%$  of the best score,  $\pm$  indicates the standard deviation over seeds, and "div." indicates that the value function diverged for at least one training run.
D4RL MuJoCo For these domains, the objective is to optimise the expected performance. Therefore, to generate the results in Table 1 we tune all algorithms to obtain the best expected performance. Note that to obtain strong expected performance on these domains, 1R2R still utilises risk-aversion to avoid distributional shift. Table 1 shows that both configurations of 1R2R outperform most of the baselines, and are competitive with the strongest risk-neutral baselines. The two existing offline RL algorithms that are able to consider risk, ORAAC and CODAC, perform poorly on these benchmarks. The ablation of the risk-sensitive sampling shows that when risk-sensitivity is removed from 1R2R, for many domains the training diverges (denoted by "div.") or performance degrades. This verifies that risk-aversion is crucial in our approach to ensure that the policy avoids distributional shift, and therefore also avoids the associated problem of value function instability [39]. Likewise, the ablation of the model ensemble also leads to unstable training for some datasets, indicating that risk-aversion to epistemic uncertainty (represented by the ensemble) is crucial to our approach.
Currency Exchange From now on, we consider stochastic domains where the objective is to obtain the best performance for the static  $\mathrm{CVaR}_{0.1}$  objective. All algorithms are tuned to optimise this
Table 1: Normalised expected value results for D4RL MuJoCo-v2.
1R2CvAR	1R2RWang	Ablate Risk	Ablate Ens.	ORAAC	CODAC	RAMBO	CQL	IQL	TD3+BC	MOPO	ATAC	COMBO
Random	HalfCheetah	36.9 ± 6.4	35.5 ± 1.7	38.6 ± 1.9	35.3 ± 4.1	22.4 ± 1.7	31.0 ± 2.0	40.0	19.6	1.9	11.0	35.4	3.9	38.8
Hopper	30.9 ± 2.3	31.8 ± 0.2	div.	div.	10.9 ± 13.3	9.2 ± 0.7	21.6	6.7	1.2	8.5	11.7	17.5	17.9
Walker2D	7.6 ± 12.2	8.1 ± 11.0	div.	div.	2.4 ± 3.5	12.8 ± 8.9	11.5	2.4	0.3	1.6	13.6	6.8	7.0
Medium	HalfCheetah	74.5 ± 2.1	75.5 ± 1.2	76.9 ± 1.2	60.3 ± 18.7	38.5 ± 4.8	62.8 ± 0.9	77.6	49.0	47.4	48.3	42.3	53.3	54.2
Hopper	80.2 ± 15.8	64.6 ± 25.1	64.4 ± 18.5	58.1 ± 23.2	29.6 ± 10.3	63.9 ± 1.7	92.8	66.6	66.3	59.3	28.0	85.6	97.2
Walker2D	63.9 ± 31.8	83.4 ± 4.8	85.3 ± 3.6	78.2 ± 4.0	45.5 ± 14.9	84.0 ± 4.0	86.9	83.8	78.3	83.7	17.8	89.6	81.9
Medium	HalfCheetah	65.7 ± 2.3	65.6 ± 1.3	65.9 ± 3.4	62.2 ± 2.8	39.3 ± 1.7	53.4 ± 1.9	68.9	47.1	44.2	44.6	53.1	48.0	55.1
Hopper	92.9 ± 10.7	93.2 ± 8.7	55.3 ± 14.4	85.0 ± 27.9	22.6 ± 12.1	68.8 ± 28.2	96.6	97.0	94.7	60.9	67.5	102.5	89.5
Walker2D	92.2 ± 2.3	88.4 ± 5.0	92.7 ± 3.6	85.9 ± 11.3	11.1 ± 5.6	73.9 ± 31.7	85.0	88.2	73.9	81.8	39.0	92.5	56.0
Medium	HalfCheetah	96.0 ± 6.0	94.5 ± 1.9	87.8 ± 6.3	73.0 ± 14.5	24.4 ± 7.6	76.7 ± 4.9	93.7	90.8	86.7	90.7	63.3	94.8	90.0
Hopper	81.6 ± 22.9	89.6 ± 17.8	88.6 ± 13.4	45.8 ± 20.1	4.1 ± 2.8	87.6 ± 6.4	83.3	106.8	91.5	98.0	23.7	111.9	111.1
Walker2D	90.9 ± 5.9	78.1 ± 7.9	div.	58.2 ± 26.8	42.8 ± 9.2	112.0 ± 2.3	68.3	109.4	109.6	110.1	44.6	114.2	96.1
Table 2: Results on Currency Exchange and HIV Treatment domains. The objective is to optimise static  $\mathrm{CVaR}_{0.1}$ . We report static  $\mathrm{CVaR}_{0.1}$  in addition to the average performance.
1R2RCAvR	1R2RWang	Ablate Risk Ablate Ens.	ORAAC	CODAC	RAMBO	CQL	IQL	TD3+BC	MOPO	ATAC	COMBO
Currency: CVaR0,1	64.0 ± 1.9	62.8 ± 2.3	31.4 ± 4.6	61.3 ± 3.0	0.0 ± 0.0	67.6 ± 0.4	28.7 ± 0.8	41.7 ± 1.3	0.0 ± 0.0	27.0 ± 8.3	55.0 ± 22.2	16.9 ± 6.5	49.0 ± 25.9
Currency: Average	78.8 ± 1.6	81.6 ± 1.1	99.7 ± 1.3	80.1 ± 1.9	0.0 ± 0.0	74.0 ± 0.1	99.6 ± 0.6	89.4 ± 1.3	0.0 ± 0.0	100.4 ± 3.6	64.7 ± 21.8	73.6 ± 8.5	55.0 ± 29.0
HIV: CVaR0,1	53.9 ± 0.5	51.4 ± 7.5	41.2 ± 21.9	47.3 ± 12.4	0.5 ± 0.7	26.7 ± 0.9	52.5 ± 1.8	27.2 ± 2.5	0.1 ± 0.1	35.8 ± 4.0	0.0 ± 0.0	0.1 ± 0.1	11.2 ± 7.3
HIV: Average	73.5 ± 1.9	68.9 ± 10.5	56.7 ± 30.1	64.3 ± 17.0	6.6 ± 13.1	59.9 ± 0.5	73.3 ± 2.1	63.5 ± 1.8	27.1 ± 4.9	70.5 ± 1.5	0.0 ± 0.0	1.0 ± 1.7	38.7 ± 22.4
objective. Table 2 shows the results for the Currency Exchange domain, and a plot of the return distributions is in Appendix D.3. We see that 1R2R and CODAC obtain by far the best performance for static  $\mathrm{CVaR}_{0.1}$ , indicating that these algorithms successfully optimise for risk-aversion. When the risk-averse sampling is ablated, 1R2R performs poorly for  $\mathrm{CVaR}_{0.1}$  demonstrating that the risk-averse sampling leads to improved risk-sensitive performance. ORAAC and the risk-neutral baselines perform poorly for the static  $\mathrm{CVaR}_{0.1}$  objective. TD3+BC and RAMBO achieve strong average performance but perform poorly for CVaR, indicating that there is a tradeoff between risk and expected value for this domain.
HIV Treatment The results for this domain are in Table 2, and full return distributions are shown in Appendix D.2. For this domain, we observe that 1R2R and RAMBO obtain the best performance for static  $\mathrm{CVaR}_{0.1}$ , as well as for average performance. The existing risk-sensitive algorithms, ORAAC and CODAC, both perform poorly on this domain. A possible explanation for this poor performance is that it is difficult to learn an accurate distributional value function from fixed and highly stochastic datasets. We observe that TD3+BC performs well in expectation, but poorly for static  $\mathrm{CVaR}_{0.1}$ . This shows that strong expected performance is not sufficient to ensure good performance in the worst outcomes.
Stochastic MuJoCo Due to resource constraints, we compare a subset of the algorithms on these benchmarks. The performance of each algorithm for static  $\mathrm{CVaR}_{0.1}$  is in Table 3. We observe that  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  performs the best for static  $\mathrm{CVaR}_{0.1}$  in the Medium and Medium-Replay datasets. For the ablation of risk-averse sampling, either training diverges or the performance degrades relative to  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  for most datasets. For these benchmarks,  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$  performs worse than  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$ . This may be because the Wang risk measure makes use of the entire distribution over outcomes, and therefore  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$  trains the policy on some over-optimistic data. On the other hand,  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  disregards the most optimistic outcomes, resulting in more robust performance. Due to these results, we recommend using  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  over  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$  by default.
$1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  considerably outperforms the prior risk-sensitive algorithms, ORAAC and CODAC. While IQL performs well for the deterministic benchmarks, it performs very poorly on all stochastic domains. This suggests that we should not rely on deterministic benchmarks for benchmarking offline RL algorithms. Several of the risk-neutral baselines outperform  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  on the multi-modal Medium-Expert datasets. This echoes previous findings indicating that model-based approaches perform less well on multi-modal datasets [45]. Thus, while 1R2R achieves very strong performance across all Random, Medium, and Medium-Replay datasets, it may be less suitable for multi-modal datasets. Finally, the expected value results in Table 6 in Appendix D.1 are similar, with the exception that  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  performs slightly less well for expected performance relative to the baselines. This indicates that for these domains, there is a modest tradeoff between risk and expected performance.
The results for the Currency Exchange domain show that only 1R2R and CODAC are able to achieve risk-averse behaviour. On most other domains, 1R2R comfortably outperforms CODAC, in addition
Table 3: Normalised static  ${\mathrm{{CVaR}}}_{0.1}$  results for Stochastic MuJoCo.
1R2RCVaR	1R2RWang	Ablate Risk	ORAAC	CODAC	RAMBO	CQL	IQL	TD3+BC
Medium	Hopper (Mod)	36.9 ± 5.1	37.3 ± 1.9	35.7 ± 2.9	10.2 ± 3.9	33.6 ± 0.1	37.8 ± 8.3	33.5 ± 0.1	33.0 ± 1.4	33.4 ± 0.1
Walker2D (Mod)	29.9 ± 8.7	28.3 ± 4.2	7.7 ± 11.6	17.7 ± 4.6	26.8 ± 0.8	8.0 ± 0.5	32.0 ± 1.7	22.7 ± 2.3	22.9 ± 11.5
Hopper (High)	38.6 ± 3.0	42.0 ± 1.2	40.2 ± 1.2	18.5 ± 10.0	36.8 ± 0.3	36.5 ± 7.3	38.7 ± 0.8	35.8 ± 0.9	37.4 ± 0.5
Walker2D (High)	19.4 ± 12.6	19.5 ± 5.2	div.	6.3 ± 7.4	32.2 ± 3.6	19.7 ± 0.7	43.1 ± 2.7	16.7 ± 3.5	48.2 ± 3.4
Medium	Hopper (Mod)	43.9 ± 11.7	37.1 ± 7.6	33.6 ± 2.8	7.4 ± 1.3	32.1 ± 4.7	35.1 ± 5.7	34.1 ± 2.2	-0.6 ± 0.0	24.2 ± 2.3
Walker2D (Mod)	30.1 ± 7.2	28.0 ± 7.1	29.7 ± 2.2	4.7 ± 2.5	23.3 ± 10.4	30.9 ± 5.7	25.4 ± 1.9	1.4 ± 0.6	28.8 ± 1.0
Hopper (High)	51.4 ± 8.1	46.1 ± 7.9	30.5 ± 10.9	14.7 ± 15.2	38.9 ± 0.8	36.4 ± 8.6	50.9 ± 2.8	7.3 ± 5.8	35.5 ± 2.0
Walker2D (High)	46.0 ± 10.1	28.8 ± 5.2	30.9 ± 14.3	1.1 ± 0.8	43.1 ± 14.4	42.2 ± 10.4	15.6 ± 6.1	-1.2 ± 0.3	29.9 ± 6.8
Medium	Hopper (Mod)	65.4 ± 31.7	45.7 ± 35.7	32.6 ± 26.3	14.9 ± 9.6	44.1 ± 10.9	32.0 ± 12.6	60.6 ± 11.7	75.0 ± 29.7	54.2 ± 3.3
Walker2D (Mod)	34.8 ± 6.5	38.5 ± 8.9	div.	14.7 ± 3.6	28.1 ± 12.5	29.9 ± 14.2	72.3 ± 9.0	24.5 ± 1.2	68.4 ± 2.7
Hopper (High)	35.5 ± 5.7	39.3 ± 4.6	30.5 ± 7.4	9.1 ± 9.4	47.1 ± 1.7	38.5 ± 2.1	47.1 ± 2.1	32.6 ± 4.3	51.0 ± 1.3
Walker2D (High)	10.8 ± 4.3	11.7 ± 5.2	div.	4.2 ± 6.2	26.7 ± 16.5	1.9 ± 1.4	55.5 ± 5.5	28.0 ± 8.8	60.2 ± 8.9
to most of the baselines. ORAAC, the other existing risk-sensitive algorithm, performs poorly across all domains tested. This demonstrates the advantage of 1R2R over existing algorithms: it is able to generate risk-averse behaviour (unlike the risk-neutral baselines), and it also achieves strong performance across many domains (unlike ORAAC and CODAC). We now discuss the limitations of our approach and possible directions for future work.
Static vs Dynamic Risk We chose to base our algorithm on dynamic risk, which evaluates risk recursively at each step, rather than static risk, which evaluates risk across episodes. The dynamic risk perspective is arguably more suitable for avoiding distributional shift, as it enforces aversion to uncertainty at each time step, thus preventing the policy from leaving the data-distribution at any step. It is also more amenable to our model-based approach, where we modify the transition distribution of synthetic data at each step. Applying this same approach to the static risk setting would likely require modifying the distribution over entire trajectories [12, 67]. However, sampling full length trajectories is not desirable in model-based methods due to compounding modelling error [32]. The disadvantage of dynamic risk is that it is difficult to choose the risk measure to apply and it can lead to overly conservative behaviour [24]. Furthermore, it is unclear how to interpret and evaluate performance for dynamic risk. For this reason, we evaluated static risk objectives in our experiments.
Modelling Stochasticity In our implementation, the transition distribution for each model is Gaussian. This means that our implementation is not suitable for domains with multimodal transition dynamics. To improve our approach, an obvious next step is to utilise better model architectures that can handle general multimodal distributions, as well as high-dimensional observations [31].
Epistemic vs Aleatoric Uncertainty Our approach is agnostic to the source of uncertainty. This is motivated by the intuition that poor outcomes should be avoided due to either source of uncertainty. A direction for future work is to use composite risk measures [18], which enable the aversion to each source of risk to be adjusted independently. This might be important for achieving strong performance if the estimation of each source of uncertainty is poorly calibrated.
Conclusion In this paper we proposed 1R2R, a model-based algorithm for risk-sensitive offline RL. Our algorithm is motivated by the intuition that a risk-averse policy should avoid the risk of poor outcomes, regardless of whether that risk is due to a lack of information about the environment (epistemic uncertainty) or high stochasticity in the environment (aleatoric uncertainty). 1R2R is: (a) simpler than existing approaches to risk-sensitive offline RL which combine different techniques, and (b) is able to generate risk-averse behaviour while achieving strong performance across many domains. As discussed, there many promising avenues for future work, including: adapting 1R2R to optimise for static (rather than dynamic) risk; utilising more powerful generative models to model the distribution over MDPs; and using composite risk measures to adjust the aversion to each source of uncertainty independently.
This work was supported by a Programme Grant from the Engineering and Physical Sciences Research Council (EP/V000748/1), the Clarendon Fund at the University of Oxford, and a gift from Amazon Web Services. Additionally, this project made use of time on Tier 2 HPC facility JADE2, funded by the Engineering and Physical Sciences Research Council (EP/T022205/1).