One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
Authors: Marc Rigter
Bruno Lacerda
Nick Hawes
Venue: NeurIPS 2023 poster (2023)
Keywords: offline reinforcement learning
model-based reinforcement learning
risk
uncertainty
TL;DR: Optimising risk-averse policies in an ensemble of models to avoid distributional shift and risky actions.

Abstract: Offline reinforcement learning (RL) is suitable for safety-critical domains where online exploration is not feasible. In such domains, decision-making should take into consideration the risk of catastrophic outcomes. In other words, decision-making should be *risk-averse*. An additional challenge of offline RL is avoiding *distributional shift*, i.e. ensuring that  state-action pairs visited by the policy remain near those in the dataset. Previous offline RL algorithms that consider risk combine offline RL techniques (to avoid distributional shift), with risk-sensitive RL algorithms (to achieve risk-aversion). In this work, we propose risk-aversion as a mechanism to jointly address *both* of these issues. We propose a model-based approach, and use an ensemble of models to estimate epistemic uncertainty, in addition to aleatoric uncertainty. We train a policy that is risk-averse, and avoids high uncertainty actions. Risk-aversion to epistemic uncertainty prevents distributional shift, as areas not covered by the dataset have high epistemic uncertainty. Risk-aversion to aleatoric uncertainty discourages actions that are risky due to environment stochasticity. Thus, by considering epistemic uncertainty via a model ensemble and introducing risk-aversion, our algorithm (1R2R) avoids distributional shift in addition to achieving risk-aversion to aleatoric risk. Our experiments show that 1R2R achieves strong performance on deterministic benchmarks, and outperforms existing approaches for risk-sensitive objectives in stochastic domains.

One Risk to Rule Them All: A lisk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
Marc Rigter, Bruno Lacerda, Nick Hawes Oxford Roboties Institute University of Oxford
Correspondence: marcrigter@@mail.coommm
Abstract
Offline reinforcement learning (RL) is suitable for safety-critical domains where on- line exploration is not feasible. such domains, decision-making should take into consideration the risk of catastrophic outcomes. In other words, decision-making should be risk-averse. An additional challenge of offline RL is avoiding distribu- tional shift,, i.c. ensuring that state-action pairs visited by the policy remain near those in the dataset. Previous offline RL algorithms that consider risk combine of- fline RL techniques (to avoid distributional shift), with risk-sensitive RL algorithms (to achieve risk-aversion). In this work, we propose risk-aversion as a mechanism to jointly address both of these issues. We propose model-based approach, and use an ensemble of models to estimate epistemic uncertainty, in addition to aleatoric un- certainty. We train policy that is risk-averse, and avoids high uncertainty actions. Risk-aversion to epistemic uncertainty prevents distributional shift, as areas not covered by the dataset have high epistemic uncertainty. Risk-aversion aleatoric uncertainty discourages actions that are risky due to environment stochasticity. Thus, by considering epistemic uncertainty via model ensemble and introducing risk-aversion, our algorithm (IR2R) avoids distributional shift in addition to achiev- ing risk- tisk-aversion to aleatoric risk. Our experiments show that IR2R achieves strong performance on deterministic benchmarks, and outperforms existing approaches for risk-sensoe sensitiee objectives in stochastic domains.
Introduction
In safety-critical applications, online reinforcement learning (RL) is impractical due to the requirement for extensive random exploration that may be dangerous costly. To alleviate this issue, offline RL aims to find the best possible policy using only pre-collected data. In safety-critical settings we often want decision maaoing be risk-averse, and take into consideration variability in the performance between each episode. However, the vast majority of offline RL research considers the expected value objective, and therefore disregards such variability.
core issue in offline RL is avoiding distributional shift: to obtain a performant policy, we must ensure that the state-action pairs visited by the policy remain near those covered by the dataset. Another way that we can view this is that we need the policy to be averse to epistemic uncertainty (uncertainty stemming from lack of data). In regions and there will be large errors in any learned model or value function. A naively optimised policy may exploit these errors, resulting in erroneous action choices.
covered by the dataset, epistemic uncertainty is high
In risk-sensitive RL, the focus is typically on finding policies that are risk-averse to environment stochasticity, i.e. aleatoric uncertainty. Previous approaches to risk-sensitive offline RL [7] 68
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

combine two techniques: offline RL techniques, such as conservative value function updates 47 or behaviour cloning constraints [68. avoid distributional shift; and risk-sensitive RL algorithms, to achieve risk-aversion. Therefore, the issues of epistemic uncertainty and aleatoric uncertainty are handled separately, using different techniques. due to distributional shift, as well as aleatoric uncertainty due to environment stochasticity, by computing a policy that is risk-averse to both sources of uncertainty. This enables us to obtain performant and risk-averse offline RL algorithm that is simpler than existing approaches.
contrast, we propose to avoid epistemic uncertainty
In this work we propose IR2R (Figure model-based algorithm for risk-averse offline RL. The intuition behind our approach is that the policy should avoid the risk of a bad outcome, regardless of whether this risk due to high uncertainty in the model (epistemic uncertainty)
Sample action
RL training
V
Train Ensemble
or due to high stochasticity in the environment (aleatoric uncertainty). To represent epistemic uncertainty, we learn an ensemble of models. To optimise risk-averse policy, we penalise taking actions that are predicted to have highly variable outcomes. In out-of -f--o---bitiitiioon regions, the disagreement between members of the ensemble high, producing high variability in the model predictions. Therefore, the risk-averse policy is trained avoid out-of-distribution regions. Risk-aversion also ensures that the policy avoids risk due to highly stochastic outcomes, i.e. avoids aleatoric risk. Our work demonstrates that incorporating risk-aversion alone is a promising approach to offline RL, and proposes an algorithm Our main contributions (a) proposing risk-aversion towards epistemic uncertainty as a mechanism to address distributional shift in offline RL, and (b) introducing the first model-based algorithm for risk-averse offline RL, which jointly addresses epistemic and aleatoric uncertainty.
Sample re-weighted distribution
Sample sucessors
Re-weight using risk measure
Figure 1: Illustration of the IR2R algorithm.
that is simpler than existing approaches risk-averse offline RL 47 68
We evaluate our approach in several scenarios. Our experiments show that algorithm achieves strong performance relative to state-of-the-arr baselines on deterministic benchmarks (21, and outperforms existing approaches for risk-sensitiv objectives in stochastic domains.
2 Related Work
Offline RL: Offline RL addresses the problem of learning policies from fixed datasets. Most works on offline RL are risk-neutral, meaning that they optimise the expected value (53]. A key issue in offline RL is avoiding distributional shift so that the states visited by the learnt policy remain within the distribution covered by the dataset [4). Approaches to model-free offline RL include: constraining the learnt policy to be similar the behaviour policy [3] 38 7 22], conservative value
function updates [1] 37 72], importance sampling algorithms 43 or using Q-ensembles
for more robust value estimates
78)
Model-based approaches learn model of the environment and generate synthetic data from that model 63 to optimise policy via planning [5] or RL algorithms 76 75 By training policy on
additional synthetic data, model-based approaches have the potential for broader generalisation [7] Approaches to preventing model exploitation include constraining the trained policy to be similar to the behaviour policy [64] or applying conservative value function updates 775 in the same fashion as model-free approaches. Another approach is to modify the learnt MDP by either applying reward
penalties for state-action pairs with high uncertainty [35] 44
74
76), or modifying the transition
model to produce pessimistic synthetic transitions 29 58
Our work is similar in spirit to [2] 58).
but these existing works do not consider risk.
Risk-Sensitive RL: Many works have argued for risk-sensitive algorithms in safety-critical do- mains 25 Risk-sensie sesk-sensitive algorithms for MDPs include approaches that adapt policy gradients More recently, a number of works have utilised distributional RI. (0). which learns a distributional value function that predicts the full return distribution. Knowl-
to risk objectives [12
66 65
67
edge of the return distribution can then be used to optimise risk- sensstenitivv criterion [15] 33 46).

---

The main focus of research on risk-sensitive online RL is risk due to environment stochasticity [[9] (i.e. aleatoric uncertainty). However, there are a number of works that explicitly address risk due to epistemic uncertainty [14 16
18 57 To our knowledge, this is the first work to consider risk-aversion to epistemic uncertainty as a means to mitigate distributional shift in offline RI..
T9
Risk-Sensitive Offline RL: To our knowledge, there are two previous algorithms for risk-sensitive offline RL: ORAAC (68) and CODAC (47). To avoid distributional shift, ORAAC utilises policy constraints, while CODAC employs pessimistic value function updates to penalise out-of-distribution stae-acction pairs. Both algorithms learn distributional value function [ which is then used to optimise risk-averse policy. Thus, both of these works are model- freee, and combine techniques from offline RL and distributional RL to address the issues of distributional shift and risk sensitivitv respectively. In this work, we provide a new perspective by proposing to optimise for risk-aversion towards both epistemic uncertainty (stemming from distributional shift) and aleatoric uncertainty (stemming from environment stochasticity). Unlike previous approaches, our approach (a) is concep- tually simpler as does not combine different techniques, (b) demonstrates that risk-aversion alone is sufficient to avoid distributional shift, (c) does not require a distributional value function (which might be computationally demanding to train), and (d) is model-based.
3 Preliminaries
An MDP is defined by the tuple M = (S, 50, 50. 7). S and are the state and action spaces, R[s, a) is the reward function, [ a,) is the transition function, sq is the initial state, and 7 E (0, 1) is the discount factor. In this work we consider Markovian policies, π E II, which map each state to distribution over actions. In offline RL we only have access to fixed dataset of transitions from the MDP: D = (($:, Oe, Te, + The goal is to find a performant policy using the fixed dataset.
Model-Basee Offline RL These approaches utilise model of the MDP. The learnt dynamics model, T. typically trained via maximum likelihood estimation: ming (ss,,,"")> (- log T('' a)]. A model of the reward function, R(s, a). is also learnt if it is unknown. This results in the learnt MDP model: M (S, A,T, R, so, 7). Thereafter, any planning or RL algorithm can be used recover the optimal policy in the learnt model, मे = arg ma - Here, J indicates the optimisation objective, which is typically the expected value of the discounted cumulative reward.
However, directly applying this approach does not perform well in the offline setting due to distribu- tional shift. In particular, if the dataset does not cover the entire state-action space, the model will inevitably be inaccurate for some state-action pairs. Thus, naive policy optimisation on a learnt model in the offline setting can result in model exploitation [32 41 (5). i.e. a policy that chooses actions that the model erroneously predicts will lead to high reward. We propose mitigate this issue by learning an ensemble of models and optimising a risk-averse policy.
Following previous works 35
our approach utilises model-based policy optimisation (MBPO) (32). MBPO performs standard off-policy RL using an augmented dataset D U D, where D is synthetic data generated by simulating short rollouts in the learnt model. To train the policy, minibatches of data are drawn from D J D. where each datapoint is sampled from the real data, D, with probability S, and from D with probability J.
75
76
Risk Measures We denote the set of all probability distributions by P. Consider a probability space, (2, P), where Q is the sample space, F is the event space, and P E P is a probability distribution over F. Let Z be the set of all random variables defined over the probability space ($2, F, P). We will interpret a random variable Z E Z as a reward, i.c. the greater the realisation of Z. the better. We denote E-weighted expectation of Z by E2[Z] := Lwea ((w)(()) ((w)dw where E: Ω + IR is a function that we-weights the original distribution of Z.
A risk measure is a function, important class of risk measures are coherent risk measures, which satisfy a set of properties that are consistent with rational risk assessments. A detailed description and motivation for coherent risk measures given by 488. An example of coherent risk measure is the conditional value risk (CVaR). CVaR at confidence level a is the mean of the a-portion of the worst outcomes.
Z + R, that maps any random variable Z to a real number. An
3

---

All coherent risk measures can be represented using their dual representation [6] A risk measure, p. is coherent and only if there exists a convex bounded and closed set, B, E P, such that
p(Z)
min (EB,(P)
[
(1)
Thus, the value of any coherent risk measure for any Z E 2 can be viewed as an E-weighted expectation of Z, where & is the worst-case weighting function chosen from suitably defined risk envelope, (P). For more details on coherent risk measures, we refer the reader to [61].
Static and Dynamic Risk In MDPs, the random variable of interest is usually the total reward received at each episode, i.e. ZUDP = Er= y' (z). The static perspective on risk applies a risk measure directly this random variable: p(ZMDP). Thus, static risk does not take into account the temporal structure of the random variable in sequential problems, such as MDPs. This leads to the issue of time inconsistency [00, meaning that actions which are considered less risky one point in time may not be considered less risky at other points in time. This leads to Mon-Markovian optimal policies, and prohibits the straightforward application of dynamic programming due to the coupling of risk preferences over time.
This motivates dynamic risk measures, which take into consideration the sequential nature of the stochastic outcome, and are therefore time-consistent. Markov risk measures [60] are a class of dynamic risk measures which are obtained by recursively applying static coherent risk measures. Throughout this paper we will consider dynamic Markov risk over an infinite horizon, denoted by Poo. For policy N, MDP M, and static risk measure p. the Markov coherent risk p (M) is defined as
P (M) = -( (80)) + o
+ R(s1, (+1)) + p(>2 ((), (82)) +)..)).
(2)
where p is static coherent risk measure, and (sp, $1, ..) indicates random trajectories drawn from the Markov chain induced by π in M. Note that in Equation Caaaaioonnthha static risk measure is evaluated each step according to the distribution over possible successor states.
We define the ieskisensttive value function under some policy æ as (" M) = p (M |so = s). This value function can be found by recursively applying the risk-sensitive Bellman equation [60]
min V (s, M) R(s, (s)) (EB.(T(s))()))))
Σ [(s, E(s') ("(s', M).
Likewise, the optimal risk-sensitive value, (s(s, .M = max Poo (M = s). is the unique solution to the risk-sensitive Bellman optimality equation:
V' (s, M)
max
min [[ss a) + (T aEA
_ ET(s,,,')) E(s') (5)
Σ
(4)
Thus, we can view Markov dynamic risk measures as the expected value in an MDP where the transition probabilities at each step are modified adversarially within the eiselonee for the chosen one-step static risk measure.
4 One Risk to Rule Them All
In this section, we present One Risk to Rule Them All (IR2R), new algorithm for risk-averse offline RL (Figure Tp. Our approach assumes that we can compute an approximation to the posterior distribu- tion over MDPs, given the offline dataset. This distribution represents the epistemic uncertainty the real environment, and enables us to reason about risk due to epistemic uncertainty. IR2R then uses an RI. algorithm train policy offline using synthetic data generated from the distribution over MDP models. To achieve risk-aversion, the transition distribution of the model rollouts is modified adversarially according to the risk envelope of chosen risk measure. This simple modification to standard model-based RL penalises high uncertainty actions, thus penalising actions that have high epistemic uncertainty (i.e. are -uitt---iittiiutiion) as well as those that have high aleatoric uncertainty (i.e. are inherently risky). Therefore, by modifying the transition distribution of model rollouts to induce risk- aversion, 1R2R avoids distributional shift and generates risk-an averee behaviour.
4.1 Problem Formulation
To simplify the notation required, we assume that the reward function is known and therefore is only the transition function that must be learnt. Note that in practice, we also learn the reward function

---

from the dataset. Given the offline dataset, D. we assume that we can approximate the posterior distribution over the MDP transition function given the dataset: P(T) D). We can integrate over all possible transition functions to obtain single expected transition function:
T(s 8,) = J, T(s,a,'') P(T D) dT.
(5)
This gives rise to a single MDP representing the distribution over plausible MDPs, M = (S, T. R, 7). This is similar to the Bayes-Adaptive MDP [27 I7 28] (BAMDP), except that in the BAMDP the posterior distribution over transition functions is updated after each step of online interaction with the environment. Here, we assume that the posterior distribution is fixed given the offline dataset. To make this distinction, we refer to M as the Bayesian MDP (BMDP) (
In this work, we pose offline RL as optimising the dynamic risk in the BMDP induced by the dataset.
Problem 4.1 (Risk-Averse Bayesian MDP for Offline RL), Given offline dataset, D, static risk measure P. and belief over the transition dynamics given the dataset, P(T) D), find the policy with the optimal dynamic risk the corresponding Bayesian MDP M:
arg max P (M)
66)
4.2 Motivation
The motivation for Problem 4.1 is that optimising for risk-aversion in the Bayesian MDP results in risk-aversion to both aleatoric and epistemic uncertainty. We first provide intuition for this approach via the following empirical example. Then, we demonstrate this theoretically for the case of Gaussian
transition models in Proposition 4.2
Illustrative Example Figure 2 illustrates a dataset for an MDP where each episode consists of one step. A single reward is received, equal to the value of the succes- sor state, s'. We approximate P(T D) using an ensemble of neural networks, and the shaded region represents the aggregated Bayesian MDP transition function a, The dashed red line indicates the standard Q- values computed by drawing transitions from
a) or S
1.5
T(s, a, s'). For this value function, the action with the highest Q-value is 1, which is out- side of the data distribution. The solid red line indicates the risk-sensitive value function for the risk measure CVaRo.1 (i.e. a 0.1). This is computed by drawing transitions from uni- form distribution over the worst 10% transi- tions from T(s, a, s'). We observe that the risk- sensitive value function penalises straying far from the dataset, where epistemic uncertainty is high. The action in the dataset with highest expected value is 0.4. However, the action = -033 has a higher risk-sensitive value than a 0.4, due the latter having high- variance transitions (i.e. high aleatoric uncer- tainty). Thus, we observe that choosing actions according to the risk-sensitive value function penalises actions that are out-of-distributie (i.e. have high epistemic uncertainty) or have high aleatoric uncertainty.
1.0
(s,
0.5
0.0
005
0.0
0.5
a
Figure 2: MDP with single transition to suc- cessor state s', with reward R(s') = 8'. Black dots illustrate dataset. Shaded region indicates +2 S.D. of successor states drawn from an approxima- tion of T(x, a..) (represented by an ensemble (I3)). Dashed red line indicates the standard Q-values from running MBPO [32]. Solid red line indicates the risk-sensitive value function computed by reweight- ing the transitions according to CVaRg.1 Vertical arrows indicate optimal actions for each Q-function.
We now consider the specific case where each member of the ensemble is a Gaussian transition function with standard deviation A- We assume that in the ensemble, the mean of each Gaussian is also normally distributed with standard deviation dg. The latter assumption allows us to quantify the epistemic uncertainty in terms of the level of disagreement between members of the ensemble. Proposition 4.2 shows that risk-averse optimisation of transitions sampled from the ensemble results in ave-sin both the aleatoric and epistemic uncertainty.
5

---

Proposition 4.2. Consider some state in a ID state space, and some action a. Assume (a) that there an ensemble of N Gaussian transition functions, each denoted T; with mean μi and standard deviation o (Ti(8'ss.) = ((),,0))) normally distributed with mean Po and standard deviation og: P4 ~ NNoo gg). ) The N transition functions jointly define T, the transition function for Bayesian MDP, M. Assume that for some policy the risk-sensitive value is linear around po with some linearity constant, K:
and (b) that the mean of each Gaussian, Pú, is
((ss,M) V (Po.M) ) + K('s' pa)
Then, the value of randomly drawn successor state is distributed according to +
N ( (H= M), =
In Proposition 4.2 I defines the level of disagreement between the members of the ensemble, and therefore represents the level of epistemic uncertainty. σл represents the aleatoric uncertainty. The proposition shows that if either σ or og are high, then there high variability over the value of the successor state when sampling from the ensemble (i.e. sampling from T in Equation 5). Risk measures penalise high variability. Therefore, applying the risk-sensitive Bellman equation (Equation 3 to samples from T penalises executing state-action pairs for which either о or a is high. For the specific case of CVaR, Corollary 4.3 illussrates how the risk-sensitive value is reduced as a or og increase.
Corollary 4.3. Under the assumptions in Proposition 1, the CVuR confidence level of the value of a randomly druwn successor state is:
CVaR. (AaRRa(V"(( (V"('', s ~T(-- (M M) [K| o
0 :(# t(a))²
exp
/ 2m
where 11 is the inverse of the standard normal CDF.
These results demonstrate that optimising for risk-aversion in the Bayesian MDP (Problem 4.1 results in risk-aversion to both aleatoric and epistemic uncertainty, as state-action pairs with either high epistemic high alcatoric uncertainty are penalised. Therefore, Problem 4.1 favours choosing actions that have both low aleatoric and low epistemic uncertainty.
4.3 Approach
Our algorithm is inspired by the model-free online RL method for aleatoric risk only introduced by [65]. but we have adapted the model-based offline RL setting. Following previous works on model-based offline RL, our algorithm makes use of a standard actor-critic off off-policy RL algorithm for policy optimisation [58] 75 76). However, the key idea of our approach is that to achieve risk- aversion, we modify the distribution of trajectories sampled from the model ensemble. Following Equation we first compute the adversarial perturbation to the transition distribution:
E"
arg min ÇE BE (s.,a.))
T(s,,a a, 8') E(s") VT (s',M).
where the value function V is estimated by the off-policy RL algorithm. When generating synthetic rollouts from the model, for each state-action pair (s, a) we sample successor states from the perturbed distribution T
8'~ 5"') T(s, Vss
(8)
This perturbed distribution increases the likelihood of transitions to low-value states. The transition
data sampled from this perturbed distribution is added to the synthetic dataset, D. This approach modifies the sampling distribution over successor states according the risk-sensitive Bellman equation in Equation Therefore, the value function learnt by the actor-critic RL algorithm under this modified sampling distribution is equal to the risk-sensitive value function. The policy is then optimised to maximise the risk-sensitive value function.
For the continuous problems we address, cannot compute the adversarially modified transition distribution in Equation 7 exactly. Therefore, we use a sample average approximation (SAA) 62 to approximate the solution. Specifically, we approximate T(s, a. -) as a uniform distribution over m states drawn from T(x a, -). Then, we compute the optimal adversarial perturbation this uniform distribution over successor states. Under standard regularity assumptions, the solution to the SAA

---

Algorithm IR2R
1: Input: Offline dataset, D; Static risk measure, Pi
2: Compute belief distribution over transition models, P(T) D);
3: for iter = l,.... Nuer
4: for rollout L,..., Nestou. generate synthetic rollouts:
Sample imitial state, E D.
6: for step = .....,k:
7:
Sample action according to current policy (s))
Sample m successor states, V = { from as)
9:
Consider T, a discrete approximation T: ((, T(s.a, As'
Compute adversarial perturbation T::
arg min (f s'ev
E fss,a, E(6') (*(
11:
Sample from adversarially modified distribution: s' ~ E as) Add (s, a, R(s, a), s') to the synthetic dataset, D.
12:
13:
4- s'
14:
Agent update: Update ग and V with actor-critic algorithm, using samples from D PUD. D.
converges to the exact solution as m + ao 62 Details of how the perturbed distribution is computed for CVaR and the Wang risk measure are in Appendix B
Algorithm Our approach is summarised in Algorithm and illustrated in Figure [ At each iteration, we generate Noolloat truncated synthetic rollouts which are branched from an initial state sampled from the dataset (Lines 6). For a given (s, a) pair we sample a set V of m candidate successor states from T (Line 8). We then approximate the original transition distribution by T. a uniform distribution over V in Line 9 Under this approximation, it straightforward to compute the worst-case perturbed transition distribution for given risk measure in Line (see Appendix (B). The final successor state is sampled from the perturbed distribution in Line and the transition to this final successor state added to D.
After generating the synthetic rollouts, the policy and value function are updated using an off-policy actor-critic algorithm by sampling data from both the synthetic dataset and the real dataset (Line 14p. Utilising samples from the real dataset means that Algorithm does not precisely replicate the we found that additionally
risk-sensitive Bellman equation (Equation 3 However, like [58 75
utilising the real data improves performance (see Appendix D.4.
uniform distribution over an ensemble of neural networks to approximate P(T DD. We also learn an ensemble of reward functions from the dataset, in addition to the transition model. Each model in the ensemble, T;, outputs a Gaussian distribution over successor states and rewards: T:(s', | a)
Implementation Details Following number of previous works 29
54
76
we
(1 a), Eg(s, a)). For policy training we used soft-actor critic (SAC) 30 in Line 14 For policy improvement, we used the standard policy gradient that is utilised by SAC. We found this worked well in practice and meant that our approach requires minimal modifications to existing RL algorithms.
Research on specialised policy gradients for risk-sensitive objectives can be found in 665 12
5 Experiments
In our experiments, seek to: (a) verify that IR2R generates performant policies in deterministic environments by avoiding distributional shift, (b) investigate whether 1R2R generates risk-averse behaviour on stochastic domains, and (c) compare the performance of IR2R against existing baselines on both stochastic and deterministic environments. To examine each of these questions, we evaluate our approach on the following domains. More information about the domains can be found in Appendix E.2 and code for the experiments can be found at hwtiuuu.ooo/mmarrriiiae rigter/1R2R
D4RL MuJoCo (1 There are three deterministic robot environments (HalfCheetah, Hopper, Walker2D), each with 4 datasets (Random, Medium, Medium-Replay Medium-Experr)
Currency Exchange We adapt the Optimal Liquidation problem 2 to offline RI. The agent aims to convert 100 units of currency A to currency B, subject to a stochastic exchange rate, before a deadline. The dataset consists of experience from random policy.
7

---

HIV Treatment We adapt this online RL domain B3 20 to the offline setting. The 6-dimensional continuous state vector represents the concentrations of various cells and viruses. The actions correspond to the dosages of two drugs. Transitions are stochastic due to the efficacy of each drug varying stochastically at each step. The reward determined by the health outcomes for the patient, and penalises side-effects due to the quantity of drugs prescribed. The dataset is constructed in the same way as the D4RL [21] Saddium-Replay datasets.
Stochastic MuJoCo We modify the D4RL benchmarks by adding stochastic perturbation forces. We focus on Hopper and Walker2D as there a risk that any episode may terminate early due to the robot falling over. We vary the magnitude of the perturbations between two different levels (Moderate- Noise, High-Noise). For each domain and noise level we construct Medium, Medium-Replay and Medium--xppet datasets in the same fashion as D4RL.
Baselines We compare 1R2R against offline RL. algorithms designed for risk-sensitivity
(ORAAC 68 and CODAC (47] in addition to performant risk-neutral model-based (RAMBO 588
COMBO 75). and MOPO (76) and model-free (IQL [38]. TD3+BC (22), CQL [40), and ATAC [
algorithms. Due to computational constraints, we only compare a subset of algorithms for Stochastic
MuJoCo. Following 21 the results are normalised approximately between 0 and 100.
Objectives and Hyperparameters A disadvantage of the dynamic risk perspective used by IR2R is that it is unclear how to evaluate performance for dynamic risk. For this reason, we evaluate the algorithms using static risk measures. For D4RL, which is deterministic, we optimise the standard expected value objective. Following [68 47, we use static CV (CVaR confidence level a = 0.1) as the optimisation objective for all other domains. This is the average total reward received on the worst 10% of runs. For ORAAC and CODAC, we set the optimisation objective the desired objective for each domain. For all other baselines, we tune hyperparameters to obtain the best online performance for the desired objective on each dataset (see Appendix E.5) ). Likewise, we tune the hyperparameters for IR2R (the rollout length and risk measure parameter) to achieve the best online performance as described in Appendix C.4
Versions of 1R2R We test two versions of IR2R which utilise different risk measures to re- re-weight the transition distribution to successor states: IR2RevaR and IR2Rwang. These risk measures defined formally in Appendix B We chose CVaR because it is commonly used and easy to interpret. However, CVaR disregards the best possible outcomes, and this lead to unnecessary loss in performance [56). Therefore, we also investigate using the Wang risk measure which is computed by reweighting the entire distribution over outcomes.
We present results for two ablations. Ablate risk is the same as IR2R, except that the transition distribution of the rollouts is not modified, making this ablation equivalent to standard model-based RL. Ablate ensemble uses only a single model, and therefore the risk aversion is only with respect to the alcatoric uncertainty.
Results Presentation The results in Tables 3 are from evaluating the policies over the last 10 iterations of training, and averaged over 5 seeds. The highlighted numbers indicate results within 10% of the best score, indicates the standard deviation over seeds, and "div." indicates that the value function diverged for at least one training run.
5.1 Results
D4RL MuJoCo For these domains, the objective is to optimise the expected performance. Therefore, to generate the results in Table [ we tune all algorithms to obtain the best expected performance. Note that to obtain strong expected performance on these domains, 1R2R still utilises rvsk-aversion to avoid distributional shift. Table T shows that both configurations of 1R2R outperform most of the baselines, and are competitive with the strongest lisk-ncutral baselines. The two existing offline RL algorithms that are able to consider risk. ORAAC and CODAC, perform poorly on these benchmarks. The ablation of the risk-sensitive sampling shows that when risk-sensitivity is removed from IR2R, for many domains the training diverges (denoted by "div.") performance degrades. This verifies that risk-aversion is crucial in our approach to ensure that the policy avoids distributional shift, and therefore also avoids the associated problem of value function instability [39]. Likewise, the ablation of the model ensemble also leads to unstable training for some datasets, indicating that risk-aversion to epistemic uncertainty (represented by the ensemble) is crucial our approach.
Currency Exchange From now on, we consider stochastic domains where the objective to obtain the best performance for the static CVaRa.1 objective. All algorithms are tuned to optimise this
8

---

Table I: Normalised expected value results for D4RL MuJoCo-v2.
1R2Reve
IRRRune
Ablate Risk Ablate Ens.
ORAAC
CODAC
RAMBO
CQL
IQL
TD3+BC MOPO
ATAC
COMBO
Export Repley Medeium Kandom
HallChoctah
966.
$5.5
35.5+
35.3
22,-122
$1.0
40.0
19.6
1.9
35.4
3.9
Hopper
30.9 2.3
S18110
div.
div.
10.0
13.3
9.2.1.. 0.7
21.6
6.7
1.2
8.5
11.7
17.5
17.9
Walker2D
7.6.1 12.2
8.1
11.0
div
div
244 9.5
12.8
8.0
11.5
2.4
0.3
1.6
13.5
6.8
7.0
HaliCheetah
2..11.111
75.5
1.2
169112
60.5
18.7
38.5
62.8
1..8
77.6
49.0
47.4
48.3
42.3
53.3
54.2
Hopper
80.2 16.8
64.
= 25.1
644
18.5
68.1
4: 23.2
20.6
10.3
61.9 1.7
92.8
66.6
66.3
59.3
28.0
K5.6
97.2
Walker2D
63.9 31.8
83.4
4.8
80.3
78.24:40
4.0
45.5
14.9
81.0 6.11
No.9
53.8
78.3
53.7
17.8
99.0
$1.9
Medium Medium
HallCheerah
6..7 4..6=1.3
8..
02.2422
1..
53.4 1.9
68.9
47.1
44.2
44.6
53.1
48.0
55.1
Hepper
92.0++10. 90.2
+ 6.7
55.3
+ 14.4
85.0
25.0
22.6
12.1
05.8
28-2
96.6
97.0
94-7
60.9
67.5
100.5
89.5
Wilker
22..2+2.3
58.4
5.0
92.7
4:0
65.9+
11.3
11.1
5.6
73.9
+ 31.7
85.0
88.2
75.9
61.8
99.0
92.5
56.0
HaliChortah
96.0++660 94.55
+ 1.0
67.8
75.0+
14.5
22.4
7.6
76.7 9..
93.7
90.8
86.7
90.7
63.5
94.8
90.0
Hooper
22, 89.6
1: 17.8
85.5 19.4
45.8
20.1
4.1
288
87.6 6.4
83.3
106.8
91.5
98,0
23.7
111.9
111.1
Walker2D
(00.9:1.. 78.1 7.9
div
58.2 26,8
42.8 - 0.2
112.0
1220
68.3
109.4
109.6
100.1
44.5
114.2
96.1
Table 2: Results on Currency Exchange and HIV Treatment domains. The objective is to optimise static CV.Ro.1. We report static CV in addition to the average performance.
Abbtte Risk Ablate Kmn.
ORAAC
CODAC:
RAMIBO
CQU.
IQr.
THB-BIC
MOPD
ATAC
COMBO
Ourency: CViRaa
1.6
02.8 2.3 13.4
:1:
676
7
0::000000
2
56.0 = 222
65
Cumancy: Ancrage
0.0
0.0
65.0 29.0
HIV. CVaR..
7.4
21. 12.4
0.7
26.7 0.5
12
25
n.l 0...0.11
0.0
11227. 1.0
0.1 #0.
HIV. Average
7 1.9 68.9 10.
36.7 20.1 17.0
6.6
7888221
63,0
18
#7.
70
0.0
36.7
objective. Table 2 shows the results for the Currency Exchange domain, and a plot of the return distributions is in [ppendix[D.3]] We see that IR2R and CODAC obtain by far the best performance for static CV indicating that these algorithms successfully optimise for aisk-iversion. When the risk-averse sampling is ablated, IR2R performs poorly for CVaaa.1 demonstrating that the risk-averse sampling leads to improved risk-sensitive performance. ORAAC and the risk-neutral baselines perform poorly for the static CVaRo.1 1 objective. TD3+BC and RAMBO achieve strong average performance but perform poorly for CVaR, indicating that there a tradeoff between risk and expected value for this domain.
HIV Treatment The results for this domain are in Table 2 and full return distributions are shown in Appendix D.2 For this domain, we observe that IR2R and RAMBO obtain the best performance for static CVaRg,1. as well as for average performance. The existing risk-sensitive algorithms, ORAAC and CODAC, both perform poorly on this domain. A possible explanation for this poor performance that it difficult learn an accurate distributional value function from fixed and highly stochastic datasets. We observe that TD3+BC performs well in expectation, but poorly for static CVaRo.1. This shows that strong expected performance is not sufficient ensure good performance in the worst outcomes.
Stochastic MuJoCo Due to resource constraints, we compare a subset of the algorithms on these benchmarks. The performance of each algorithm for static CVaRo.1 is in Table We observe that IR2RevaR performs the best for static CVaRe.1 in the Medium and Medium- Reeilaamm-eyyyyyae datasets. For the ablation of risk-averse sampling, either training diverges or the performance degrades relative to IR2Reva for most datasets. For these benchmarks, IR2R Wang performs worse than IR2RevaR- This may be because the Wang risk measure makes use of the entire distribution over outcomes, and therefore 1R2R trains the policy on some optrioptiiistic data. On the other hand, 1R2RevaR disregards the most optimistic outcomes, resulting in more robust performance. Due to these results, we recommend using IR2RRvaR over 1R2Rwang by default.
IR2Rcva considerably outperforms the prior risk-sensitive algorithms, ORAAC and CODAC. While IQL performs well for the deterministic benchmarks, it performs very poorly on all stochastic domains. This suggests that we should not rely on deterministic benchmarks for benchmarking offline RL algorithms. Several of the risk-neutra baselines outperform IR2Rcva on the multi-modal Medium- Expert datasets. This echoes previous findings indicating that model-based approaches perform less well on multi-modal datasets (5). Thus, while 1R2R achieves very strong performance across all Random, Medium, and Medium- Medium-Replay datasets, it may be less suitable for multi-moda datasets. Finally, the expected value results in Table Appendix [.] are similar, with the exception that IR2Revr performs slightly less well for expected performance relative to the baselines. This indicates that for these domains, there is modest tradeoff between risk and expected performance.
6 Discussion and Conclusion
The results for the Currency Exchange domain show that only 1R2R and CODAC are able to achieve risk-averse behaviour. On most other domains, IR2R comfortably outperforms CODAC, in addition

---

Table 3: Normalised static CVaRg.1 results for Stochastic MuloCo.
IR2Rcva
1R2R
Ablute Risk
ORAAC
CODAC
RAMBO
COL
10L
TD3+BC
Mosiun
Hopper (Mod)
5.1
1.9
35.7
2.9
10.2
3.9 33.6 0.1
37.8 8.3
33.5 4 0.1
33.0
14
33.4 1:
0.1
Walker2DD (Mod)
29.9++8.7
2.2
11.
17.7
268
80+0.
22.7
22.4
Hopper (Blig)
38.6
4:2
4.22
1855
10.0
36.8
0.3
36.473
08.7
35.8
3744 0.5
Walker2D (High)
6.2
63
14
322436
1.77
Medium
Roploy
Walker2D (Mod)
Hopper (Mod)
30.1 00.1-+77.2 7:2
7.6
28.0
37.1
83.6
20.7 4:2
2.5
7.4 1.3
4.7 = 2.5
32.1
23.3
4.7
10.4
30.9 5.7
$5.1
24.122
25.4
1.9
-0.6
1.4
24.2
Hopper (High)
51.4++8..1
7.9
10.9
14.7
2.
Walker2D (High)
(46.0
10.1
28.8 5.2
30,9 1. 14.3
1.1 = 0.8
43.1
14.1
42.2
10.4
15.6
6.1
-.22
Mediam Expent
Hopper (Mod)
35.7
4:377
32.6
a: 26.3
11. 9.6
44.1 +
10.9
52.0
12.6
00.6
11.7
75.0
29.7
5124
Walker2D (Mod)
6.5
8.9
14.7
28.1
29.9
72.3
24.5
68.4
Hopper (High)
330.5+
39.3 4.6
30.0 +.4
9.1 = 94
47.1
+
1.7
38.5
2.1
47.1
+
2.1
32.6
+
4.3
51. 1.3
Walker2D (High)
10.8 1.3
11.7
div.
4.3 6.2
26.7
+
16.5
194
144
36.5
4
5.5
28.0
8.8
60.2
2
to most of the baselines. ORAAC, the other existing risk-sensitive algorithm, performs poorly across all domains tested. This demonstrates the advantage of IR2R over existing algorithms: it is able to generate risk-averse behaviour (unlike the risk-neutral baselines), and it also achieves strong performance across many domains (unlike ORAAC and CODAC). We now discuss the limitations of our approach and possible directions for future work.
Static vs Dynamic Risk We chose to base algorithm on dynamic risk, which evaluates risk recursively at each step, rather than static risk, which evaluates risk across episodes. The dynamic risk perspective is arguably more suitable for avoiding distributional shift, as it enforces aversion to uncertainty at each time step, thus preventing the policy from leaving the data-distribution at any step. It is also more amenable to our model-based approach, where we modify the transition distribution of synthetic data at each step. Applying this same approach to the static risk setting would likely require modifying the distribution over entire trajectories I12 677. However, sampling full length trajectories
is not desirable in model-based methods due to compounding modelling error (32. The disadvantage
of dynamic risk is that it is difficult choose the risk measure to apply and it can lead to overly conservative behaviour (24). Furthermore, it is unclear how to interpret and evaluate performance for dynamic risk. For this reason, we evaluated static risk objectives in our experiments.
Modelling Stochasticity In our implementation, the transition distribution for each model is Gaussian. This means that our implementation is not suitable for domains with multimodal transition dynamics. To improve our approach, an obvious next step is to utilise better model architectures that can handle general multimodal distributions, as well as high-dimensional observations [31.
Epistemic vs Alcatoric Uncertainty Our approach is agnostic to the source of uncertainty. This is motivated by the intuition that poor outcomes should be avoided due to either source of uncertainty. direction for future work is to use composite risk measures [18], which enable the aversion to each source of risk to be adjusted independently. This might be important for achieving strong performance if the estimation of each source of uncertainty is poorly calibrated.
In this paper we proposed 1R2R, a model- based algorithm for risk-sensitive offline RL. Our algorithm is motivated by the intuition that a risk-averse policy should avoid the risk of poor outcomes, regardless of whether that risk is due to a lack of information about the environment (epistemic uncertainty) or high stochasticity in the environment (aleatoric uncertainty). 1R2R is: (a) simpler than existing approaches to risk sensitive offline RL which combine different techniques, and (b) is able to generate risk-averse behaviour while achieving strong performance across many domains. As discussed, there many promising avenues for future work, including: adapting 1R2R to optimise for static (rather than dynamic) risk; utilising more powerful generative models to model the distribution over MDPs; and using composite risk measures adjust the aversion to each source of uncertainty independently.
Conclusion
Acknowledgements
This work was supported by a Programme Grant from the Engineering and Physical Sciences Research Council (EP//000044//1, the Clarendon Fund at the University of Oxford, and gift from Amazon Web Services. Additionally, this project made use of time on Tier 2 HPC facility JADE2, funded by the Engineering and Physical Sciences Research Council (EP/TO22205/I).
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114, PMLR, 2020.
[2] Robert Almgren and Neil Chriss. Optimal execution of portfolio transactions. Journal of Risk, 3:5 40, 2001.
[3] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified Q-ensemble. Advances in Neural Information P Processing Systems, 34:7436-7447, 2021.
[4] Giorgio Angelotti, Nicolas Drougard, and Caroline Ponzoni Carvalho Chanel. Exploitation vs caution: Risk- sensitive policies for offline learning. arXiv preprint arXiv:210S.13431, 2021.
[5] Arthur Argenson and Gabriel Dulac-Amold. Model-basee offline planning. In International Conference on Learning Representations, 2021.
[6] Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9:203-228, 1999.
[7] Philip J Ball, Cong Lu, Jack Parker-Holdes and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from single offline environment. In International Conference on Machine Learning, pages 619-629. PMLR, 2021.
[8] Wenhang Bao and Xiao-yang Liu. Multi-agent deep reinforcement learning for liquidation strategy analysis. arXiv preprint arXiv: 1906. 1046, 2019.
[9] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforce- ment learning. In International Conference on Machine Learning, pages 449-458, PMLR, 2017.
[10] Kang Boda and Jerzy A Filar. Time consistent dynamic risk measures. Mathematical Methods of Operations Research, 63:169-186, 2006.
[11] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. 2022.
[12] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk fisk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 1 18:6070-6120. 2017.
[13] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
[14] William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. ICML Workshop on Uncertainry and Robustness in Deep Learning, 2020.
[15] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pages 1096-1105. PMLR, 2018
[16] Stefan Depeweg, Jose-Miguel Hernandez-Lobato Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184-1193, PMLR, 2018.
[17] Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning identifia- bility challenges and effective data collection strategies. In Advances Neural Information Processing Systems, volume 34, 2021.

---

[18] Hannes Eriksson, Debabrota Basu, Mina Alibeigi, and Christos Dimitrakakis, Sentinel: Taming uncertainty with ensemble based distributional reinforcement learning. In Uncertainty in Artificial Intelligence, pages 631-640. PMLR, 2022.
[19] Hannes Eriksson and Christos Dimitrakakis. Epistemic sessitensiiive reinforcement learning. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2020.
[20] Damien Ernst, Guy-Bart Stan, Jorge Goncalves, and Louis Wehenkel. Clinical data based optimal STI strategies for HIV: reinforcement learning approach. In Proceedings of the 45th IEEE Conference on Decision and Control, pages 667-672. IEEE, 2006.
[21] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.00211, 2020.
[22] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[23] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pages 2052-2066 PMLR, 2019.
[24] Christopher Gagne and Peter Dayan. Two steps to risk sensitivity. Advances in Neural Information Processing Systems, 14:22209-222277 2021.
[25] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Leurning Research, 16:1437-1480, 2015.
[26] Alborz Geramifard, Christoph Dann, Robert Klein, William Dabney, and Jonathan How. RLPy: soaleeffuntion--bae reinforcement learning framework for education and research. Journal of Machine Learning Research, 16:1573-1578, 2015.
[27] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian rein- forcement learning: A survey. Foundations and Trends@ in Machine Learning, 8:359
2015.
[28] Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline RL policies should be trained to be adaptive. In International Conference on Machine Learning, pages 7513-7530. PMLR, 2022.
[29] Kaiyang Guo, Yunfeng Shao, and Yanhui Geng. Model-based offline reinforcement learning with pessimism-modulated dynamics belief. Advances in Neural Information Processing Systems, 2022.
[30] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[31] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. ICLR, 2021.
[32] Michael Janner, Justin Fu. Marvin Zhang, and Sergey Levine. When to trust your model: boseel-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
[33] Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 4433-4443, 2020.
[34] Valentyn Khokhlov. Conditional value-at-risk for elliptical distributions. Evropský časopis ekonomiky managementu, 2(6):70-79, 2016.
[35] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. Advances Neural Information Processing Systems, 33:21800-2182, 2020.
12

---

[36] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
[37] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In International Conference Machine Learning, pages 5774 5783. PMLR, 2021.
[38] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, 2022.
[39] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[40] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[41] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Picter Abbecl. Model-ensemble trust-region policy optimization. In International Conference Learning Representations, 2018.
[42] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[43] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. International Conference Machine Learning RLARealLife Workshop, 2019.
[44] Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, and Stephen J Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference Learning Representations, 2022.
[45] Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint
2022..
[46] Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004. 14547, 2020.
[47] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 14:19235-19247, 2021.
[48] Anirudha Majumdar and Marco Pavone. How should robot assess risk? towards axiomatic theory of risk in robotics. In Robotics Research, pages 75-84. Springer, 2020.
[49] Ross A Maller, Gernot Müller, and Alex Szimayer. Ornstein-uhlenber processes and exten- sions. Handbook of financial time series, pages 421-437. 2009.
[50] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. Conference Uncertainty in Artificial Intelligence, 2010.
[51] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans, Al- gaeDICE: Policy gradient from arbitrary experience. arXiv preprint 191.00791, 2019.
[52] Wojeiech Popko, Matthias Wächter, and Philipp Thomas. Verification of continuous time random walk wind model. In The 26th International Ocean and Polar Engineering Conference. OnePetro, 2016.
[53] Martin I. Puterman. Markov decision processes: Discrete stochastic dynamic programming. John Wiley & Sons, 2014.
13

---

[54] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. EPOpt: Learning robust neural network policies using model ensembles. ICLR, 2017.
[55] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. game theoretic framework for model based reinforcement learning. In International Conference Machine Learning, pages 7953-7963. PMLR, 2020.
[56] Mare Rigter, Paul Duckworth, Bruno Lacerda, and Nick Hawes. Planning for risk-aversion and expected value in MDPs. International Conference on Automated Planning and Scheduling, 2022.
[57] Mare Rigter. Bruno Lacerda, and Nick Hawes. Risk-averse Bayes-adaptive reinforcement learning, Advances in Neural Information Processing Systems, 34:1142-1155 2021.
[58] Mare Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 2022.
[59] Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of banking & finance, 26: 1443-1471, 2002.
[60] Andrzej Ruszezyúski. Risk-averse dynamic programming for Markov decision processes. Mathematical programming, 125:235-261, 2010.
[61] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. Lectures on stochastic programming: modeling and theory, chapter 6. SIAM, 2021.
[62] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezynski. Lectures on stochastic programming: modeling and theory, chapter 5. SIAM, 2021.
[63] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2:160-163, 1991.
[64] Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104: 104666 2021.
[65] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. Advances in Neural Information Processing Systems, 28, 2015.
[66] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. International Conference on Machine Learning, 2012.
[67] Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via sampling. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
[68] Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. International Conference on Learning Representations, 2021.
[69] Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34, 2021.
[70] Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pages 15-36, 2000.
[71] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 19 2019.
[72] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[73] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 2022.
14

---

[74] Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. In International Conference on Learning Representations, 2021.
[75] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954 28422995--2867, 2021.
[76] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offiine policy optimization. Advances in Neural Information Processing Systems, 11144129914442 2020.
15

---

Appendices
A Proofs
Proposition A.1. Consider some state in a ID state space, and some action a. Assume (a) that there is an ensemble of N Gaussian transition finnciins, each denoled Tú with mean μi and standard deviation OA: (Ti(s' 18, a) = N (2)), and (b) that the mean of each Gaussian, pi normally distributed with mean μo and standard deviation o Ps N NNu, g)) The N transition functions jointly define T, the transition function for Bayesian MDP, M. Assume that for some policy n, the risk-sensitive value is linear around with some linearity constant, K:
(" ( MM = V (Po. M) + K(ss Po)
Then, the value of randomly drawn successor state is distributed according to ( (H (M, =3
A.1 Proof of Proposition 4.2
Proof: The joint probability of sampling ensemble member with mean, (4, and then sampling successor state, is:
P(pi,s') P(us)P(s' (G).
We marginalise over
P()) J P(u))((s') px)du
P(() and P(s') p4) are both Gaussian. Substituting the appropriate probability density functions we have that
00
1
(14 (p) 207
(K(- 2n
P(s')
exp
exp
dps
OE
2T
OAV 2
(of-p9)2
2/09 = -Nuo, + + +o)).
exp
(9)
VOa +
0 V
2T
Thus, s' is normally distributed with mean Mo and standard deviation Vog+ + 0².. The random vari- able K's'' - (40) is therefore normally distributed with mean and standard deviation [K| / + o² Finally, V(s') = V(Ho) + K(s' -Mo) is also normally distributed with mean V(Ho) and standard deviation [ 0 + t D
Corollary 4.3 follows directly from Proposition and the formula for the CVaR for Gaussian
random variable
34
B Risk Measures
In our experiments, we make use of two different risk measures our algorithm: the conditional value at risk (CVaR) [59], and the Wang risk measure 77). For parameter a, CVaR is the mean of the a-portion the worst outcomes: PCVAR (2, a) = E[z ~ 2 z < Fz (ax)), where Fz is the CDF of Z. For parameter 7). the Wang risk measure for random variable can be defined as the expectation under distorted cumulative distribution function |Wang (Z, 1)) = x x
F2 (F2((2)) (z)) and the distortion function is g(r) = Φ + 7), where is the CDF of the standard normal distribution. In both instances, the parameters and n control the level of risk-aversion.
16

---

We test our approach with these two risk measures because CVaR is commonly used and easy to interpret, but only considers the a-portion of the distribution, while the Wang risk measure takes into consideration the entire distribution. However, the general approach we propose is compatible with any coherent risk measure.
B.1 Computing Perturbed Distributions
Here, we explain how the distribution over candidate successor states, E V is modified for CVaR and the Wang risk measures in Algorithm[] I To avoid the need for excessive notation assume that all sampled candidate successor states in are unique, and that no two successor states have exactly the same value, i.c. V(s))= o (s for any two different sy 8 V.
Conditional Value Risk Consider probability space (Ω, P), where Ω is the sample space.
F is the event space, and P is probability distribution over F. For a continuous random variable Z, CVaR is the mean of the a-portion of the worst outcomes: pCvaR a) = E[z ~ Z | < Fz (a)].
where Fz is the CDF of Z. For CVaR confidence level a, the corresponding risk envelope is:
Bevan., (P) = {5 ((W) [0. 2 Yws /f
((w)P(w)dw= =1}
(10)
wEn
The risk envelope allows the likelihood for any outcome be increased by at most 1/a. In Algo- rithm [ for state-action pair (s, a). we approximate the distribution over successor states as a discrete uniform distribution over candidate successor states s' E N where Vi is a set of m states sampled from (ss, a, s'). This discrete distribution T' a, s') m for all s' e
In Line 10 of Algorithm| we denote by Ç* the adversarial perturbation to the discrete transition distribution:
g arg min
Σ T(s (5,,a,8) E(8) V
(EB,((((,a.)). s'EW
Define Ve the random variable representing the value of the successor state, V=(('), when the successor state drawn from s' [(s, a,-). Also define the value at risk confidence level (0, 1], which is the a-quantile of a distribution: VaR,(Z) minfz F2 2 a). Then, from the risk envelope in Equation 10
we can derive that the adversarial perturbation
if V(s') < VaR, (Ver).
(cvan(s)-T(ss,a,3 as)
0,
if V(s)) > VaR, (Vrr). (11) if V(s') = VaR, (Va)).
EveR m.a (V()') Vara),
The last line of Equation II ensures that the perturbed distribution is a valid probability distribution (as required by the risk envelope in Equation I0).
Wang Risk Measure For parameter y,, the Wang risk measure for random variable, Z, can be defined as the expectation under a distorted cumulative distribution function PWang (Z, 77) 1 2 az (Fz(z))dz. The distortion function is g(r) = ( + 17), where is the CDF the standard normal distribution.
To compute the perturbation to as, according to the Wang risk measure, we order the candidate successor states according their value. We use the subscript indicate the ordering s where V(x1) < (s²) < ((4)) < (s'm). Then, the perturbed probability distribution is:
g (
if i if Fiiinn m i-
Ewang (s') T( a, s()
((
(12)
The transition distribution perturbed according to the Wang risk measure assigns gradually decreasing probability to higher-value successor states. The parameter y controls the extent to which the probabilities are modified. For large 7) >> 0, almost all of the probability placed on the worst outcome(s), and 77 + 0, the perturbed distribution approaches uniform distribution.
17

---

C Implementation Details
The code for the experiments can be found at gtgthubb..om//aaa-nrgg/////
C.1 Model Training
We represent the model as an ensemble of neural networks that output a Gaussian distribution over the next state and reward given the current state and action:
To's' r|s, = (No a), (s))))
Following previous works 77
76 58 during model training train ensemble of 7 such dynamics models and pick the best 5 models based on the validation error on a held-out test set of 1000 transitions from the dataset D. Each model in the ensemble is 4-layer feedforward neural network with 200 hidden units per layer. P(T) D) is assumed to be uniform distribution over the 5 models. Increasing the number of networks in the ensemble has been shown to improve uncertainty estimation 44 However, we used the same ensemble as previous works to facilitate a fair comparison.
The learning rate for the model training is 3e-4 for both the MLE pretraining, and it is trained using the Adam optimiser [36] The hyperparameters used for model training are summarised in Table 4
C.2 Policy Training
We sample batch of 256 transitions train the policy and value function using soft actor-critic (SAC) 30
We set the ratio of real data to ʃ = 0.5 for all datasets, meaning that we sample 50% of the batch transitions from D and the remaining 50% from D. We chose this value because we found it worked better than using 100% synthetic data (i.e. 0), and was found to work well in previous works [75 588. We represent the Q-networks and the policy as three layer neural networks with 256 hidden units per layer.
For SAC (30 we use automatic entropy tuning, where the entropy target is set to the standard heuristic of -d-immm (A). The only hyperparameter that modify from the standard implementation of SAC is the learning rate for the actor, which we set to le-4 as this was reported to work better in the CQL paper (40] which also utilised SAC. For reference, the hyperparameters used for SAC are included in Table
C.3 1R2R Implementation Details
When sampling successor states in Algorithm I we sample m 10 candidate successor states to generate the set V. For each run, we perform 2,000 iterations of 1,000 gradient updates. The rollout policy used to generate synthetic data is the current policy. The base hyperparameter configuration for 1R2R that is shared across all runs shown in Table
In our main experiments, the only parameters that we vary between datasets are the length of the synthetic rollouts (k), and the risk measure parameter. Details are included in the Hyperparameter Tuning section below. In Appendix D.4, we additionally include results where we increase m from its base value, and increase the size of the ensemble.
C.4 1R2R Hyperparameter Tuning
The hyperparameters that we tune for 1R2R are the length of the synthetic rollouts (k), and the parameter associated with the chosen risk measure (a for 1R2RevaR 1y for 1R2Rwang). For each dataset, we select the parameters that obtain the best online performance for the desired objective for that dataset. The objectives for each of the datasets are: optimising expected value for D4RL MuloCo and optimising static CVaRg.1 for all other domains.
The rollout length is tuned from k E {1, 5). For each of the two versions of our algorithm, IR2RovaR and W22g. the parameters are tuned from the following settings:
1R2Rcv the rollout length, k and CVaR parameter, a, are selected from: (k, E {1,5} x (0.5, 0.7, 0.93.
18

---

IR2R wang: the rollout length, k, and Wang risk parameter, 77, are selected from: (k, m) [1.5} x {0.1, 0.5, 0.753.
Note that while the risk measures are only moderately risk-ave averse, in the dynamic risk approach of IR2R, they are applied recursively at each step. This can result in a high degree of risk-aversion over many steps 24
Please refer to Table for a list of the hyperparameters used for IR2R for each dataset, and to Table for the base set of hyperparameters that is shared across all datasets.
Table 4: Base hyperparameter configuration shared across all runs of 1R2R.
Hyperparameter
Value
critic learning rate
3e-4
actor learning rate
le-4
SAC
discount factor (?)
0.99
soft update parameter (r)
5c-3
target entropy
dim(A)
batch size
256
of model networks
no. of clites ratio data (f)
1R2R
no. of iterations (Nier)
2000
no. of candidate successor states (m)
10
rollout policy
current
Table 5: Hyperparameters used by IR2R for each dataset, chosen according best online perfor- mance. is the rollout length, is the parameter for CVaR, and is the parameter for the Wang risk measure.
IR2R (CVaR)
IR2R (Wang)
k
1
Medium Medium Medium Random
HaliCheetah
0.1
Hopper
5
0.7
0.5
Walker2D
5
0.9
5
0.1
HallCheetah
0.9
0.1
Hopper
0.9
0.1
Walker2D
5
0.9
0.1
Expert Replay
HalfChectah
5
0.9
0.1
Hopper
0.5
0.75
Walker2D
0.9
0.1
HalfChectah
0.9
0.1
Hopper
5
0.9
0.1
Walker2D
0.7
0.5
Medium
Hopper (Mod)
5
0.9
0.1
Walker2D (Mod)
0.9
0.5
Hopper (High)
0.9
0.1
Walker2D (lligh)
0.1
Hoopper (Mod)
5
0.7
0.5
Medium
Replay
Walker2D (Mod)
0.1
Hopper (High)
5
0.5
0.75
Walker2D (High)
0.9
0.1
Hopper (Mod)
0.7
0.75
Medium
Expert
Walker2D (Mod)
0.5
0.75
Hopper (High)
5
0.9
5
0.5
Walker2D (High)
0.7
0.5
HIVTreatment
0.7
0.5
CurrencyExchange
0.5
0.75
19

---

D Additional Results
D.1
MuJoCo Stochastic Expected Value Results
Table 6: Expected value results for the Stochastic MuJoCo benchmarks using the normalisation procedure proposed by [21]. We report the normalised performance during the last iterations of training averaged over 5 seeds. + captures the standard deviation seeds. Highlighted numbers
indicate results within 10% of the best score. "div." indicates that the value function diverged for at least one run.
IR2Revae
IR2Roee
Ablation
O-RAAC
CODAC
RAMBO
CQ1.
IQI.
TD3-BC
Medum
Walker2D (Mod)
Hopper (Mod)
67.22:3..7
67.5 7.8
62.2 6.5
65.3 7.0
61.1 10.0
27.6 25.4
988
1..
61.3 1.5
60.7
72.9 14.7
36.5+222 2.2
50.22 0.6
68.4+
20.6 = 4.1
45.0 + 1.5
41.4 17.4
48.4
Hopper
64.4
74.7
06.3
(High)
58.8
54.8
Malhon
Walker2D (Mod)
Hopper (Med)
M0.5 18.6
51.9 1-091110 9.4
51.7.:111 11.4
76.
$1.4 3.0
9.
24.2 1 6,8
61.2 10.7
47.8 19.7
67.6 11.1
63.5 3: 4.5
06.9 10.8
30.1
00.5 0.0
16.2 2.8
38.22
48.4
Hopper
10.8
1600
64.5
58.4
Walker2D (High)
19.1
78.4
Malhon
Expent
Walker2D (Mod)
Hopper (Med)
94.1 17:2
72.2 7.3
71.8 42.2
77.11 9.1
61.1 41.7
div.
828.887.7
27.6 12.4
96.6
81.7 4.8
71.9 20.7
58.0 2: 22.3
106.7
94.2 1.1
107.5 7.7
69.2 1: 1.7
100.0 2.
50.5 2: 1.9
Hopper (lligh)
54.1 14.11.5.5
60.2
48.4 1.4
18.9 10 11.2
1.39..3..
60.0 1.001.6..
88.5 5.5
64.0 16.5
89.9
Walker2D (High)
15.4
44.4
72.7
D.2 HIV Treatment Return Distributions
Figure iillustrate the return distributions for each algorithm from all evaluation episodes aggregated over 5 seeds per algorithm. Note that in the ablation in Figure 3c] training became unstable for subset of the runs. This resulted in very poor performance for subset of the runs. Because the plots Figure are aggregated across all runs for each algorithm, this results in the strongly multi-modal performance observed for the ablation in Figure 3c
20

---

0.05
20
40
100
120
140
160
(a) IR2RevaR
0.00
20
100
140
160
(b) 1R2R 1R2R
0.05
(c) Ablation
0.00
20
100
120
140
160
ORAAC
(e) CODAC
0.05
0.00
20
40
60
80
100
120
140
160
(1) RAMBO
(g) CQL
0.05
0.00
20
40
60
80
100
120
140
160
(h) IQL
TD3+BC
Figure 3: Return distributions for each algorithm the HIV Treatment domain. The plots generated by aggregating all evaluation episodes. There are 200 evaluation episodes per iteration for each of the last 10 iterations of training, across 5 seeds per algorithm.
D.3 Currency Exchange Return Distributions
To compare aivereeerse behaviour to risk-neutral behaviour on the Currency Exchange domain, we plot the return distributions for IR2RcvaR and RAMBO Figure 4 We observe that RAMBO performs well in expectation, but has long tail of poor returns, resulting in poor performance for static CV 1R2R performs less well in expectation, but there less variability in the returns, resulting in better performance for static CVaRp.1-
21

---

0.04
0.02
0.000
20
40
60
80
100
120
140
160
Total Reward
Figure 4: Return distributions for the Currency Exchange domain, aggregated 5 seeds. Blue: IR2RevaR. Grey: RAMBO. Solid vertical bars indicate average return, dashed vertical bars indicate the mean of the worst 10% of returns (i.e. static CV) 1R2RevaR performs better for static CVaRg.1 while RAMBO performs hetter in expectation.
D.4 Results with Modified Parameters
In Table we present additional results for 1R2R using parameter values that have been modified from those used in main experiments. Due to computational limitations, we focus on IR2RevaR on the Stochastic Walker2D domains.
The results in Table 7 show that by either increasing the number of networks in the model ensemble (1R2RcvaR (Larger Ensemble)), or increasing the number of samples successor states drawn from the model (1R2RcvaR (More Samples)), similar or slightly worse performance is obtained compared the base version of our algorithm. This indicates that is not possible improve the performance of IR2R by naively increasing the computation available.
In the base configuration of algorithm we the ratio of real data sampled during policy training to f = 0.5, as described in Appendix C.2 The results in Table 7 show that the performance degrades significantly when training the policy and value function entirely from synthetic samples generated by the model (IR2RcvaR (No Real Data)). This indicates that sampling training data from the real dataset in addition to synthetic data is crucial to obtaining strong performance. This is probably because sampling data from the real dataset helps to mitigate errors due to inaccuracies in the model. As discussed in Section [6, the model may be somewhat inaccurate for these domains due to the restriction of using Gaussian transition functions.
Table 7: Additional results using different parameter values for IR2RcvaR The table includes results for both the Expected Value and the CVaR objective, as indicated by the Objective column. The column IR2RevaR (Base) shows the results for 1R2RevaR using the base algorithm configuration described in Appendix C In the remaining columns, the algorithm is modified as follows. IR2RcvaR (Larger Ensemble) uses ensemble of 20 total neural networks with the best 15 selected. IR2R (No Real Data) does not sample any real data when performing updates (i.e. the ratio of real data is = 0). 1R2R (More Samples) samples 50 successor states when generating the of successor states in Line 8 of Algorithm (i.e. = 50).
Objective
IM3Row (Blase)
IR2How (Larger Ensemble!
1R2Reca ONo Real Data)
1R2I2cue (More Samples)
Medium
Walker2D (Mod)
C'aRRa
299
27.5 7.5
15.0 7.7
26.6 5.1
Experted Value
67.3
60.7 0.9
36.0 19.3
58.2
19.4 12.6
0.7
(High)
Expested Value
24.4
28.0
$1.4
Modoum
Walker2D (Mod)
10.1.:77
27.0
10.8 1d.4
26.8 6.0
Replyy
Expected Value
53.9 1: 9.4
$3.3 7.2
30.9 19.7
51.2 à: 8.4
Walker2D (High)
CVaRo.
46.0 10.1
1.7
6.5
Expected Value
84.14:555
$1.8 4.8
26.4 10.8
78.7 7.00
Modhna Exper
(Mod)
U'aaou
Expested Value
72:2 7.5
31.7
78.5.+ 7.2
22.9 + 9.6
70.3 11.0
16.7
Walker2D (High)
CNaRu.
10.8+44.3
11.5+ 1.7
7.3 5.4
12.3
Expected Value
49.9 15.4
12.7 9.9
21.3 11.2
SI1.4 10.5
22

---

E Experiment Details
E.1
Computation Requirements
Each run of IR2R requires 16-24 hours of computation time on system with an Intel Core i7-8700 CPU @ 3.20GHz CPU and an NVIDIA GeForce GTX 1070 graphics card.
E.2
Domains
The D4RL MuloCo domains are from the D4RL offline RL benchmark 21 In this work, we introduce the following domains.
Currency Exchange This domain an adaptation of the Optimal Liquidation Problem to offline RL. We assume that the agent initially holds 100 units of currency A, and wishes to convert all of this to currency B before deadline T, subject to an exchange rate which changes stochastically. At each time step, the agent may choose how much (if any) of currency A to convert to currency B at the current exchange rate.
An aggressive strategy is to delay converting the currency, in the hope that random fluctuations will lead to a more favourable exchange rate before the deadline T'. However, this may lead to a poor outcome if the exchange rate decreases and fails to recover before the deadline. More risk-averse strategies are to either: gradually convert the currency over many steps, b) immediately convert all of the currency even if the current exchange rate is mediocre, to avoid the risk of having to convert the money at even worse exchange rate in the future, to meet the deadline T.
The state space is 3-dimensional, consisting of: the timestep. / E {0, 1,......... ,.....T): the amount of currency remaining, E [0, 100]; and the current exchange rate, p [0, x)). We assume that the exchange rate evolves according to an Ornstein-Uhlenber process, which is commonly used for financial modelling 49
dp: = 0(u dt) + adW,
(13)
where Wg denotes the Wiener process. In our implementation we set: = 1.5, o = 0.2, 0 0.05. The exchange rate at the initial time step is Po ~ (1, 0.05²). The action space single dimensional, a E []., We assume that a positive action corresponds to the proportion of m to convert to currency B at the current step, while negative action corresponds converting The reward is equal to the amount of currency B received at each step.
of the currency.
To generate the dataset, generate data from a random policy that at each step chooses uniform random proportion currency to convert with probability 0.2, and converts none of the currency with probability 0.8.
HIV Treatment The environment is based on the implementation in 26 of the physical model described by [20]. The patient state a 6-dimensional continuous vector representing the concentra- tions of different types of cells and viruses in the patients blood. There two actions, corresponding to giving the patient Reverse Transcriptase Inhibitors (RTI) or Protease Inhibitors (PI) (or both). Unlike 20 which considered discrete actions, modify the domain allow for continuous actions representing variable quantities of each drug. Following previous work 33
we assume that the
Table 8: Expected return received by randomly initialised policy and an expert policy trained online using SAC. The expert SAC policy is trained online for 2e6 updates for the Stochastic MuJoCo domains, and 5e6 updates for HIVTreatment. In Tables 2 and B the values reported are normalised between 0 and 100 using the procedure from [1 and the values in this table.
Random Policy
Expert Policy
Hopper (Moderate-Noisee
3049.4
Walker2D (Moderatee-Noiee)
-1.9
4479.2
Hopper (High-Noise)
11.2
2035.1
Walker2D (High-Noise)
-5.3
4101.3
HIVTreatment
-48.7
5557.9
CurrencyExchange
0.0
135.0
23

---

treatment at each time step is applied for 20 days, and that there are 50 time steps. Therefore, each episode corresponds to total period of 1000 days. Also following [3), we introduce stochasticity into the domain by adding noise to the efficacy of each drug (e1 and 62 in (0)). The noise added to the efficacy is normally distributed, with a standard deviation of 15% of the standard value.
For this domain, the dataset is constructed in the same manner as the Medium-Replay datasets from D4RL [21]. Specifically, a policy is trained with SAC [30] until reaches 40% of the performance of an expert policy. The replay buffer of data collected until reaching this point is used as the offline dataset.
The reward function is that used by (26), except that we scale the reward by dividing it by 1e6. A low concentration of HIV viruses is rewarded, and a cost is associated with the magnitude of treatment applied, representing negative side effects. The average reward received by a random policy and an expert policy trained online using SAC (30) can be found in Table [ In the paper, values normalised between O and 100 using the method proposed by [21] and the values in Table Thus, a score of 100 represents the average reward of an expert policy, and 0 represents the average reward of a random policy.
Stochastic MuJoCo These benchmarks are obtained by applying random perturbation forces to the original Hopper and Walker2D MuJoCo domains. We chose focus on these two domains as there is a risk of the robot falling over before the end of the episode. The perturbation forces represent disturbances due strong wind or other sources of uncertainty. The force is applied in the x-direction at each step, according to a random walk on the interval |- fMAX. fMAX]. At each step, the change in force relative to the previous step is sampled uniformly from Af fni fMAX. 0.1 fMAX). This is motivated by the fact that the strength of wind is often modelled as random walk (52)
We test using two levels of noise applied. For the Moderate-Noise datasets, the level of noise results in moderate degradation in the performance of an online SAC agent, compared to the original deterministic domain. For the High-Noise datasets, the level of noise results in significant degradation in the performance of an online SAC agent. The value of fMAX is set to the following in each of the domains:
Hopper Moderate Noise: fMAX 2.5 Newtons.
Hopper High-NNise: SMAX- = Newtons.
Walker2D Moderate-Noise: fMAX- 7 Newtons.
Walker2D High-Noise: fMAX= 12 Newtons.
We construct three datasets: in the same manner as the D4RL. datasets (21). The expert policy is policy trained online using SAC until convergence. The medium policy is trained until it obtains x 40% of the performance of the expert policy. The Mediun-Repla, dataset consists of the replay buffer collected while training the medium policy. The Medium dataset contains transitions collected from rolling out the medium policy. The Medium- Medium-Expert dataset contains a mixture of transitions generated by rolling out both the medium and the expert policy.
Medium, Medium-RRepla and Eepeen-Expert. These are generated
Following previous works 668 47 we assume that the objective on all of the stochastic domains sis to optimise the static CVa (i.e. the average total reward in the worst 10% of runs).
E.3 Evaluation Procedure
For D4RL MuJoCo, evaluate the policy for 10 episodes at each of the last 10 iterations of training, for each of 5 seeds. The average normalised total reward received throughout this evaluation is reported in the D4RL results table (Table II.
For Stochastic MuJoCo, we evaluate the policy for 20 episodes at each of the last 10 iterations of training. CVaRg.1 is computed by averaging the total reward in the worst two episodes at each iteration. The value of CVaRo.1 is then averaged across the last 10 iterations, and across each of the seeds. These are the results reported for CVaRg.1 in Table For HIVTreatment and CurrencyExchange, we evaluate the policy for 200 episodes at each of the Tast 10 iterations of training, for each of seeds. The average total reward for the worst 20 episodes at each evaluation is used to compute CV
24

---

E.4 Baselines
We compare 1R2R against offline RL algorithms designed for sisk-senstiivity (ORAAC 668 and CODAC 47 in addition to performant risk-neutral model-based (RAMBO (58, COMBO 75
and
MOPO 76 and model-free (IQL. [38. TD3+BC (22). CQL [4), and ATAC
algorithms.
For the baselines, we use the following implementations:
O-RAAC:
ggtub.ccom//urrt55500RRAACA
CODAC: gtthbccom//aooNaa211110000
RAMBO: gihuhh.omm/maar--itt///naooo
IQL:: guthbb.com//lcsssko///nniicii.. g_leaningg
TD3+BC: 1133+BCC.ghhhhcomm/ffi////111111
CQL and COMMBBO.gghuu...mm//aauusneoodor
ATAC: http:///ggtthhocommmmmmmmotttttr
MOРO: http://athhubccom//aaaayyyu//opp
The results for D4RL MuJoCo-v2 for RAMBO, ITA,ATAC, COMBO, MOPO and TD3+BC taken from the original papers. Because the original CQL paper used the -vO domains, we report the results for CQL the -v2 domains from [69]. The IQL papers does not report results for the random datasets, so we generate the results for IQL. in the random datasets by running the official implementation algorithm ourselves. The results for CODAC in the original paper use D4RL MuJoCo-vO, and O-RAAC does not report results on the D4RL benchmarks. Therefore, we obtain the results for D4RL MuJoCo-v2 for CODAC and O-RAAC by running the algorithms ourselves.
For the stochastic domains (Currency Exchange, HIV Treatment, and MuJoCo Stochastic) the results were obtained by running the algorithms ourselves.
E.5 Baseline Hyperparameter Tuning
We tune each of the algorithms online to achieve the best performance for the desired objective in each dataset. The desired optimisation objectives in each domain are: expected value in D4RL MuJoCo and static CVaRo. in all other domains. O-RAAC and CODAC can be configured to optimise a static risk measure, or expected value. Therefore, we set O-RAAC and CODAC optimise the appropriate objective for each dataset.
For each of the baseline algorithms, tune the following parameters based on the best online performance (averaged over seeds) for the target objective. Where a hyperparameter value is not specified, it set the default value in the corresponding original paper.
CODAC: The risk measure optimise is set to the desired objective (Expected value/CVaRo.1) For Stochastic MuloCo and HIV'Treatment, we tune the parameters in the same manner is the original paper [47), choosing (w,{) {0.1, 1, 10) x {-1, 10). For Stochastic MuJoCo and HIVTreatment we use the default critic learning rate of
Meritie 3e 5. To generate the results for CODAC on D4RL MuJoCo-v2, we use the same parameters as reported in the paper for D4RL MuJoCo-vO.
O-RAAC: The risk measure to optimise is set to the desired objective (Expected value/CVaRo.11. Following the original paper [68] we tune the imitation learning weighting: X E {0.25, 0.5).
RAMBO: We tuned the rollout length and adversarial weighting from: (k. E {(1,3e 4). (2 4), (5, 3e - 4), (5. 0)}. This is the same procedure in the origi- nal paper [58], except that we additionally included a rollout length of as we found this was necessary for RAMBO to train stably on some of the stochastic domains.
IQL: For Stochastic MuJoCo, we used the same parameters as for D4RL MuJoCo in the original paper [38 For HIVTreatment and Currency Exchange, we chose the best performance between the parameters for D4RL. MuJoCo and D4RI. AntMaze from the original paper.
25

---

CQL: Following the original paper 40), we tuned the Lagrange parameter, T E {5, 10). All other parameters are set their default values.
TD3+BC: Following the original paper,
to 2.5 22
MOPO: The rollout length and conservative weighting are each tuned within {1,5) following the original paper [76].
The conservative weighting was tuned within {0.5, 1, 1,5) following the original paper [75].
COMBO:
ATAC: 3 was tuned within {0,4-1,4-1 4- 1,,4* 1,,4,,5 1,42,44 4.
26


Reviews:
[{"review_id": "XnFLmtMqt6", "reviewer": "Reviewer_RPWe", "summary": "One Risk to Rule Them All (1R2R) is an offline-RL method that seeks to reduce aleatoric and epistemic uncertainty. Their method is simple; the authors introduce the notion of risk in the bellman update by adding learned adversarial perturbation to the learned transition dynamics models in the model-based setting. This is different from prior risk-based methods which adversarially perturbate the value function as opposed to the TD function.", "questions": "N/A", "limitations": "- Unrelated, it would be nice to have a model-free version, as I'd generally avoid model-based approaches (as does industry). For example, if you add the adversarial perturbation to the distributional value function (instead of the learned TD function), despite prior work.\n- I'd upgrade this paper to strong accept (as possibly higher) if it explains WHY adding risk-aversion reduces epistemic and aleotoric uncertainty, as this is a very beautiful insight. Bonus if there is math.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The narrative/flow is clear; this work focuses on minimizing both aleatoric and epistemic uncertainty, both of which hamper offline RL agent performance. \n- Proposed methodology is simple, just add a learned adversarial perturbation to TD model. \n- Figure 1 is an excellent figure that illustrates that the learned risk-aware value function penalizes both uncertainties. Yet, it is difficult to absorb/fully understand, and it might be better to split it into two subfigures. For example, highlight regions with different colors outside of dataset and regions where  TD uncertainty is high. (It is also nice to have to compare the value function to CQL's too, to show that CQL might be aleatoric unaware)\n- Evaluation is thorough and evaluated over difficult environments with large continuous action spaces.", "weaknesses": "- The intro was not clear. At the end, it says that epistemic uncertainty is avoided through model-ensemble variance. However, the analogy to aleatoric uncertainty was not made clear. It might be better to straight-up state that risk-averse RL applied in the offline setting reduces both uncertainties for reasons X and Y.\n- Authors claims method is simpler than prior work due to only considering risk; however, the risk incorporates model-based approaches, which is much more complicated and much harder to work in practice.\n- Why does risk-averse RL reduce aleatoric uncertainty? (good to show math in appendix) It seems to be a miraculous cure to a problem.\n- Methodologically, this paper is not novel; it seems more like A + B, where A = model-based ensemble in MOPO and B = risk-averse RL objective in CvAR, applied in the offline RL setting. The real novelty seems to stem from identifying applicability of risk in offline RL.\n- The Risk background is difficult to understand; suggest removing it and immediately diving into the Bellman risk-sensitive representation.\n- It'd be nice to have a figure illustrating the final algorithm architecture and well as typical reward x training iteration graphs in the evaluation.\n- Where is the evaluation against MOPO/COMBO, another model-based offline RL method? It's be worthwhile to show that this work is the SOTA model-based offline RL approach.\n- Suggesting removing one of the evaluation environments and focusing more on ablation and analysis.\n- Why does 1R2R flounder on expert dataset (medium expert)?\n- Another way to avoid such uncertainty is the field of safe RL, which seeks to minimize constraint violations while maximizing environmental reward (Recovery RL). There is a lot of prior work in this field and is somehow obfuscated in this paper.\n-Nit: There are several typos, including in Algorithm 1 line 9,  and Section 3 Static and Dynamic Risk MDP formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time you have spent reviewing our paper. We respond to your main comments below. We are unable to respond to all comments due to space constraints.\n\n### Why does our approach reduce both epistemic and aleatoric uncertainty?\n\nIn Proposition 1, and the explanation that follows, we provide mathematical justification for why our approach reduces both epistemic and aleatoric uncertainty. We will add this to the final paper.\n\nIn Proposition 1, we assume that each transition function in the ensemble is Gaussian, with standard deviation $\\sigma_A$ representing the aleatoric uncertainty. We also assume that the distribution of the means of each member of the ensemble is Gaussian with standard deviation $\\sigma_E$ (representing epistemic uncertainty). Proposition 1 and the corollary that follows demonstrate that our approach penalises taking actions that have *either* large $\\sigma_A$ or $\\sigma_E$. We discuss Proposition 1 in detail below.\n\n---\n\n**Proposition 1:**  Consider some state $s$ in a 1D state space, and some action $a$. Assume 1) that there is an ensemble of $M$ Gaussian transition functions, each denoted $T_i$ with mean $\\mu_i$ and standard deviation $\\sigma_A$: $\\\\{T_i(s' | s, a)  = \\mathcal{N}(\\mu_i, \\sigma_A^2) \\\\}_{i=1}^M$; and 2) that the mean of each Gaussian, $\\mu_i$, is normally distributed with mean $\\mu_0$ and standard deviation $\\sigma_E$: $\\mu_i \\sim \\mathcal{N}(\\mu_0, \\sigma_E^2)$. According to Eq. 5, to sample a successor state $s'$ from $\\overline{T}$ we must sample a transition function $T_i$ from the ensemble, and then sample $s' \\sim T_i(s' | s, a)$.\n\nAssume that the value of the successor state is linear around $\\mu_0$ with some linearity constant, $K$:\n\n$$\nV(s') = V(\\mu_0) + K (s' - \\mu_0)\n$$\n\nThen $V(s')$ is distributed according to: $V(s') \\sim \\mathcal{N}\\Big(\\mu = V(\\mu_0), \\sigma^2 = K^2(\\sigma_A^2 + \\sigma_E^2) \\Big)$.\n\n**Proof**: The joint probability of sampling an ensemble member with mean, $\\mu_i$, and then sampling a successor state, $s'$, is:\n\n$$\nP(\\mu_i, s') = P(\\mu_i) P(s'\\ |\\ \\mu_i).\n$$\n\nWe marginalise over  $\\mu_i$:\n\n$$\nP(s') = \\int_{-\\infty}^\\infty P(\\mu_i) P(s'\\ |\\ \\mu_i) \\mathrm{d} \\mu_i \n$$\n\n$P(\\mu_i)$ and $P(s'\\ |\\ \\mu_i)$ are both Gaussian. Substituting the appropriate probability density functions we have that\n\n$$\nP(s') = \\int_{-\\infty}^\\infty \\frac{1}{\\sigma_E \\sqrt{2\\pi} } \\exp^{- \\frac{(\\mu_i - \\mu_0 )^2}{2 \\sigma_E^2}}  \\frac{1}{\\sigma_A \\sqrt{2\\pi} } \\exp^{- \\frac{(s' - \\mu_i )^2}{2 \\sigma_A^2}} \\mathrm{d} \\mu_i \\\\\n= \\frac{1}{\\sqrt{\\sigma_A^2 + \\sigma_E^2} \\sqrt{2\\pi} } \\exp^{-\\frac{(s' - \\mu_0 )^2}{2 \\sqrt{\\sigma_E^2 + \\sigma_A^2}}} = \\mathcal{N}(\\mu_0, \\sigma_E^2 + \\sigma_A^2).\n$$\n\nThus, $s'$ is normally distributed with mean $\\mu_0$ and standard deviation $\\sqrt{\\sigma_E^2 + \\sigma_A^2}$. The random variable $K(s' - \\mu_0)$ is therefore normally distributed with mean 0 and standard deviation $|K|\\sqrt{\\sigma_E^2 + \\sigma_A^2}$. Finally, $V(s') = V(\\mu_0) + K (s' - \\mu_0)$ is also normally distributed with mean $V(\\mu_0)$ and standard deviation $|K|\\sqrt{\\sigma_E^2 + \\sigma_A^2}$. $\\square$\n\n---\n\nIn Proposition 1, $\\sigma_E$ defines the level of disagreement between the members of the ensemble, and therefore represents the level of epistemic uncertainty. $\\sigma_A$ represents the aleatoric uncertainty. The proposition tells us that if either $\\sigma_A$ or $\\sigma_E$ are high, then there is high variability over the value of the successor state when sampling from the ensemble (i.e. sampling from $\\overline{T}$ in Eq. 5).\n\nRisk measures penalise high variability. The risk-sensitive Bellman equation (Eq. 4) applies a risk-measure the value over the successor state. Therefore, applying Eq. 4 to samples from the ensemble penalises executing state-action pairs for which either $\\sigma_A$ or $\\sigma_E$ is high. Thus, our approach penalises actions with *either* high aleatoric or epistemic uncertainty. It therefore favours choosing actions that have *both* low aleatoric and epistemic uncertainty.\n\nThe following corollary uses the example of conditional value at risk (CVaR) to show how the value computed in the risk-sensitive Bellman equation decreases as either $\\sigma_A$ or $\\sigma_E$ increase.\n\n---\n\n**Corollary:** Under the assumptions in Proposition 1, the CVaR at confidence level $\\alpha$ of the value of the successor state is:\n\n$$\n\\mathrm{CVaR}_\\alpha\\big(V(s')\\big) = V(\\mu_0) - \\frac{|K|\\sqrt{\\sigma_E^2 + \\sigma_A^2}}{\\alpha \\sqrt{2 \\pi}}  \\exp ^ {-\\frac{1}{2}( \\Phi^{-1}(\\alpha) )^2}\n$$\n\nwhere $\\Phi^{-1}$ is the inverse of the standard normal CDF.\n\n**Proof:** This follows directly from Proposition 1 and the formula for the CVaR for a Gaussian random variable [1].\n\n[1] Khokhlov, V.. \"Conditional value-at-risk for elliptical distributions.\" European Journal of Economics and Management (2016).\n\n---\n\n### Extra Figures\n\nIn Fig 1 of the Global Response, we have added a summary figure. We will add this to the paper. We have also included example plots of the performance vs training iterations. We will add these plots for all datasets to the final paper.\n\n### Extra Ablation\n\nWe have added an **additional ablation**. See the response to Reviewer 8yT3 for details.\n\n### MOPO/COMBO\n\nWe have **expanded the results** to include MOPO, COMBO, and ATAC. See the response to Reviewer unPm for details.\n\n### Medium-Expert Datasets\n\nPoor performance of model-based approaches on high-quality datasets is an established issue [1, 2]. The cause of this is an open question in the community which we wish to investigate in future work. Model-based methods are generally strongest on noisy sub-optimal data [1].\n\n[1] Lu, C, et al. \"Challenges and opportunities in offline reinforcement learning from visual observations.\" TMLR, 2023\n\n[2] Rigter, M, et al. \"Robust adversarial model-based offline reinforcement learning.\" NeurIPS, 2022\n\n### Related Work\n\nWe will expand the related work to discuss constrained RL and recovery RL."}, {"review_id": "N5pGj6gmZf", "reviewer": "Reviewer_uPGp", "summary": "This paper considers the problem of offline reinforcement learning for risk-averse decision-making with the distributional shift. The core insight of this paper is to incorporate epistemic uncertainty (from the distributional shift) and aleatoric uncertainty (from usual statistical errors) together and develop a way to simply penalize high uncertainty, which can handle both problems simultaneously. ", "questions": "**How to choose the distribution over MDPs to capture the epistemic uncertainty?** \n\nOne of the key ideas in this paper is to use the distribution over MDPs to represent the epistemic uncertainty over the real environment. As far as I understand, on page 7 (Implementation Details), the authors suggest using an ensemble of neural networks to estimate different distributions over the MDPs. It seems to me that this choice of model classes is actually the fundamentally important part of capturing epistemic uncertainty. For example, if we include a large number of different neural networks (e.g., more than 100), the epistemic uncertainty is guaranteed to be larger than cases when we include a smaller number of neural networks (e.g., 5). **But, without an explicit model or assumption about the distributional shifts, how can researchers choose what models to estimate MDPs for approximating the epistemic uncertainty?**\n\n**In Section 5 (Experiments), how did the authors choose the model class for capturing the epistemic uncertainty?** \n\nI think this is an important question as there is an inherent tradeoff --- if we include more models, we can be more robust to a wide range of potential distributional shifts, but if we include too many models, we might be too conservative. **Is there any theoretical guidance about how to approximate the epistemic uncertainty?** \n\n", "limitations": "The paper clarifies its limitation clearly. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the biggest strength of this paper is the novel idea of combining both epistemic uncertainties from the distributional shift and aleatoric uncertainty together and using a simple risk-aversion algorithm to handle both problems. I think this key idea can be applied not only to offline reinforcement learning but also to other areas where the distributional shifts are key, e.g., predictions and causal inference problems. ", "weaknesses": "I do not think there is any obvious weakness in this paper. I only have some suggestions below. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable time you have spent reviewing our paper.\n\nWe agree that deciding how to represent epistemic uncertainty in deep learning is an important problem, and a key aspect of our work.\n\nWe used a small ensemble of 5 neural networks to ensure that our work is a fair comparison to previous works, which also use a small ensemble. MOPO [1], COMBO [2], and RAMBO [3] each use an ensemble of 5 models, and MOReL [4] uses an ensemble of 4 models. Like our work, MOPO and MOReL use these small ensembles to estimate epistemic uncertainty.\n\nWe did conduct experiments using a larger ensemble of 15 models. These results can be found in Table 7 in Appendix C.4. We found that the results obtained with a larger ensemble are similar to our main results which utilise the small ensemble.\n\nThe influence of the size of the model ensemble on the uncertainty estimate *depends on the type of uncertainty estimate used*. The recent work [5] analyses this empirically in the context of offline RL. The variance or standard deviation between the members of the ensemble is fairly stable as the size of the ensemble changes - see Figure 2 of [5]. However, other metrics, such as the maximum disagreement between any two members of the ensemble, increase significantly as the size of the ensemble increases (again, see Figure 2 of [5]). In the latter case, we would indeed expect the uncertainty estimate to be highly sensitive to the number of models in the ensemble. \n\nIn our approach, the risk-sensitive Bellman equation (Equation 4) penalises the *variability* in samples drawn from the ensemble. As stated above, the variance between ensemble members is quite stable as the size of the ensemble increases. Thus, we do not expect our approach to be highly sensitive to the size of the ensemble, as supported by the results in Table 7. \n\nFurthermore, we present a theoretical analysis in Proposition 1 in the response to Reviewer RPWe that thoroughly analyses how our approach mitigates epistemic and aleatoric uncertainty in an ensemble of models.\n\n[1] Yu, Tianhe, et al. \"Mopo: Model-based offline policy optimization.\" NeurIPS (2020).\n\n[2] Yu, Tianhe, et al. \"Combo: Conservative offline model-based policy optimization.\" NeurIPS (2021).\n\n[3] Rigter, Marc, Bruno Lacerda, and Nick Hawes. \"Rambo-rl: Robust adversarial model-based offline reinforcement learning.\" NeurIPS (2022).\n\n[4] Kidambi, Rahul, et al. \"Morel: Model-based offline reinforcement learning.\" **NeurIPS (2020).\n\n[5] Lu, Cong, et al. \"Revisiting Design Choices in Offline Model-Based Reinforcement Learning.\" ICLR, 2022."}, {"review_id": "fASrxwxhNj", "reviewer": "Reviewer_8yT3", "summary": "> Our concerns are addressed by the authors. Thanks for your effort. We will upgrade the rating to weak accept. \n\nThis paper introduces a model-based risk-averse algorithm that utilizes risk aversion as a mechanism to jointly address the distributional shift problem and risk-related decision-making problems in risk-sensitive offline RL. The authors employ a risk-averse risk measure to simultaneously perturb the belief distribution and transition functions in an adversarial manner, enabling risk aversion towards both epistemic uncertainty and aleatoric uncertainty. Risk aversion to epistemic uncertainty reduces transition probabilities to state-action pairs that are out-of-distribution, and risk aversion to aleatoric uncertainty helps avoid actions that are inherently risky. The authors conduct experiments in both deterministic and stochastic environments, confirming the superior performance of the 1R2R algorithm.", "questions": "1. How does the argmin is performed in Line 10 of Algorithm 1?\n2. In my opinion, instead of characterizing this approach as risk-aversion methods, I think it resembles a conservative value update method. By introducing adversarial perturbations to the transition function, it increases the likelihood of transitioning to low-value successor states. Can the author clarify this?\n3. For the experimental results of D4RL MuJoCo, how can you validate that the performance improvement of the 1R2R algorithm is indeed due to addressing the 'distributional shift' problem?\n4. In Line 139, the equation is written as ($Z_{\\mathrm{MDP}}=\\sum_{t=0}^{\\infty} \\gamma R\\left(s_t, a_t\\right)$). Shouldn’t this be ($Z_{\\mathrm{MDP}}=\\sum_{t=0}^{\\infty} \\gamma^{t} R\\left(s_t, a_t\\right)$)?", "limitations": "This approach is less flexible compared to the previous methods which can address distributional shift and risk-sensitivity separately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written. The original contributions are highlighted clearly.\n2. The paper provides a concise and understandable introduction to the background and related work. It is highly reader-friendly, ensuring ease of comprehension for the intended readers.\n3. The paper demonstrates a well-organized structure, and the approach of utilizing risk aversion to address the distributional shift problem is innovative and holds theoretical viability.\n", "weaknesses": "1. While adopting a joint risk-aversion mechanism for both epistemic uncertainty and aleatoric uncertainty is simple and efficient, I think this approach is less flexible compared to previous methods which can address distributional shift and risk-sensitivity separately. \n\n2. The design of ablation experiment merely removes the risk-sensitivity, which I think to be overly simplistic. Ablation Experiments part requires appropriate expansion, such as comparing the computational costs with the approaches using distributional value functions, and comparing the performance between using dynamic risk measures and using static risk measures.\n\n3. I think 1R2R does not have a significant advantage over other baselines (i.e., RAMBO). Can the authors discuss more in-depth regarding this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable time you have spent reviewing our paper. We respond to your comments and questions below.\n\n### Flexibility of the Approach\n\nWe agree that our approach is simpler yet less flexible than existing approaches that can separately adjust how they address each source of uncertainty. In Lines 370-374, we discuss this limitation. We also discuss that a straightforward extension of our work is to apply *composite* risk measures [1]. Composite risk measures allow the aversion to aleatoric vs epistemic uncertainty to be adjusted separately. This is a direction we wish to explore in future work.\n\n[1] Eriksson, Hannes, et al. \"Sentinel: taming uncertainty with ensemble based distributional reinforcement learning.\" *UAI*, 2022.\n\n### Ablations\n\nAs you have suggested, we have **expanded the ablations** in Tables 1 and 2 of the global response.\n\nThe two key components of our approach are 1) risk-averse sampling and 2) using an ensemble of models. We include an additional ablation so that we separately ablate each of these components.\n\nWhen either of these components is removed, we observe that training may diverge. This illustrates that both components of our approach are necessary to prevent value function instability resulting from backing-up out-of-distribution value estimates (i.e. avoiding epistemic uncertainty). The ablations also show that risk-averse sampling is necessary to achieve strong risk-sensitive performance to aleatoric uncertainty on the stochastic Currency Exchange domain.\n\n### Advantage Over Baselines\n\nOn the D4RL domains (which are deterministic) our approach obtains similar to performance to some of the strongest baselines such as RAMBO. However, in the stochastic Currency Exchange domain (where the objective is to optimise CVaR) only 1R2R and CODAC are able to achieve strong risk-averse performance. In this domain, RAMBO performs well for average performance but not for the desired objective of CVaR. CODAC performs poorly on most of the other benchmarks. Thus, 1R2R is the only algorithm that achieves strong performance in the deterministic domains, as well as generating performant risk-averse behaviour in the stochastic domains.\n\n### Question 1.\n\nBecause the minimisation is performed over a discrete set of candidate successor states, it is a straightforward optimisation problem. To perform the minimisation, we first sort the sampled successor states from lowest to highest value. Then, the solution is computed in closed-form according to Equations 11 and 12 in Appendix A.1 for CVaR and the Wang risk measure, respectively.\n\n### Question 2.\n\nConservative value function methods (e.g. [2]) aim to learn a value function that lower-bounds the value function in the true environment. Risk-measures consider mappings from *distributions* to scalar values. This means that risk-measures can be viewed as the expectation under an adversarially perturbed distribution [3]. Our approach optimises the policy under an adversarially perturbed transition function, which is why we characterise it as a risk-sensitive approach.\n\nUnder some conditions, it may also be the case that the value function that our algorithm computes is a lower bound on the true value function. If this is the case, our approach can also be considered conservative in this sense. Exploring this connection is an exciting direction for future work.\n\n[2] Kumar, Aviral, et al. \"Conservative Q-learning for offline reinforcement learning.\" NeurIPS, 2020.\n\n[3] Artzner, Philippe, et al. \"Coherent measures of risk.\" Mathematical finance, 1999.\n\n### Question 3.\n\nIf we naively apply online RL algorithms in the offline setting, the RL algorithm will back-up values estimated for out-of-distribution (OOD) state-action pairs, where the value function is inaccurate. This is referred to as the “distributional shift” problem. Because RL algorithms choose the highest value actions, this leads to the policy selecting OOD state-action pairs where the Q-value is overestimated. Repeatedly performing Bellman backups in this manner often leads to the value function diverging (see [4], Figure 1).\n\nIn the results for our algorithm, we do not observe this issue of the value function diverging. However, if we ablate either component of our approach (either the ensemble or risk-averse sampling), then for some datasets the value function is unstable and diverges. These runs are labelled “div.” in the results when the value function reaches a magnitude greater than 1e9. See Table 1 of the Global Response.\n\nThus, the fact that our algorithm avoids this issue of value function instability indicates that our algorithm prevents the value function being updated on transitions to out-of-distribution state-action pairs. This is because we do not see an alternative reason for the instability of the ablated algorithms, which operate using the same dataset and hyperparameters. Therefore, this is a strong indication that our algorithm mitigates the issue of distributional shift,\n\n[4] Kumar, Aviral, et al. \"Stabilizing off-policy Q-learning via bootstrapping error reduction.\" NeurIPS, 2019.\n\n### Question 4.\n\nThank you for pointing out this typo, we will fix this."}, {"review_id": "375BfREyFq", "reviewer": "Reviewer_unPm", "summary": "This paper proposes a new model-based offline RL algorithm that learns a policy which is risk-averse (wrt aleatoric uncertainty measured by some dynamic risk measure) and pessimistic (wrt epistemic uncertainty from modeling). First, the algorithm learns a posterior distribution over MDP transitions given the dataset, using an ensemble of neural nets. Then, the algorithm samples successor states from the worst-case perturbation of the learned model, effectively modeling the Bellman equation for dynamic risk. ", "questions": "1. Why is Line 184 labeled \"Problem 1\"? I don't see a Problem 2.\n2. Why do you take on the Bayesian perspective for learning \\bar T? Another way of learning the model is the MLE, ie train a single neural net to maximize log likelihood of successor state, so I'm wondering why you choose to learn P(T|D) and then derive \\bar T from that? \n3. How is the adversarial perturbation actually computed in practice (Line 10)? Does you need to perform a two-stage optimization procedure?\n4. How are the hyperparameters of the method selected, and how many online samples were used for hyperparameter tuning? (In theory, papers about offline RL should only be using offline samples for algorithm design, but in practice, this is almost always violated. So, it would be nice to report how many online samples were used, especially given that this paper introduces many more hyperparameters (ie Table 4)). \n5. Have you compared with ATAC (Adversarially Trained Actor Critic for Offline Reinforcement Learning), which is one of the SOTA offline RL methods?", "limitations": "Please see weaknesses/questions.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The method is clean and the high-level ideas are clearly explained. Extensive experiments are also encouraging and paper is pretty well written.", "weaknesses": "1. My understanding is that this method is targeting the dynamic risk of the coherent risk measure with envelope B_p. However, all of the evaluations, as well as most of the paper, seem to suggest that 1R2R is a good algorithm for risk-neutral and static risk objectives (which the authors indeed show in experiments). It's not clear to me why optimizing for the dynamic risk should result in good performance for risk-neutral or static risk objectives, and so the method seems more hand-wavy. For example, if we assume that model learning succeeds, can we prove any PAC bounds wrt risk-neutral or static risk objectives? \n2. It would be interesting to have some ablations on the success of 1R2R: are the improvements in performance mostly due to pessimism wrt aleatoric uncertainty, or epistemic uncertainty, or simply model-based offline RL? \n\nAlso, please see Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable time you have spent reviewing our paper. We respond to your comments and questions below.\n\n### Theory and Dynamic vs Static Risk\n\nWe have **added theoretical justification** for why our approach avoids both epistemic and aleatoric uncertainty in Proposition 1 in the response to Reviewer RPWe.\n\nAs you point out, our approach utilises dynamic risk and we evaluate our approach using static risk objectives. This is necessary because we cannot evaluate the performance for dynamic risk in practice. In Lines 355-365 of the paper we discuss this limitation.\n\nDynamic and static risk can be related in the following way. Dynamic risk is equivalent to adversarially modifying the transition dynamics *independently* at every time step according to the risk envelope. Static risk is equivalent to adversarially modifying the transition dynamics at each time step subject to the constraint that the perturbation to each *entire trajectory* remains within the risk envelope (see e.g. [1, 2]). Thus, the optimal behaviour with respect to both static and dynamic risk is the optimal behaviour in an adversarially perturbed MDP, where the worst transitions are made more likely. This is why we expect optimal behaviour for dynamic risk to be similar to that for static risk.\n\nWe are unaware of PAC bounds relating the two. We think this is an exciting direction for future work.\n\n[1] Chow, Yinlam, et al. \"Risk-sensitive and robust decision-making: a CVaR optimization approach.\" NeurIPS, 2015.\n\n[2] Rigter, Marc, et al. \"Planning for risk-aversion and expected value in MDPs.\" ICAPS, 2022.\n\n### Additional Ablations\n\nAs you have suggested, we have **expanded the ablations** in Tables 1 and 2 of the global response.\n\nThe two key components of our approach are 1) risk-averse sampling and 2) using an ensemble of models. In the new results, we separately ablate each of these components.\n\nWhen either of these components is removed, we observe that training may diverge. This illustrates that both components are necessary to prevent value function instability resulting from backing-up out-of-distribution value estimates (i.e. avoiding epistemic uncertainty). The ablations also show that risk-averse sampling is necessary to achieve strong risk-averse performance to aleatoric uncertainty on the stochastic Currency Exchange domain.\n\nWe have also added comparisons to more model-based RL algorithms, COMBO and MOPO. We observe that 1R2R easily outperforms these algorithms on the stochastic domains (Currency Exchange and HIV Treatment). This indicates that the strong performance of 1R2R is not solely due to being model-based.\n\n### Question 1.\n\nIn the final version we will use the name “Dynamic Risk Optimisation in Bayesian MDP” for our problem formulation, to make this clearer.\n\n### Question 2.\n\nWe define $\\overline{T}$ using a distribution over models, $P(T | D)$, so that we can relate learning in an ensemble of models to risk-sensitivity. Recall that risk-measures require a *distribution* as input (they map distributions to scalar values), and that we require an ensemble of models to capture epistemic uncertainty. Given our ensemble of models, we need to be able to define $\\overline{T}$: the distribution over successor states given the model ensemble. We do this using Equation 5 in the paper, which defines $\\overline{T}$ using $P(T | D)$.\n\nIn our practical implementation (Lines 246-247), we use a finite set of $M$ neural networks to define the model ensemble, and each of these networks is separately trained using MLE. We assume that $P(T | D)$ is a uniform distribution over these $M$ networks, i.e. $P(T | D) = \\frac{1}{M}$ for all $T$ in the ensemble. This assumption means that when sampling from $\\overline{T}$ in Line 8 of Algorithm 1, we first uniformly sample from the $M$ models in the ensemble, and then sample a successor from that model.\n\n### Question 3.\n\nThe optimisation problem in Line 10 of Algorithm 1 is over a discrete set of samples, making it straightforward. We first sort the samples from lowest to highest value successor states. Then, for the risk measures we consider (CVaR and Wang) the solution is computed in closed-form using Equations 11 and 12 in Appendix A.1.\n\n### Question 4.\n\nTo choose the hyperparameters, we evaluate each policy for 10 episodes in D4RL and 20 episodes in the stochastic domains at the end of training (for 5 seeds). Then, we choose the hyperparameters that obtained the best performance for the desired objective. We also optimise the hyperparameters for the baselines in this manner.\n\n### Question 5. Additional Baselines\n\nIn the Global Response, we have added **additional results** for ATAC as well as MOPO and COMBO (as requested by Reviewer RPWe). We observe that MOPO performs very poorly on all domains. 1R2R only slightly outperforms ATAC and COMBO on the D4RL domains. However, on the stochastic domains (Currency Exchange and HIV Treatment) 1R2R easily outperforms both ATAC and COMBO. \n\nThis demonstrates that 1R2R outperforms current state-of-the-art algorithms. It also highlights that we should not rely on benchmarking only on deterministic environments.\n\n**Experiment Details**\n\nThe D4RL results for ATAC, MOPO, and COMBO are from the original papers. \n\nFor Currency Exchange and HIV Treatment, we ran the algorithms ourselves. For ATAC and MOPO we used the official implementations. For COMBO, there is no official implementation so we used the implementation from d3rlpy.  \n\nWe tuned the hyperparameters to obtain the best performance for the CVaR objective:\n\n- MOPO: rollout length and conservative weighting each tuned within {1, 5} following the original paper.\n- COMBO: the conservative weighting was tuned within {0.5, 1, 5} following the original paper.\n- ATAC: $\\beta$ was tuned within {$0, 4^{-4}, 4^{-2}, 1, 4^2, 4^4$}. "}]

Author rebuttal:
Thank you for your reviews.

Please find attached the Global Response. In the attachment, we provide updated results which include an **additional ablation**, as well as comparisons to **additional baseline algorithms**: MOPO, COMBO, and ATAC.

We also include additional figures. The summary figure (Figure 1 in the global response) will be added to the final version of the paper. We have also provided examples of performance vs training iterations (Figure 2 in the global response). These figures will be added for all datasets in the final version.

Finally, we have also added stronger **theoretical justification** for why our approach reduces both aleatoric and epistemic uncertainty. Please see Proposition 1 in the response to Reviewer RPWe. This theoretical motivation will also be added to the final version of our paper.

Please let us know if there is anything further that you would like to discuss!

Kind regards,

The authors

Meta-review: All the reviewers like the work and see it above the bar. There are questions/concerns about parts of the paper that some were answered during the rebuttal phase. It would be great if the authors revise their work to properly address the reviewers' comments when they prepare the camera-ready version.
Decision: Accept (poster)