SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[74] Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. In International Conference on Learning Representations, 2021.
[75] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
[76] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Proposition A.1. Consider some state  $s$  in a 1D state space, and some action  $a$ . Assume (a) that there is an ensemble of  $N$  Gaussian transition functions, each denoted  $T_{i}$  with mean  $\mu_{i}$  and standard deviation  $\sigma_{A}$ :  $\{T_{i}(s'|s,a) = \mathcal{N}(\mu_{i},\sigma_{A}^{2})\}_{i=1}^{N}$ ; and (b) that the mean of each Gaussian,  $\mu_{i}$ , is normally distributed with mean  $\mu_{0}$  and standard deviation  $\sigma_{E}$ :  $\mu_{i} \sim \mathcal{N}(\mu_{0},\sigma_{E}^{2})$ . The  $N$  transition functions jointly define  $\overline{T}$ , the transition function for Bayesian MDP,  $\overline{M}$ . Assume that for some policy  $\pi$ , the risk-sensitive value is linear around  $\mu_{0}$  with some linearity constant,  $K$ :
A.1 Proof of Proposition 4.2
Proof: The joint probability of sampling an ensemble member with mean,  $\mu_{i}$ , and then sampling a successor state,  $s^{\prime}$ , is:
$$
P \left(\mu_ {i}, s ^ {\prime}\right) = P \left(\mu_ {i}\right) P \left(s ^ {\prime} \mid \mu_ {i}\right).
$$
We marginalise over  $\mu_{i}$ :
$$
P (s ^ {\prime}) = \int_ {- \infty} ^ {\infty} P (\mu_ {i}) P (s ^ {\prime} \mid \mu_ {i}) \mathrm {d} \mu_ {i}
$$
$P(\mu_i)$  and  $P(s' \mid \mu_i)$  are both Gaussian. Substituting the appropriate probability density functions we have that
$$
\begin{array}{l} P (s ^ {\prime}) = \int_ {- \infty} ^ {\infty} \frac {1}{\sigma_ {E} \sqrt {2 \pi}} \exp^ {- \frac {(\mu_ {i} - \mu_ {0}) ^ {2}}{2 \sigma_ {E} ^ {2}}} \frac {1}{\sigma_ {A} \sqrt {2 \pi}} \exp^ {- \frac {(s ^ {\prime} - \mu_ {i}) ^ {2}}{2 \sigma_ {A} ^ {2}}} \mathrm {d} \mu_ {i} \\ = \frac {1}{\sqrt {\sigma_ {A} ^ {2} + \sigma_ {E} ^ {2}} \sqrt {2 \pi}} \exp^ {- \frac {(s ^ {\prime} - \mu_ {0}) ^ {2}}{2 \sqrt {\sigma_ {E} ^ {2} + \sigma_ {A} ^ {2}}}} = \mathcal {N} (\mu_ {0}, \sigma_ {E} ^ {2} + \sigma_ {A} ^ {2}). (9) \\ \end{array}
$$
Thus,  $s'$  is normally distributed with mean  $\mu_0$  and standard deviation  $\sqrt{\sigma_E^2 + \sigma_A^2}$ . The random variable  $K(s' - \mu_0)$  is therefore normally distributed with mean 0 and standard deviation  $|K| \sqrt{\sigma_E^2 + \sigma_A^2}$ . Finally,  $V(s') = V(\mu_0) + K(s' - \mu_0)$  is also normally distributed with mean  $V(\mu_0)$  and standard deviation  $|K| \sqrt{\sigma_E^2 + \sigma_A^2}$ .
Corollary 4.3 follows directly from Proposition 1 and the formula for the CVaR for a Gaussian random variable [34].
B Risk Measures
In our experiments, we make use of two different risk measures in our algorithm: the conditional value at risk (CVaR) [59], and the Wang risk measure [70]. For parameter  $\alpha$ , CVaR is the mean of the  $\alpha$ -portion of the worst outcomes:  $\rho_{\mathrm{CVaR}}(Z,\alpha) = \mathbb{E}[z\sim Z\mid z\leq F_Z^{-1}(\alpha)]$ , where  $F_{Z}$  is the CDF of  $Z$ . For parameter  $\eta$ , the Wang risk measure for random variable  $Z$  can be defined as the expectation under a distorted cumulative distribution function  $\rho_{\mathrm{Wang}}(Z,\eta) = \int_{-\infty}^{\infty}z\frac{\partial}{\partial z} g(F_Z(z))\mathrm{d}z$ , and the distortion function is  $g(\tau) = \Phi (\Phi^{-1}(\tau) + \eta)$ , where  $\Phi$  is the CDF of the standard normal distribution. In both instances, the parameters  $\alpha$  and  $\eta$  control the level of risk-aversion.
We test our approach with these two risk measures because CVaR is commonly used and easy to interpret, but only considers the  $\alpha$ -portion of the distribution, while the Wang risk measure takes into consideration the entire distribution. However, the general approach we propose is compatible with any coherent risk measure.
B.1 Computing Perturbed Distributions
Here, we explain how the distribution over candidate successor states,  $s' \in \Psi$  is modified for CVaR and the Wang risk measures in Algorithm 1. To avoid the need for excessive notation we assume that all sampled candidate successor states in  $\Psi$  are unique, and that no two successor states have exactly the same value, i.e.  $V(s_i') \neq V(s_j')$ , for any two different  $s_i', s_j' \in \Psi$ .
Conditional Value at Risk Consider a probability space  $(\Omega, \mathcal{F}, P)$ , where  $\Omega$  is the sample space,  $\mathcal{F}$  is the event space, and  $P$  is a probability distribution over  $\mathcal{F}$ . For a continuous random variable  $Z$ , CVaR is the mean of the  $\alpha$ -portion of the worst outcomes:  $\rho_{\mathrm{CVaR}}(Z, \alpha) = \mathbb{E}\left[z \sim Z \mid z \leq F_Z^{-1}(\alpha)\right]$ , where  $F_Z$  is the CDF of  $Z$ . For CVaR at confidence level  $\alpha$ , the corresponding risk envelope is:
$$
\mathcal {B} _ {\mathrm {C V a R} _ {\alpha}} (P) = \left\{\xi : \xi (\omega) \in \left[ 0, \frac {1}{\alpha} \right] \forall \omega , \int_ {\omega \in \Omega} \xi (\omega) P (\omega) \mathrm {d} \omega = 1 \right\} \tag {10}
$$
The risk envelope allows the likelihood for any outcome to be increased by at most  $1 / \alpha$ . In Algorithm 1, for state-action pair  $(s,a)$ , we approximate the distribution over successor states as a discrete uniform distribution over candidate successor states  $s' \in \Psi$ , where  $\Psi$  is a set of  $m$  states sampled from  $\overline{T}(s,a,s')$ . This discrete distribution is  $\widehat{T}(s,a,s') = \frac{1}{m}$  for all  $s' \in \Psi$ .
In Line 10 of Algorithm 1, we denote by  $\widehat{\xi}^*$  the adversarial perturbation to the discrete transition distribution:
$$
\widehat {\xi} ^ {*} = \underset {\xi \in \mathcal {B} _ {\rho} (\widehat {T} (s, a, \cdot))} {\arg \min } \sum_ {s ^ {\prime} \in \Psi} \widehat {T} (s, a, s ^ {\prime}) \cdot \xi (s ^ {\prime}) \cdot V ^ {\pi} (s ^ {\prime})
$$
Define  $\mathcal{V}_{s'}$  as the random variable representing the value of the successor state,  $V^{\pi}(s')$ , when the successor state is drawn from  $s' \sim \widehat{T}(s, a, \cdot)$ . Also define the value at risk at confidence level  $\alpha \in (0, 1]$ , which is the  $\alpha$ -quantile of a distribution:  $\mathrm{VaR}_{\alpha}(Z) = \min \{z \mid F_Z(z) \geq \alpha\}$ . Then, from the risk envelope in Equation 10, we can derive that the adversarial perturbation is:
$$
\widehat {\xi} _ {\mathrm {C V a R}} \left(s ^ {\prime}\right) \cdot \widehat {T} (s, a, s ^ {\prime}) = \left\{ \begin{array}{l l} \frac {1}{m \cdot \alpha}, & \text {i f} V \left(s ^ {\prime}\right) <   \operatorname {V a R} _ {\alpha} \left(\mathcal {V} _ {s ^ {\prime}}\right). \\ 0, & \text {i f} V \left(s ^ {\prime}\right) > \operatorname {V a R} _ {\alpha} \left(\mathcal {V} _ {s ^ {\prime}}\right). \\ 1 - \sum_ {s ^ {\prime} \in \Psi} \frac {1}{m \cdot \alpha} \mathbb {1} \left[ V \left(s ^ {\prime}\right) <   \operatorname {V a r} _ {\alpha} \right], & \text {i f} V \left(s ^ {\prime}\right) = \operatorname {V a R} _ {\alpha} \left(\mathcal {V} _ {s ^ {\prime}}\right). \end{array} \right. \tag {11}
$$
The last line of Equation 11 ensures that the perturbed distribution is a valid probability distribution (as required by the risk envelope in Equation 10).
Wang Risk Measure For parameter  $\eta$ , the Wang risk measure for random variable,  $Z$ , can be defined as the expectation under a distorted cumulative distribution function  $\rho_{\mathrm{Wang}}(Z,\eta) = \int_{-\infty}^{\infty}z\frac{\partial}{\partial z} g(F_Z(z))\mathrm{d}z$ . The distortion function is  $g(\tau) = \Phi (\Phi^{-1}(\tau) + \eta)$ , where  $\Phi$  is the CDF of the standard normal distribution.
To compute the perturbation to  $\widehat{T}(s, a, \cdot)$  according to the Wang risk measure, we order the candidate successor states according to their value. We use the subscript to indicate the ordering of  $s_i'$ , where  $V(s_1') < V(s_2') < V(s_3') \ldots < V(s_m')$ . Then, the perturbed probability distribution is:
$$
\widehat {\xi} _ {\text {W a n g}} \left(s _ {i} ^ {\prime}\right) \cdot \widehat {T} (s, a, s _ {i} ^ {\prime}) = \left\{ \begin{array}{l l} g \left(\frac {1}{m}\right), & \text {i f} i = 1. \\ g \left(\frac {i}{m}\right) - g \left(\frac {i - 1}{m}\right), & \text {i f} 1 <   i <   m. \\ 1 - g \left(\frac {i - 1}{m}\right), & \text {i f} i = m. \end{array} \right. \tag {12}
$$
The transition distribution perturbed according to the Wang risk measure assigns gradually decreasing probability to higher-value successor states. The parameter  $\eta$  controls the extent to which the probabilities are modified. For large  $\eta >> 0$ , almost all of the probability is placed on the worst outcome(s), and as  $\eta \to 0$ , the perturbed distribution approaches a uniform distribution.
The code for the experiments can be found at github.com/marc-rigter/1R2R.
C.1 Model Training
We represent the model as an ensemble of neural networks that output a Gaussian distribution over the next state and reward given the current state and action:
$$
\widehat {T} _ {\phi} \left(s ^ {\prime}, r | s, a\right) = \mathcal {N} \left(\mu_ {\phi} (s, a), \Sigma_ {\phi} (s, a)\right).
$$
Following previous works [75, 76, 58], during model training we train an ensemble of 7 such dynamics models and pick the best 5 models based on the validation error on a held-out test set of 1000 transitions from the dataset  $\mathcal{D}$ . Each model in the ensemble is a 4-layer feedforward neural network with 200 hidden units per layer.  $P(T\mid \mathcal{D})$  is assumed to be a uniform distribution over the 5 models. Increasing the number of networks in the ensemble has been shown to improve uncertainty estimation [44]. However, we used the same ensemble as previous works to facilitate a fair comparison.
The learning rate for the model training is 3e-4 for both the MLE pretraining, and it is trained using the Adam optimiser [36]. The hyperparameters used for model training are summarised in Table 4.
C.2 Policy Training
We sample a batch of 256 transitions to train the policy and value function using soft actor-critic (SAC) [30]. We set the ratio of real data to  $f = 0.5$  for all datasets, meaning that we sample  $50\%$  of the batch transitions from  $\mathcal{D}$  and the remaining  $50\%$  from  $\widehat{\mathcal{D}}$ . We chose this value because we found it worked better than using  $100\%$  synthetic data (i.e.  $f = 0$ ), and it was found to work well in previous works [75, 58]. We represent the  $Q$ -networks and the policy as three layer neural networks with 256 hidden units per layer.
For SAC [30] we use automatic entropy tuning, where the entropy target is set to the standard heuristic of  $-\dim(A)$ . The only hyperparameter that we modify from the standard implementation of SAC is the learning rate for the actor, which we set to 1e-4 as this was reported to work better in the CQL paper [40] which also utilised SAC. For reference, the hyperparameters used for SAC are included in Table 4.
C.3 1R2R Implementation Details
When sampling successor states in Algorithm 1, we sample  $m = 10$  candidate successor states to generate the set  $\Psi$ . For each run, we perform 2,000 iterations of 1,000 gradient updates. The rollout policy used to generate synthetic data is the current policy. The base hyperparameter configuration for 1R2R that is shared across all runs is shown in Table 4.
In our main experiments, the only parameters that we vary between datasets are the length of the synthetic rollouts  $(k)$ , and the risk measure parameter. Details are included in the Hyperparameter Tuning section below. In Appendix D.4, we additionally include results where we increase  $m$  from its base value, and increase the size of the ensemble.
C.4 1R2R Hyperparameter Tuning
The hyperparameters that we tune for 1R2R are the length of the synthetic rollouts  $(k)$ , and the parameter associated with the chosen risk measure ( $\alpha$  for  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  or  $\eta$  for  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$ ). For each dataset, we select the parameters that obtain the best online performance for the desired objective for that dataset. The objectives for each of the datasets are: optimising expected value for D4RL MuJoCo and optimising static  $\mathrm{CVaR}_{0.1}$  for all other domains.
The rollout length is tuned from  $k \in \{1, 5\}$ . For each of the two versions of our algorithm,  $1R2R_{\mathrm{CVaR}}$  and  $1R2R_{\mathrm{Wang}}$ , the parameters are tuned from the following settings:
-  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$ : the rollout length,  $k$ , and CVaR parameter,  $\alpha$ , are selected from:  $(k,\alpha) \in \{1,5\} \times \{0.5,0.7,0.9\}$ .
-  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$ : the rollout length,  $k$ , and Wang risk parameter,  $\eta$ , are selected from:  $(k,\eta)\in \{1,5\} \times \{0.1,0.5,0.75\}$ .
Note that while the risk measures are only moderately risk-averse, in the dynamic risk approach of 1R2R, they are applied recursively at each step. This can result in a high degree of risk-aversion over many steps [24].
Please refer to Table 5 for a list of the hyperparameters used for 1R2R for each dataset, and to Table 4 for the base set of hyperparameters that is shared across all datasets.
Table 4: Base hyperparameter configuration shared across all runs of 1R2R.
Hyperparameter	Value
SAC	critic learning rate	3e-4
actor learning rate	1e-4
discount factor (γ)	0.99
soft update parameter (τ)	5e-3
target entropy	-dim(A)
batch size	256
IR2R	no. of model networks	7
no. of elites	5
ratio of real data (f)	0.5
no. of iterations (Niter)	2000
no. of candidate successor states (m)	10
rollout policy	current π
Table 5: Hyperparameters used by 1R2R for each dataset, chosen according to best online performance.  $k$  is the rollout length,  $\alpha$  is the parameter for CVaR, and  $\eta$  is the parameter for the Wang risk measure.
1R2R (CVaR)	1R2R (Wang)
k	α	k	η
Random	HalfCheetah	5	0.9	5	0.1
Hopper	5	0.7	5	0.5
Walker2D	5	0.9	5	0.1
Medium	HalfCheetah	5	0.9	5	0.1
Hopper	5	0.9	5	0.1
Walker2D	5	0.9	5	0.1
Medium Replay	HalfCheetah	5	0.9	5	0.1
Hopper	1	0.5	1	0.75
Walker2D	1	0.9	1	0.1
Medium Expert	HalfCheetah	5	0.9	5	0.1
Hopper	5	0.9	5	0.1
Walker2D	1	0.7	1	0.5
Medium	Hopper (Mod)	5	0.9	5	0.1
Walker2D (Mod)	1	0.9	1	0.5
Hopper (High)	5	0.9	5	0.1
Walker2D (High)	1	0.7	1	0.1
Medium Replay	Hopper (Mod)	5	0.7	5	0.5
Walker2D (Mod)	1	0.9	1	0.1
Hopper (High)	5	0.5	5	0.75
Walker2D (High)	1	0.9	1	0.1
Medium Expert	Hopper (Mod)	5	0.7	5	0.75
Walker2D (Mod)	1	0.5	1	0.75
Hopper (High)	5	0.9	5	0.5
Walker2D (High)	1	0.7	1	0.5
HIVTreatment	1	0.7	1	0.5
CurrencyExchange	1	0.5	1	0.75
D.1 MuJoCo Stochastic Expected Value Results
Table 6: Expected value results for the Stochastic MuJoCo benchmarks using the normalisation procedure proposed by [21]. We report the normalised performance during the last 10 iterations of training averaged over 5 seeds.  $\pm$  captures the standard deviation over seeds. Highlighted numbers indicate results within  $10\%$  of the best score. "div." indicates that the value function diverged for at least one run.
1R2RCVaR	1R2RWang	Ablation	O-RAAC	CODAC	RAMBO	CQL	IQL	TD3+BC
Medium	Hopper (Mod)	67.2 ± 5.7	65.9 ± 7.0	64.1 ± 10.0	15.6 ± 9.4	50.7 ± 3.0	72.9 ± 14.7	50.2 ± 0.6	50.6 ± 4.1	48.4 ± 1.4
Walker2D (Mod)	67.3 ± 7.8	62.2 ± 6.3	27.6 ± 23.4	38.8 ± 7.1	44.1 ± 1.5	34.5 ± 2.2	58.4 ± 1.6	45.6 ± 1.3	41.4 ± 17.4
Hopper (High)	57.1 ± 4.5	58.1 ± 5.9	53.0 ± 4.7	28.8 ± 11.0	69.9 ± 2.1	54.4 ± 4.4	74.7 ± 5.1	61.6 ± 8.0	66.3 ± 1.3
Walker2D (High)	59.5 ± 11.8	58.8 ± 8.9	div.	41.6 ± 9.6	65.7 ± 0.6	56.9 ± 3.5	75.2 ± 1.3	54.8 ± 2.0	77.6 ± 2.3
Medium	Hopper (Mod)	83.6 ± 18.6	76.8 ± 9.9	64.5 ± 9.8	15.8 ± 3.0	61.2 ± 10.7	67.6 ± 11.3	66.9 ± 10.8	-0.5 ± 0.0	38.2 ± 4.9
Walker2D (Mod)	53.9 ± 9.4	51.7 ± 11.4	57.4 ± 3.0	24.2 ± 6.8	47.8 ± 19.7	63.6 ± 4.5	50.1 ± 1.9	16.2 ± 2.8	48.4 ± 1.5
Hopper (High)	114.3 ± 10.8	102.0 ± 9.1	48.4 ± 16.0	26.2 ± 16.0	54.5 ± 2.9	54.1 ± 19.7	96.7 ± 13.0	15.1 ± 9.6	58.4 ± 6.1
Walker2D (High)	84.1 ± 5.3	76.4 ± 6.0	75.4 ± 8.0	19.1 ± 4.7	80.4 ± 11.6	84.1 ± 4.0	61.1 ± 3.0	-0.1 ± 0.8	73.4 ± 2.3
Medium	Hopper (Mod)	94.1 ± 17.2	71.3 ± 42.2	64.0 ± 41.7	32.8 ± 8.7	96.6 ± 8.8	71.9 ± 20.7	106.7 ± 3.8	107.5 ± 7.7	100.9 ± 2.0
Walker2D (Mod)	72.2 ± 7.3	77.3 ± 9.1	div.	27.6 ± 12.4	81.7 ± 4.8	58.0 ± 22.3	94.2 ± 1.1	63.2 ± 1.7	93.5 ± 0.9
Hopper (High)	54.1 ± 5.5	63.2 ± 11.0	48.4 ± 7.4	18.9 ± 11.2	79.9 ± 3.5	60.0 ± 6.1	88.3 ± 3.5	64.0 ± 16.5	89.9 ± 1.4
Walker2D (High)	49.9 ± 15.4	44.4 ± 17.6	div.	27.3 ± 13.4	79.5 ± 18.0	31.1 ± 14.9	104.3 ± 2.3	72.7 ± 4.9	107.0 ± 2.3
D.2 HIV Treatment Return Distributions
Figure 3 illustrates the return distributions for each algorithm from all evaluation episodes aggregated over 5 seeds per algorithm. Note that in the ablation in Figure 3c, training became unstable for a subset of the runs. This resulted in very poor performance for a subset of the runs. Because the plots in Figure 3 are aggregated across all 5 runs for each algorithm, this results in the strongly multi-modal performance observed for the ablation in Figure 3c.
(a)  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$
(b)  $1\mathrm{R}2\mathrm{R}_{\mathrm{Wang}}$
(c) Ablation
(d) ORAAC
(e) CODAC
(f) RAMBO
(g) CQL
(h) IQL
(i)  $\mathrm{TD3 + BC}$
Figure 3: Return distributions for each algorithm on the HIV Treatment domain. The plots are generated by aggregating all evaluation episodes. There are 200 evaluation episodes per iteration for each of the last 10 iterations of training, across 5 seeds per algorithm.
D.3 Currency Exchange Return Distributions
To compare risk-averse behaviour to risk-neutral behaviour on the Currency Exchange domain, we plot the return distributions for  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  and RAMBO in Figure 4. We observe that RAMBO performs well in expectation, but has a long tail of poor returns, resulting in poor performance for static  $\mathrm{CVaR}_{0.1}$ . 1R2R performs less well in expectation, but there is less variability in the returns, resulting in better performance for static  $\mathrm{CVaR}_{0.1}$ .
Figure 4: Return distributions for the Currency Exchange domain, aggregated over 5 seeds. Blue:  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$ , Grey: RAMBO. Solid vertical bars indicate average return, dashed vertical bars indicate the mean of the worst  $10\%$  of returns (i.e. static  $\mathrm{CVaR}_{0.1}$ ).  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  performs better for static  $\mathrm{CVaR}_{0.1}$  while RAMBO performs better in expectation.
D.4 Results with Modified Parameters
In Table 7 we present additional results for 1R2R using parameter values that have been modified from those used in our main experiments. Due to computational limitations, we focus on  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  on the Stochastic Walker2D domains.
The results in Table 7 show that by either increasing the number of networks in the model ensemble (1R2RCVaR (Larger Ensemble)), or increasing the number of samples of successor states drawn from the model (1R2RCVaR (More Samples)), similar or slightly worse performance is obtained compared to the base version of our algorithm. This indicates that it is not possible to improve the performance of 1R2R by naively increasing the computation available.
In the base configuration of our algorithm we set the ratio of real data sampled during policy training to  $f = 0.5$ , as described in Appendix C.2. The results in Table 7 show that the performance degrades significantly when training the policy and value function entirely from synthetic samples generated by the model (1R2RCVaR (No Real Data)). This indicates that sampling training data from the real dataset in addition to synthetic data is crucial to obtaining strong performance. This is probably because sampling data from the real dataset helps to mitigate errors due to inaccuracies in the model. As discussed in Section 6, the model may be somewhat inaccurate for these domains due to the restriction of using Gaussian transition functions.
Table 7: Additional results using different parameter values for  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$ . The table includes results for both the Expected Value and the  $\mathrm{CVaR}_{0.1}$  objective, as indicated by the Objective column. The column  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  (Base) shows the results for  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  using the base algorithm configuration described in Appendix C. In the remaining columns, the algorithm is modified as follows.  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  (Larger Ensemble) uses an ensemble of 20 total neural networks with the best 15 selected.  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  (No Real Data) does not sample any real data when performing updates (i.e. the ratio of real data is  $f = 0$ ).  $1\mathrm{R}2\mathrm{R}_{\mathrm{CVaR}}$  (More Samples) samples 50 successor states when generating the set of successor states in Line 8 of Algorithm 1 (i.e.  $m = 50$ ).
Objective	1R2RCVaR (Base)	1R2RCVaR (Larger Ensemble)	1R2RCVaR (No Real Data)	1R2RCVaR (More Samples)
Medium	Walker2D (Mod)	CVaR0.1	29.9 ± 8.7	27.5 ± 7.5	15.0 ± 7.7	26.6 ± 5.1
Expected Value	67.3 ± 7.8	60.7 ± 9.9	34.0 ± 13.3	58.2 ± 6.9
Walker2D (High)	CVaR0.1	19.4 ± 12.6	4.1 ± 4.7	0.6 ± 0.7	9.5 ± 1.0
Expected Value	59.5 ± 11.8	38.7 ± 24.4	28.0 ± 4.9	51.4 ± 9.7
Medium
Replay	Walker2D (Mod)	CVaR0.1	30.1 ± 7.2	27.9 ± 3.5	10.8 ± 14.4	26.8 ± 6.0
Expected Value	53.9 ± 9.4	53.3 ± 7.2	30.9 ± 19.7	51.2 ± 8.4
Walker2D (High)	CVaR0.1	46.0 ± 10.1	39.1 ± 6.9	1.8 ± 1.7	39.6 ± 6.5
Expected Value	84.1 ± 5.3	81.8 ± 4.8	26.4 ± 10.8	79.7 ± 7.0
Medium
Expert	Walker2D (Mod)	CVaR0.1	34.8 ± 6.5	31.7 ± 6.5	7.7 ± 5.6	42.9 ± 16.7
Expected Value	72.2 ± 7.3	73.5 ± 7.2	22.9 ± 9.6	79.3 ± 11.0
Walker2D (High)	CVaR0.1	10.8 ± 4.3	11.5 ± 1.7	7.3 ± 5.4	12.3 ± 0.6
Expected Value	49.9 ± 15.4	52.7 ± 9.9	21.3 ± 11.2	50.4 ± 10.5
E.1 Computation Requirements
Each run of 1R2R requires 16-24 hours of computation time on a system with an Intel Core i7-8700 CPU @ 3.20GHz CPU and an NVIDIA GeForce GTX 1070 graphics card.
E.2 Domains
The D4RL MuJoCo domains are from the D4RL offline RL benchmark [21]. In this work, we introduce the following domains.
Currency Exchange This domain is an adaptation of the Optimal Liquidation Problem [2, 8] to offline RL. We assume that the agent initially holds 100 units of currency A, and wishes to convert all of this to currency B before a deadline  $T$ , subject to an exchange rate which changes stochastically. At each time step, the agent may choose how much (if any) of currency A to convert to currency B at the current exchange rate.
An aggressive strategy is to delay converting the currency, in the hope that random fluctuations will lead to a more favourable exchange rate before the deadline  $T$ . However, this may lead to a poor outcome if the exchange rate decreases and fails to recover before the deadline. More risk-averse strategies are to either: a) gradually convert the currency over many steps, or b) immediately convert all of the currency even if the current exchange rate is mediocre, to avoid the risk of having to convert the money at even worse exchange rate in the future, to meet the deadline  $T$ .
The state space is 3-dimensional, consisting of: the timestep,  $t \in \{0,1,2,\dots,T\}$ ; the amount of currency A remaining,  $m \in [0,100]$ ; and the current exchange rate,  $p \in [0,\infty)$ . We assume that the exchange rate evolves according to an Ornstein-Uhlenbeck process, which is commonly used for financial modelling [49]:
$$
d p _ {t} = \theta (\mu - p _ {t}) d t + \sigma d W _ {t}, \tag {13}
$$