SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $W_{t}$  denotes the Wiener process. In our implementation we set:  $\mu = 1.5$ ,  $\sigma = 0.2$ ,  $\theta = 0.05$ . The exchange rate at the initial time step is  $p_0 \sim \mathcal{N}(1, 0.05^2)$ . The action space is single dimensional,  $a \in [-1,1]$ . We assume that a positive action corresponds to the proportion of  $m$  to convert to currency B at the current step, while a negative action corresponds to converting none of the currency. The reward is equal to the amount of currency B received at each step.
To generate the dataset, we generate data from a random policy that at each step chooses a uniform random proportion of currency to convert with probability 0.2, and converts none of the currency with probability 0.8.
HIV Treatment The environment is based on the implementation in [26] of the physical model described by [20]. The patient state is a 6-dimensional continuous vector representing the concentrations of different types of cells and viruses in the patients blood. There are two actions, corresponding to giving the patient Reverse Transcriptase Inhibitors (RTI) or Protease Inhibitors (PI) (or both). Unlike [20] which considered discrete actions, we modify the domain to allow for continuous actions representing variable quantities of each drug. Following previous work [33], we assume that the
Table 8: Expected return received by a randomly initialised policy and an expert policy trained online using SAC. The expert SAC policy is trained online for 2e6 updates for the Stochastic MuJoCo domains, and 5e6 updates for HIVTreatment. In Tables 1, 2, and 3 the values reported are normalised between 0 and 100 using the procedure from [21] and the values in this table.
Random Policy	Expert Policy
Hopper (Moderate-Noise)	20.1	3049.4
Walker2D (Moderate-Noise)	-1.9	4479.2
Hopper (High-Noise)	11.2	2035.1
Walker2D (High-Noise)	-5.3	4101.3
HIVTreatment	-48.7	5557.9
CurrencyExchange	0.0	135.0
treatment at each time step is applied for 20 days, and that there are 50 time steps. Therefore, each episode corresponds to a total period of 1000 days. Also following [33], we introduce stochasticity into the domain by adding noise to the efficacy of each drug ( $\epsilon_{1}$  and  $\epsilon_{2}$  in [20]). The noise added to the efficacy is normally distributed, with a standard deviation of  $15\%$  of the standard value.
For this domain, the dataset is constructed in the same manner as the Medium-Replay datasets from D4RL [21]. Specifically, a policy is trained with SAC [30] until it reaches  $\approx 40\%$  of the performance of an expert policy. The replay buffer of data collected until reaching this point is used as the offline dataset.
The reward function is that used by [26], except that we scale the reward by dividing it by 1e6. A low concentration of HIV viruses is rewarded, and a cost is associated with the magnitude of treatment applied, representing negative side effects. The average reward received by a random policy and an expert policy trained online using SAC [30] can be found in Table 8. In the paper, values are normalised between 0 and 100 using the method proposed by [21] and the values in Table 8. Thus, a score of 100 represents the average reward of an expert policy, and 0 represents the average reward of a random policy.
Stochastic MuJoCo These benchmarks are obtained by applying random perturbation forces to the original Hopper and Walker2D MuJoCo domains. We chose to focus on these two domains as there is a risk of the robot falling over before the end of the episode. The perturbation forces represent disturbances due to strong wind or other sources of uncertainty. The force is applied in the  $x$ -direction at each step, according to a random walk on the interval  $[-f_{\mathrm{MAX}}, f_{\mathrm{MAX}}]$ . At each step, the change in force relative to the previous step is sampled uniformly from  $\Delta f \sim Unif(-0.1 \cdot f_{\mathrm{MAX}}, 0.1 \cdot f_{\mathrm{MAX}})$ . This is motivated by the fact that the strength of wind is often modelled as a random walk [52].
We test using two levels of noise applied. For the Moderate-Noise datasets, the level of noise results in a moderate degradation in the performance of an online SAC agent, compared to the original deterministic domain. For the High-Noise datasets, the level of noise results in a significant degradation in the performance of an online SAC agent. The value of  $f_{\mathrm{MAX}}$  is set to the following in each of the domains:
- Hopper Moderate-Noise:  $f_{\mathrm{MAX}} = 2.5$  Newtons.
- Hopper High-Noise:  $f_{\mathrm{MAX}} = 5$  Newtons.
- Walker2D Moderate-Noise:  $f_{\mathrm{MAX}} = 7$  Newtons.
- Walker2D High-Noise:  $f_{\mathrm{MAX}} = 12$  Newtons.
We construct three datasets: Medium, Medium-Replay, and Medium-Expert. These are generated in the same manner as the D4RL datasets [21]. The expert policy is a policy trained online using SAC until convergence. The medium policy is trained until it obtains  $\approx 40\%$  of the performance of the expert policy. The Medium-Replay dataset consists of the replay buffer collected while training the medium policy. The Medium dataset contains transitions collected from rolling out the medium policy. The Medium-Expert dataset contains a mixture of transitions generated by rolling out both the medium and the expert policy.
Following previous works [68, 47], we assume that the objective on all of the stochastic domains is to optimise the static  $\mathrm{CVaR}_{0.1}$  (i.e. the average total reward in the worst  $10\%$  of runs).
E.3 Evaluation Procedure
For D4RL MuJoCo, we evaluate the policy for 10 episodes at each of the last 10 iterations of training, for each of 5 seeds. The average normalised total reward received throughout this evaluation is reported in the D4RL results table (Table 1).
For Stochastic MuJoCo, we evaluate the policy for 20 episodes at each of the last 10 iterations of training.  $\mathrm{CVaR}_{0.1}$  is computed by averaging the total reward in the worst two episodes at each iteration. The value of  $\mathrm{CVaR}_{0.1}$  is then averaged across the last 10 iterations, and across each of the 5 seeds. These are the results reported for  $\mathrm{CVaR}_{0.1}$  in Table 3. For HIVTreatment and CurrencyExchange, we evaluate the policy for 200 episodes at each of the last 10 iterations of training, for each of 5 seeds. The average total reward for the worst 20 episodes at each evaluation is used to compute  $\mathrm{CVaR}_{0.1}$ .
E.4Baselines
We compare 1R2R against offline RL algorithms designed for risk-sensitivity (ORAAC [68] and CODAC [47]) in addition to performant risk-neutral model-based (RAMBO [58], COMBO [75], and MOPO [76]) and model-free (IQL [38], TD3+BC [22], CQL [40], and ATAC [11]) algorithms.
For the baselines, we use the following implementations:
- O-RAAC: github.com/nuria95/O-RAAC
- CODAC: github.com/JasonMa2016/CODAC
- RAMBO: github.com/marc-rigter/rambo
- IQL: github.com/ikostrikov/implicit_q_learning
- TD3+BC: github.com/sfujim/TD3_BC
CQL and COMBO: github.com/takuseno/d3rlpy
- ATAC: https://github.com/microsoft/lightATAC
- MOPO: https://github.com/tianheyu927/mopo
The results for D4RL MuJoCo-v2 for RAMBO, IQL, ATAC, COMBO, MOPO and TD3+BC are taken from the original papers. Because the original CQL paper used the -v0 domains, we report the results for CQL on the -v2 domains from [69]. The IQL papers does not report results for the random datasets, so we generate the results for IQL in the random datasets by running the official implementation algorithm ourselves. The results for CODAC in the original paper use D4RL MuJoCo-v0, and O-RAAC does not report results on the D4RL benchmarks. Therefore, we obtain the results for D4RL MuJoCo-v2 for CODAC and O-RAAC by running the algorithms ourselves.
For the stochastic domains (Currency Exchange, HIV Treatment, and MuJoCo Stochastic) all of the results were obtained by running the algorithms ourselves.
E.5 Baseline Hyperparameter Tuning
We tune each of the algorithms online to achieve the best performance for the desired objective in each dataset. The desired optimisation objectives in each domain are: expected value in D4RL MuJoCo and static  $\mathrm{CVaR}_{0.1}$  in all other domains. O-RAAC and CODAC can be configured to optimise a static risk measure, or expected value. Therefore, we set O-RAAC and CODAC to optimise the appropriate objective for each dataset.
For each of the baseline algorithms, we tune the following parameters based on the best online performance (averaged over 5 seeds) for the target objective. Where a hyperparameter value is not specified, it was set to the default value in the corresponding original paper.
- CODAC: The risk measure to optimise is set to the desired objective (Expected value/CVaR<sub>0.1</sub>). For Stochastic MuJoCo and HIVTreatment, we tune the parameters in the same manner as the original paper [47], choosing  $(\omega, \zeta) \in \{0.1, 1, 10\} \times \{-1, 10\}$ . For Stochastic MuJoCo and HIVTreatment we use the default critic learning rate of  $\eta_{\text{critic}} = 3e - 5$ . To generate the results for CODAC on D4RL MuJoCo-v2, we use the same parameters as reported in the paper for D4RL MuJoCo-v0.
- O-RAAC: The risk measure to optimise is set to the desired objective (Expected value/CVaR<sub>0.1</sub>). Following the original paper [68] we tune the imitation learning weighting:  $\lambda \in \{0.25, 0.5\}$ .
- RAMBO: We tuned the rollout length and adversarial weighting from:  $(k, \lambda) \in \{(1, 3e - 4), (2, 3e - 4), (5, 3e - 4), (5, 0)\}$ . This is the same procedure as in the original paper [58], except that we additionally included a rollout length of 1 as we found this was necessary for RAMBO to train stably on some of the stochastic domains.
- IQL: For Stochastic MuJoCo, we used the same parameters as for D4RL MuJoCo in the original paper [38]. For HIVTreatment and Currency Exchange, we chose the best performance between the parameters for D4RL MuJoCo and D4RL AntMaze from the original paper.
- CQL: Following the original paper [40], we tuned the Lagrange parameter,  $\tau \in \{5,10\}$ . All other parameters are set to their default values.
- TD3+BC: Following the original paper, we set  $\alpha$  to 2.5 [22].
- MOPO: The rollout length and conservative weighting are each tuned within  $\{1,5\}$  following the original paper [76].
COMBO: The conservative weighting was tuned within  $\{0.5,1,5\}$  following the original paper [75].
- ATAC:  $\beta$  was tuned within  $\{0,4^{-4},4^{-2},1,4^{2},4^{4}\}$ .