Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
Authors: Cong Lu
Philip J. Ball
Tim G. J. Rudner
Jack Parker-Holder
Michael A Osborne
Yee Whye Teh
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning has shown great promise in leveraging large pre-collected datasets for policy learning, allowing agents to forgo often-expensive online data collection. However, offline reinforcement learning from visual observations with continuous action spaces remains under-explored, with a limited understanding of the key challenges in this complex domain. In this paper, we establish simple baselines for continuous control in the visual domain and introduce a suite of benchmarking tasks for offline reinforcement learning from visual observations designed to better represent the data distributions present in real-world offline RL problems and guided by a set of desiderata for offline RL from visual observations, including robustness to visual distractions and visually identifiable changes in dynamics. Using this suite of benchmarking tasks, we show that simple modifications to two popular vision-based online reinforcement learning algorithms, DreamerV2 and DrQ-v2, suffice to outperform existing offline RL methods and establish competitive baselines for continuous control in the visual domain. We rigorously evaluate these algorithms and perform an empirical evaluation of the differences between state-of-the-art model-based and model-free offline RL methods for continuous control from visual observations. All code and data used in this evaluation are open-sourced to facilitate progress in this domain.

Published in Transactions on Machine Learning Research (07/2023)
Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
Cong Lu* University of Oxford
cong.lu@stats..z..ccm
Philip J. Ball* University of Oxford
ball@robots.or.cco uk
Tim G. J. Rudner University of Oxford
tim.rudner@cs.or.cco uk
Jack Parker-Be Holder University of Oxford
jackph@robots.oz.com ww..:ottsssrr........
Michael A. Osborne University of Onford
mosh@robots.oz.acc.k
Yee Whye Teh University of Orford
y. y.w. w.w.thhssttts.o..co
Reviewed on OpenReview: https: openreview. net/ formf? 1QqIfGZOW.
Abstract
Offline reinforcement learning has shown great promise in leveraging large pre-collected datasets for policy learning, allowing agents forgo often-expensive online data collection. However, offline reinforcement learning from visual observations with continuous action spaces remains under-explored, with limited understanding of the key challenges in this complex domain. In this paper, we establish simple baselines for continuous control in the visual domain and introduce a suite of benchmarking tasks for offline reinforcement learning from visual obscrvations designed to better represent the data distributions present real-world oflline RL problems and guided by a set of desiderata for oflline RL from visual observations, including robustness to visual distractions and visually identifiable changes in dynamics. Using this suite of benchmarking tasks, show that simple modifications to two popular vision-based online reinforeement learning algorithms, DreamerV2 and DrQ- v2, suffice to outperform existing offline RL methods and establish competitive baselines for continuous control in the visual domain. We rigorously evaluate these algorithms and perform an empirical evaluation of the differences between state-of-the-ars model-based and model-free offline RL methods for continuous control from visual observations. All code and data used this evaluation are open-sourcod to facilitate progress this domain.
Open- code and data for the V-D4RL benchmarking suite are available at: httpe://ggtttbb.ccc///oou///////
"Equal contribution. Correspondence cong. www.tats......... ax.ac.....ccc...... and ball@robots.co

---

Published in Transactions on Machine Learning Research (07/202) 2023)
K
Figure 1: V-D4RL is a benchmarking suite for offline reinforcemenl learning from visual observalions based on the DMCONTROL SUITE (Tassa et al., 2020), which includes a comprehensive set of D4RL-style datasets and modalities unique to learning from visual observations.
1 Introduction
The reinforcement learning (RL, Sutton & Barto (1992)) paradigm is conventionally characterized learning from online interaction with an enviromment. However, in many real-world settings, such robotics clinical pocision-making online interactions be expensive or impossible to perform due to physical safety constraints. Offline (or batch) reinforcement learning (Erust et al., 2005; Levine et al., 2020) aims to address this issue by leveraging pre-collected datasets to train and deploy autonomous agents without requiring online interaction with an environment.
While offline reinforcement learning algorithms, both model-based (Yu et al., 2020b; Kidambi al., 2020; Argenson & Dulac-Aroold, 2021) and model-free (Kumar al., 2020; Kostrikov al., 2021; Fujimoto & Gu, 2021; Wu al., 2019; Fujimoto al., 2019; Kumar et al., 2019), have mastered challenging continuous control tasks, most prior works have relied on access to proprioceptive states focusing on standardized benchmarking suites (Fu et al., 2021). In contrast, studies of offline reinforcement learning from visual observations (Rafailov et al., 2021; Shah et al., 2021; Chen et al., 2021; Florence et al., 2021) have often focused on disparate tasks with bespoke datasets due to the lack of a well-designed benchmarking suite with carefully evaluated baselines. With this work, we aim to address both of these needs.
Training agents from visual observations offline provides an opportunity make reinforcement learning more widely applicable to real-world scttings, where have access to vast quantities of visual observations of desirable behaviors. For example, in autonomous driving (Kendall et al., 2018), large quantities of visual offline data already exist but have not been fully utilized (Maddern et al., 2016; Yu et al., 2020a). Similarly, in robotics, data collection is expensive due to costs associated with set-up and human supervision. Effective, transferable offline reinforcement learning could allow us to reuse datasets gathered previously for different tasks or settings for unseen new problems (Chebotar et al., 2021). Unlocking this potential would represent significant progress towards lcarning gencral-purpose agents for realistic problems.
To enable the development of effective, robust, and adaptive algorithms for offline RL from visual observa- tions, present suite of carefully designed datasets and benchmarking tasks for this burgeoning domain. We use these tasks to establish simple performance basclines, to study how the composition of vision-basod oflline datasets affects the performance of different types of RL algorithms, and to evaluate the extent to which algorithms for offline RL from visual observations satisfy a set of desiderata, including robustness to visual distractions, generalization across environment dynamics, and improved performance at seale. Our evaluation identifies clear failure modes of the baseline methods and highlights opportunities and open prob- lems for future work which can be tackled with our benchmark.
2

---

Published in Transactions on Machine Learning Research (07/2023)
Recent progress in offline reinforcement learning from proprioceptive observations has been driven by well- designed and easy-to-use evaluation testbeds and baselines. We hope that V-D4RI, and the analysis in this paper will help facilitate the development of robust RL agents that leverage large, diverse, and often imperfect offline datascts of visual observations across tasks and deployment scttings.
The core contributions of this paper are as follows:
We present a benchmark for offline RL from visual observations of DMCONTROL SUITE (DMC) tasks (Tassa ct al., 2020). This benchmark, Vision Datascts for Deep Data-Driven RL (V-D4RL), follows the design principles of the popular DÀRL benchmark (Fu et al., 2021), and to our knowledge is the first publicly available benchmark for continuous control which features wide variety of behavioral policies.
2. We identify three key desiderata for realistic offline RL from visual observations: robustness to distractions (Stone et al., 2021), generalization across dynamics (Zhang et al., 2021), and improved performance for offline reinforcement learning at scale. We present suite of evaluation protocols designed to test whether offline RL algorithms satisfy these desiderata.
3. We establish model-based and model-free baselines for offline RL from visual observations. We do so by modifying two popular online RL algorithms, Dreamer (Hafner et al., 2020b) and DrQ-v2 (Yarats al., 2021a), which showcase the relative strengths of model-based and model-free algorithms for offline RL from visual observations. We use these algorithms to provide simple baselines for the aforementioned desiderata to serve as measure of progress for future advances in this domain.
2 Preliminaries
We model the enviromment as Markov Decision Process (MDP), defined as tuple M = (S, 7); where S and A denote the state and action spaces respectively, ((s'|s, a) the transition dynamics, R(s,al a) the reward function, the initial state distribution, and y E (0, 1) the discount factor. The standard goal in online reinforcement learning is to optimizc a policy π T (s) that maximizes the expected discounted return P, 1E: R(s,, at)] through interactions with the environment.
In offiine reinforcement learning, the policy is not deployed in the environment until test time. Instead, the algorithm has access to fixed dataset Denv {s;, ag T2 (1+1)) collected by one more behavioral policies wh. Following Yu et al. (2020b), we refer to the distribution from which Danv was sampled as the behavioral distribution.
We first describe recent advancements in offline RL and RL from visual observations through the lens of model-based and model-free hoth
2.1 Offline Reinforcement Learning Paradigms
Model-based. A central problem in offline reinforcement. learning is over-estimation of the value func- tion (Sutton & Barto, 1992) due to incomplete data (Kumar et al., 2019). Model-based methods in offline RL provide a natural solution to this problem by penalizing the reward from model rollouts by a suitable measure of uncertainty. Yu et al. (2020b) provide a theoretical justification for this approach by constructing pessimistic MDP (P-MDP) and lower-bounding the expected true return, T)a())) using the error between the estimated and true model dynamics. Since this quantity is usually not available without access to the true enviromment dynamics, algorithms such as MOPO and MOReL (Yu et al., 2020b; Kidambi et al., 2020) pe- nalize reward with a surrogate measure of uncertainty. These algorithms train an ensemble of K probabilistic dynamics models (Nix & Weigend, 1994) and define a heuristic based on the ensemble predictions. Recent work (Lu et al., 2022) has shown that a better approach to approximating the true dynamics error to use the standard deviation of the ensemble's mixture distribution instead, as proposed by Lakshminarayanan et al. (2017).
3

---

Published in Transactions on Machine Learning Research (07/2023)
Model-free. In the model-free paradigm, we lose the natural measure of uncertainty provided by the model. In lieu of this, algorithms such as CQL (Kumar et al., 2020) attempt to avoid catastrophic over- estimation by penalizing actions outside the support of the offiine dataset with a wide sampling distribution over the action bounds. Recently, Fujimoto & Gu (2021) have shown that a minimal approach to offiine reinforcement learning works proprioceptive settings, where oflline policy learning with TD3 (Fujimoto et al., 2018) can be stabilized by augmenting the loss with a behavioral cloning term.
2.2 Reinforcement Learning from Visual Observations
Recent advances in reinforcement learning from visual observations have been driven by use of data aug- mentation, contrastive learning and learning recurrent models of the environment. We describe the current dominant paradigms in model-based and model-free methods below.
Model-based (Dreamer V2, Hafner et al. (2020b)). Dreamer learns model of the environment using a Recurrent State Space Model (RSSM, Hafner et al. (2019; 2020a), and predicts ahead using compact model latent states. The particular instantiation used in DreamerV2 uses model states containing deterministic component h4, implemented as the recurrent state of Gated Recurrent Unit (GRU, Chung et al. (2014)), and a stochastic component with a categorical distribution. The actor and critic are trained from imagined trajectories of latent states, starting at encoded states of previously encountered sequences.
Model-free (DrQ-v2, Yarats et al. (2021b)). DrQ-v2 is off-policy algorithm for vision-based con- tinuous control, which uses data-angmentation (Laskin et al., 2020; Yarats et al., 2021c) of the state and next state observations. The base policy optimizer is DDPG (Lillicrap et al., 2016), and the algorithm uses a convolutional neural network (CNN) encoder to learn a low-dimensional feature representation.
3 Related Work
There has boen significant progress in offline RL, accompanicd and facilitated by the creation of several widely used benchmarking suites. We list several here; to our knowledge, no contemporary datasets both publicly available and feature the same range of behaviors as D4RI, nor do they feature tasks related to distractions or changed dynamics.
Benchmarks for continuous control on states. D4RL (Fu et al., 2021) is the most prominent benchmark for continuous control with proprioceptive states. The large variety of data distributions has allowed for comprehensive benchmarking (Kumar et al., 2020; Kidambi et al., 2020; Yu et al., 2020b; 2021; Kostrikov et al., 2021) and understanding the strengths and weaknesses of offline algorithms. Our work aims to establish a similar benchmark for tasks with visual observations. RL Unplugged (Gulcehre et al., 2020) also provides public visual data on DMControl tasks but only provides mixed data which filtered on the harder locomotion suite. COMBO (Yu et al., 2021) also provides results on the DMC walker-walk environment however does not benchmark over full set of behavioral policies and the datasets are not publicly available at the time of writing.
Analysis on characteristics of offline datasets. Recent work (Florence et al., 2021) has sought to understand when offline RL algorithms outperform behavioral cloning in the proprioceptive setting. Ku- mar et al. (2021) recommend BC for many settings but showed theoretically that offline RL was preferable in settings combining expert and suboptimal data. This finding is corroborated by our analysis (sce Ta- bles 1 and 4).
basedn discrete control datasets. Whilst there has been a lack of suitable bench- marks for vision-based offine continuous control, vision-based datasets for discrete control have been created for Atari (Agarwal et al., 2020). However, at 50M samples per environment, this benchmarking suite can be prohibitive in terms of its computational hardware requirements (see htttps://gitee com/ htwgggglleeesserchbareeeee benchmark represents more achievable task for practitioness.
r1/issues/10). We believe that V-D4RL'S 100,000-sample
4

---

Published in Transactions on Machine Learning Research (07/2023)
Offline vision-based robotics. Learning control offline from visual observations is an active area of robotics research closely-related to the problems considered in V-DARI. Interesting pre-collected datasets exist in this domain but with a primary focus on final performance rather than comprehensive benchmarking on different data modalities; we list some of these here. In QT-Opt (Kalashnikov et al., 2018), data is initially gathered with scripted exploration policy and then re-collected with progressively more successful fine-Luned robots. Chebotar et al. (2021) follow a similar setup, and explores representation learning for controllers. Mandlekar et al. (2021); Cabi et al. (2020) consider learning from combined sets of human demonstratinns, data from scripted policies and random data.
4 Baselines for Offline Reinforcement Learning from Visual Observations
In this section, we begin by motivating our creation of new benchmark (V-D4RL), introduce our new simple baselines combining recent advances in offline RL and vision-based online RL, and present comparative evaluation of current methods on V-D4RL. Comprehensive and rigorous benchmarks are crucial to progress in nascent. fields. To our knowledge, the only prior work that trains vision-based offline RL, agents continuous control tasks is LOMPO (Rafailov et al., 2021). We analyze their datasets in Section 4.3.1 and find they do not conform to standard D4RL convention.
4.1 Adopting D4RL Design Principles
In this section, we outline how lo generate D4RI-like vision-based datasets for V-DARL. To generate oflline datasets of visual observations, we consider the following three DMCONTROL SUITE (DMC) environments (these environments are easy, medium and hard respectively (Yarats et al., 2021a)):
walker-walk: planar walker is rewarded for being upright and staying close to target velocity.
cheetah-run: planar biped agent is rewarded linearly proportional its forward velocity.
walk: a 21-jointed humanoid is rewarded for staying close to a target velocity, Due to the huge range of motion styles possible, this environment is extremely challenging with many local minima and is included as stretch goal.
From these environments, we follow a D4RI-style procedure in considering five different behavioral policies for gathering the data. As DARL, the base policy used to gather the data is Soft Actor Critic (SAC, Haarnoja et al. (2018)) on the propriocoptive states. We consider the following five scttings:
random: Uniform samples from the action space. medium-replay (mixed): The initial segment of the replay buffer until the SAC agent reaches medium-level performance.
medium: Rollouts of fixed medium-performmace policy.
expert: Rollouts of fixed expert-level policy.
medium-expert (medexp): Concalenation of medium and expert datasets above.
We provide precise specifications for medium, mixed and expert in Section 4.1.1 and discuss the choice of offline behavioral policy in Section 4.1.2. By default, each dataset consists of 100,000 lotal transitions (often 10x less than in D4RI.) in order to respect the memory demands of vision-based tasks. The cheetah and humanoid medium-replay datasets consist of 200,000 and 600,000 transitions respectively due to the increased number of samples required to train policies on these environments. Full statistics of each dataset are given in Appendix A. In Section 5, we further extend the original V-D4RI. datasets to study problem settings with multiple tasks and visual distractions as illustrated in Figure 1.
4.1.1 Data Generation Policy
To create the offline medium and expert datasets, we first train SAC (Haarnoja et al., 2018) policies on the proprioceptive states until convergence, taking checkpoints every 10,000 frames of interaction. We use frame skip of 2, the default in other state-of state-of-the-art vision RL algorithms (Hafner et al., 2020b; Yarats et al., 2021a). Medium policies are defined as the first saved agent during training that is able to consistently
5

---

Published in Transactions on Machine Learning Research (07/202) /023)
walker-walk random
walker-walk mixed
walker-walk medium
walker-walk medexp
walker-walk expert
1009
1000
600
600
759
730
200
400
569
500
200
200
cheetah-run random
cheetah-run mixed
cheetah-run medium
cheetah-run medexp
cheetah-run expert
600
800
Eval Retum
3000
600
200
400
1.00
200
200
200
humanold-walk random
humanold-waik mixed
humanold-walk medium
aumannodd w-wkkk medexp
humanold-war expert
130
159
100
103
99
200
400
800
1000
200
400
500
500
1000
%
200
400
600
1000
%
200
400
600
800
1000
200
430
620
800
1000
Offline Epochs
Offline DV2
DrQ+BC
CQL
BC
LOMPO
Figure 2: Rigorous comparison on the V-D4RL, benchmark, each setting is averaged over six random seeds with error bar showing one standard deviation. Total gradient steps are normalized under epochs, and we plot the un-normalized evaluated return. We note that the model-free and BC baselines are far more stable than the model-based.
achieve above 500 reward in the environment. Expert policies are defined as those that have converged in the limit. For DMControl tasks, this typically mcans near 1,000 reward on the environment they were trained on. We confirm these thresholds reasonable, we observe a noticeable gap between the behavior of the medium and expert-level policies.
In order to generate the offline visual observations, we deploy the proprioceptive agents in the environment, and save the visual observations rendered from the simulator instead of the proprioceptive state. This provides us with the flexibility to generate observations of any size without having to retrain for that resolution (e.g., 84 x 84 or 64 64). As was done D4RL, we generate data using stochastic actor, which involves sampling actions from the Gaussian conditional distribution of the SAC policy, featuring a parameterized variance head that determines the amount of action stochasticity at each state.
For the mixed datasets, simply store the replay buffer of the medium agent when it finishes training, and convert the proprioceptive observations into visual observations.
4.1.2 Choice of Offline Behavioral Policy
We choose proprioceptive SAC our behavioral policy %, mirroring Fu et al. (2021); Rafailov al. (2021). We found that using online DrQ-v2 as the offline behavioral policy made the tasks significantly easier to learn for all agents. This suggests that the proprioceptive agent may learn behavior modes that are less biased towards being easy under vision-based methods; for instance, DrQ-v2 may be biased towards behavior modes that induce fewer visual occlusions compared to proprioceptive SAC.
4.2 Baselines
We show that for the two state-of-the-eeee the-attertt online rision-based RL algorithms described in Section 2.2, simple adjustments from the proprioceptive literature suffice to transfer them to the offline setting. In Section 4.3, we demonstrate that these baselines provide new frontier on our benchmark and on prior datasets. Additional details and hyperparameters for our algorithms are given in Appendix B.
Model-based. For D2eamerV2, we follow Sekar al. (2020) in constructing a bootstrap ensemble for the dynamics model, which allows to naturally define penalty for the reward based on the dynamics

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table 1: Final performance on V-D4RL averaged over six random seeds, with one standard deviation error. Evaluated return mapped from [0, 1000] to [0. 100). Our model-based method does best on the diverse low-reward datasets, model-free on the diverse high-reward datasets, and behavioral cloning on the narrow expert data. Furthermore, provide the mean return of the dataset for reference; full statistics are included in Appendix A.
Environment
Offline DV2
DrQ+BC
CQL
BC
LOMPO
Data Mean
random
28.7 113.0
5.5 10.9
14.4 112.4
2.0 10.2
21.9 18.1
4.2
mixed
56.5 +18.1
28.7 16.9
11.4 112.4
16.5 +4.3
34.7 #19.7
14.5
walker-walk
medium
34.1 +19.7
46.8 +2.3
14.8 +16.1
40.9 #3.1
43.4 #11.1
44.0
medexp
43.9 #34.4
86.4 +5.6
56.4 #38.4
47.7 #3.9
39.2 #19.5
70.4
expert
4.8 10.6
68.4 17.5
89.6 16.0
91.5 13.9
5.3 17.7
97.0
random
31.7 +2.7
5.8 +0.6
5.9 +8.4
0.0 #0.0
11.4 #5.1
0.7
mixed
61.6 11.0
44.8 13.6
10.7 112.8
25.0 13.6
36.3 113.6
19.1
cheetah-run
medium
17.2 #3.5
53.0 #3.0
40.9 #5.1
51.6 #1.4
16.4 #8.3
52.4
medexp
10.4 43.5
50.6 +8.2
20.9 45.5
57.5 16.3
11.9 11.
70.7
expert
10.9 +3.2
34.5 +8.3
61.5 44.3
67.4
+6.8
14.0 #3.8
89.1
random
0.1 #0.0
0.1 =0.0
0.2 #0.1
0.1
#0.0
0.1 #0.0
0.1
mixed
0.2 #0.1
15.9 13.8
0.1 #0.0
18.8
14.2
0.2 #0.0
27.6
humanoid-wal walk
medium
0.2 10.1
6.2 -244
0.1 10.0
13.5
14.1
0.1 10.0
57.3
medexp
0.1 +0.0
7.0 +2.3
0.1 #0.0
17.2
#4.7
0.2 #0.0
71.6
expert
0.2 #0.1
2.7 +0.9
1.6 #0.5
6.1
#3.7
0.1 #0.0
85.8
uncertainty as Yu al. (2020b). We adopt an analogous approach that studied in Lu et al. (2022);
Sokar et al. (2020) and use the moan-disagreement of the ensemble. Thus, the reward at cach step becomes:
= ( Σ a)) - (x",))))
(1)
where À is penalty weight and n² (s(ss a) Σk= (s, a) is the mean over the dynamics ensemble. Instead of interleaving model-training steps and policy optimization steps, we simply perform phase of each. We refer to this algorithm as Offline DV2.
Model-free. For DrQ-v2, we note that the base policy optimizer shares similarities with TD3 (Fujimoto et al., 2018), which has recently been applied effectively in offline settings from proprioceptive states by simply adding a regularizing behavioral-cloning term to the policy loss, resulting in the algorithm TD3+BC (Fuji- moto & Gu, 2021). Concretely, the policy objective becomes:
(rgmaxB(x.))D..... [AQ(s, =|
(2)
where À is normalization term, Q is the learned value function and is the learned policy. We apply the same regularization to DrQ-v2, and call this algorithm: DrQ+BC.
Prior work. Since DrQ-v2 is actor-critic algorithm, may also use it to readily implement the CQL (Kumar et al., 2020) algorithm by adding the CQL regularizers to the Q-function update. We ad- ditionally compare against LOMPO (Rafailov et al., 2021), and behavioral cloning (BC, Bain & Sammut (1995); Bratko et. al. (1995)), where we apply supervised learning to mimic the behavioral policy. Ofline DV2 is closely related to LOMPO as both use an RSSM (Hafner et al., 2019; 2020a) as the fundamental model, however, Offline DV2 is based on the newer discrete RSSM with an uncortainty penalty more closoly resem- bling the ensemble penalties in supervised learning (Lakshminarayanan al., 2017) and uses KL balancing during training (Hafner et al., 2020b).

---

Published in Transactions on Machine Learning Research (07/2023)
Table 2: We confirm that our simple baselines outperform LOMPO on the original walker-walk data provided by Rafailov al. (2021). We report final performance mapped from [0, 1000] to [0, 100) averaged over six random seeds." We show our baselines are more performant than LOMPO on their benchmark. CQL were numbers taken from Rafailov et al. (2021).
LOMPO Dataset
LOMPO
Offline DV2
DrQ+BC
CQL
medium-replay
61.3 +9.1
76.3 +3.1
31.1 +3.7
14.7
medium-expert
69.0 124.1
72.3 120.1
73.3 13.5
45.1
expert
52.4 #35.7
59.4 +26.6
90.8 +2.2
40.3
4.3 Comparative Evaluation
We now evaluate the five algorithms described in Section 4.2 on a total of fifteen datasets. To provide a fair evaluation, we provide full training curves for each algorithm Figure 2 and summarize final performance with error in Table 1. Since no online data collection is required, we measure progress through training via "offline epochs" metric which define in Appendix C.
Table 1 shows clear trend: Offline DV2 is the strongest on the random and mixed datasets, consisting of lower-quality but diverse data, DrQ+BC is the best on datasets with higher-quality but still widely- distributed data and pure BC outperforms on the high-quality narrowly-distributed expert data. We see from Table and Figure 2 that DBQ+BC is extremely stable across seeds and training steps and has the highest overall performance. CQL is also a strong baseline, especially on expert data, but requires siguificant hyperparameter tuning per dataset, often has high variance across seeds, and is also far slower than DrQ+BC to train. Finally, no algorithm achieves strong performance on the challenging humanoid datasets, mirroring the online RL challenges (Hafner et al., 2020b; Yarats et al., 2021a), with only the supervised BC and, by exlension, DrQ+BC showing marginal positive returns.
Perhaps surprisingly, Offline DV2 learns mid-level policies from random data on DMC enviromments. Fur- thermore, the random data are more challenging than their DARL equivalents because there no carly termination, and thus mostly consists of uninformative failed states; this shows the strength of model-based methods in extracting signal from large quantities of suboptimal data. On the other hand, Offline DV2 is considerably weaker the expert datasets that have narrow data distributions. For these environments, we find the uncertainty penalty is uninformative, discussed in Appendix D.4.1.
Taking all these findings into considcration leads us to our first open problem, which we believe continued research using our benchmark can help to answer:
Can a single algorithm outperform both the model-free and model-based basclines, and produce strong performance across all offline datasets?
Challenge 1:
4.3.1 Comparison the LOMPO Benchmark
For a fair comparison to LOMPO, we also benchmark on the data used in Rafailov et al. (2021) on the DMC Walker- Walk task. In the LOMPO benchmark, the datasets are limited to three types: (medium-replay; medium-expert and expert]. We provide final scores in Table 2. While LOMPO struggles on V-D4RL, it performs reasonably on its own benchmark. However, LOMPO is still outperformed by Offline DV2 on all datasets.
datasets, whereas DrQ+BC is the best
We may explain the relative strength of LOMPO this benchmark by noting that the medium-expert dataset used by Rafailov et al. (2021) is described as consisting of the second half of the replay buffer after the agent reaches medium-level performance, thus containing far more diverse data than a bimodal D4RL- style concatenation of two datasots. Furthermore, the export data is far more widely distributed than that of a standard SAC expert, as we coufirm in the statistics Table of Appendix
It. unclear how the original scores Rafailov al. (2021) normalized.

---

Published in Transactions on Machine Learning Research (07/2022 2023)
Original Env Return
Shifted Train Return
Shifted Test Return
Offline DV2
100
100
100
Level Distraction
75
75
Low
50
50
Normalized lized Return
25
25
25
0
Moderate
25
50
75
100
0
25
50
75
100
25
50
75
100
100
100
100
DrQ DrQ+BBC
75
75
High
50
50
25
25
25
0
0
25
50
75
100
25
50
75
100
25
50
75
100
% Data Shifted
Figure 3: Both DrQ+BC and Offline DV2 readily support training datasets with different distractions (mixture of original and shifted train). Offline DV2 additionally shows the ability to generalize to unseen distractions (shifted test). whereas DC is more brittle. Return normalized against unshifted perfor- mance without distraction from Table and averaged over six random seeds. Unnormalized returns are provided in Tables 11 and 12 in Appendix D.1.
5 Desiderata for Offline RL from Visual Observations
A key ingredient in recent advances in deep learning is the use of large and diverse datasets, which are particularly prevalent in vision, to train models. Such datasets enable the learning of general features that can be successfully transferred to downstream tasks, even when the original task bears little immediate similarity with the transferred task (Xie & Richmond, 2019; Peters et al., 2019). This clear rationale for adopting visual observations in offline RL; by leveraging large quantities of diverse high-dimensional inputs, we should be able learn generalizable features and agents for control. However, combining rich visual datasets with RL presents its unique challenges. In this section, present important desiderata that highlight this, and conclude each with an open problem that requires further rescarch.
5.1 Desideratum: Robustness to Visual Distractions
One desideratum for offiine RL from visual observations is the ability to learn policy from data collected under multiple different settings. For example, quadrupedal robots deployed at different times of day (e.g.. morning and night) will likely gather data having significantly different visual properties, such as brightness and range of visibility. Although the robot may produce similar behaviors in both deployments, superficial differences in visual outputs present a host of opportunities for the agent to learn spurious correlations that prevent generalization (Song et al., 2020; Railcanu & Fergus, 2021).
A key opportunity that arises is the potential to disentongle sources of distractions through training on multiple settings, facilitating the learning of general features. By contrast, proprioceptive observations do not generally have distractions, as they are typically low-dimensional and engineered for the task of interest (Lesort et al., 2018). This also limits their ability to transfer, as it is unclear how to incorporate features learned under one set of agent geometries to another.
To test a simplified version of this challenge, we train our baseline agents using newly created datasets featuring varying visual augmentations from the Distracting Control Suite (Stone al., 2021). This suite provides three levels of distractions (i.c., low, moderate, high), and each distraction represents shift in the data distribution. We subsequently refer to the level of distraction as the "shift severity" (Schneider et al., 2020). The oflline dalasets are then constructed as mixtures of samples from the original environment
As CQL quite sensitive hyperparameters per environment, following sections we more robust. Offline DV2 and DrQ+BC algorithms.
9

---

Published in Transactions on Machine Learning Research (07/2023)
without distractions and samples from environment with fized distraction level. The learned policies are then evaluated on test environments featuring unseen distractions of the same shift severity.
To create these datasets, we note that the visual distractions from the Distracting Control Suite do not manifest themselves in the proprioceptive states, so we can naturally generate the shifted visual observations by simply rendering the distractions on top of the existing visual observations. Thus, we can simply use the same saved checkpoints as in Section 4.1.1 for the standard datasets.
We compare the baseline algorithms, Offiine DV2 and DrQ+BC datasets that they excel on in Section 4 and Section 5.3: walker-walk random with 100,000 datapoints and cheetah-run medium-oxpert. with mil- lion respectively. We visualize returns normalized by this unshifted performance in Figure 3 to show the generalization of each algorithm. We further present full labular results for Offline DV2 and DrQ+BC in Table and Table 12 respectively Appendix D.1. The highlighted base unshifted results in both tables are the same as in Table for Offline DV2 and Table for DrQ+BC.
Offlin DV2 able to accommodate datasets with mixed distractions and generalizes reasonably well to unseen test distractions, especially when trained with 'low" and 'moderate' levels of shift severity. Similarly, DrQ+BC is robust to multiple training distractions, with little to no degradation in performance on mixed datasets. However, in contrast, the policy learned is brittle with respect to unseen distractions, and performs significantly worse on the test environments.
Overall, both Offline DV2 and DrQ+BC represent strong baselines for mixed datasets. Interestingly, Offline DV2 demonstrates strong generalization to unseen distractions. This can be explained by generalization that occurs in the trained world-model, which uses self-supervised loss; discuss reasons behind this in Appendix D.5. This could be improved even further with recent reconstruction-fre variants of Dream- Vr (Okada & Taniguchi, 2021; Nguyen et al., 2021) which have shown robustness to distractions. On the other hand, we observe DrQ++BC generalizes poorly to unseen distractions, presenting direction for future work using our datascts to learn robust model-free agents. This directly lcads us to our next open problem:
Challenge 2: How improve the generalization performance of offiine model-free methods to unseen visual distractions?
5.2 Desideratum: Generalization Across Environment Dynamics
Another desideratum of offline RL from visual obscrvations is lcarning policies that can generalize across multiple dynamics. This challenge manifests in three clear ways. Firstly, we will likely collect data from multiple agents that each have different dynamics, and must therefore learn a policy that can perform well when deployed on any robot that gathered the data (i.c., train time robustness). Sccondly, we may be provided with asymmetric data, featuring scarce coverage of particular dynamics, and therefore require the
Table 3: Evaluation on the DMC-Multitask benchmark using random data for Offline DV2 and mederp data for DCQ+BC and BC. Normalized performance from [o, 1000] to [0, 100) over random seeds. Our algorithms learn multitask policies from visual observations, with slight generalization gap for extrapolation tasks. Different dataset types used for each algorithm to reflect realistic use cases.
Eval. Return
Algorithm
Environment
Train Tasks
Test Interp.
Test Extrap.
walker
90.8
91.4
65.1
DrQ+BC
chectah walker
71.6
65.1
43.2
medexp
61.2
61.4
47.2
BC
cheetah
69.7
61.3
39.6
walker
24.4
25.3
24.9
Offlne DV2
random
cheetah
31.6
31.1
31.1
10

---

Published in Transactions on Machine Learning Research (07/2023)
Table 4: The reinforcement learning algorithms readily scale to higher dataset sizes, compared to supervised behavioral cloning, with a particular benefit the medezp and expert datasets for DrQ+BC and the random and medium datasets for Offiine DV2. Results are averaged over six random seeds, with one standard deviation given error. The evaluated return is mapped from [0, 1000] to 100], and the fixed-size mixed dataset is excluded.
Offline DV2
DrQ+BC
BC
Environment
100K
500K
100K
500K
100K
500K
random
28.7 +13.0
49.9 #1.6
5.5 +0.9
3,5 10.6
2.0 40.2
2.1 10.3
medium
34.1 119.7
61.3 =10.9
46.8 +2.3
51.0 #1.1
40.9 #3.1
40.9 #3.0
walker
medexp
43.9 #34.4
38.9 =28.1
86.4 +5.6
94.1 +2.0
47.7 #3.9
48.8 45.3
expert
4.8 +0.6
7.1 +5.3
68.4 +7.5
94.2 +2.3
91.5 #3.9
95.1 +2.5
random
31.7 +2.7
40.8 #4.2
5.8 +0.6
10.6 #0.7
0.0 +0.0
0.0 #0.0
medium
17.2 13.5
39.2 114.
53.0 13.0
57.3 11.2
51.6 11.4
52.9 11.3
cheetah
medexp
10.4 #3.5
9.7 +5.0
50.6 +8.2
79.1 +5.6
57.5 16.
69.6 110.6
expert
10.9 +3.2
11.3 #4.7
34.5 +8.3
75.3 #7.5
67.4 #6.8
87.8 +1.9
Average Overall
22.7 110.1
32.3 19.3
43.9 14.6
58.1 12.6
44.8 13.2
49.7 13.1
Percentage Gain
+42.1%
+-32.5%
+ 10.8%
ability loverage data from more abundant sources (i.c., transferability). Thirdly, we may be presented with unseen dynamics at deployment time, and must therefore learn a policy that is robust these changes (i.e., test time robustness).
A key opportunity that arises in visual observations is the improved richness of the underlying dataset com- pared to proprioceptive data. For instance, some dynamics changes may be visually obvious (e.g., changed limb sizes, broken actuators), whereas in the proprioceptive setting, such information may not be available. Without this information, we must turn to meta-RL (Rakelly et al., 2019; Zintgraf et al., 2021) or HiP- MDP (Killian et al., 2017: Zhang al., 2021) approaches that try to infer the missing information from gathered trajectories, adding complexity to the RL process. In contrast, this information can be contained explicitly in visual observations and should allow adaption to range of downstream tasks without complex inference methods.
To test this hypothesis, we consider two settings from the MTEπV benchmark (Sodhani et al., 2021) which adapts DMC: choctah-run with modified torso length and walker-walk with modified leg length. We follow an analogous approach to Zhang et (2021) where we consider eight different settings {A - H} ordered in terms of limb length, and construct new offline datasets using (B, C, F, G) in equal proportions as our training data. The settings {A, H} are considered the extrapolation generalization environments and (D, E} interpolation generalization. Since these environments are different from the original, retrain SAC policies on the proprioceptive states from the modified tasks and define medium and expert policies in the same way as described in Section 4.1.1.
As before, to provide a comparison on realistic use cases of each algorithm, we evaluate Offline DV2 on random datasets of size 100,000 and DrQ+BC on medium-expert. datasets containing one million samples and show the results Table We see that DrQ+BC learns policics that are suitable for transfer across multiple tasks in both walker and cheetah, and maintains that performance on the interpolation test environments. For the extrapolation environments, performance, especially compared to medium policy, it is striking drop when compared to performance on in-distribution dynamics. This suggests there is a dynamics generalization gap that remains for model-free methods when extrapolating, and represents clear opportunities for further research.
average drop of around 30%. While this may represent adequate
Offline DV2 displays similar trends (results on medexp datasets are in Appendix D.2). On the random data, Offline DV2 learns similar quality policy to that on the base environment, but experiences no deterioration in performance the test environments in walker cheetah. Thus, demonstrate the sufficiency of
11

---

Published in Transactions on Machine Learning Research (07 (2722)
both Offine DV2 and DrQ+BC baselines in multitask offline RL without any modification; model-based approaches further admit opportunities for zero-shot generalization (Ball et al., 2021).
We now contrast our work to that of Zhang et al. (2021), where a multitask policy was trained using total of 3.2 million time steps of online data collection. Whilst is hard to compare offine and online results, our DrQ-BC algorithm uses less data, with million total time steps, and obtains similar extrapolation return on the walker environments. This supports a similar conclusion reached by Kurin et al. (2022) who show that our approach, simply minimizing the sum of the task losses, is drastically underestimated in the literature. As noted before, suffer comparatively larger drop in performance, lending further evidence that closing this generalization gap should be prioritized.
In conclusion, we believe there are many further avenues for future research using these benchmarks; an immediate open problem we have identified is as follows:
Challenge 3: How can we improve generalization to new dynamics that are not contained in the offline dataset?
5.3
Desideratum: Improved Performance with Scale
DrQ+BC
Offline DV2
800
400
Eval Retum
600
300
400
200
200
%
200
400
600
800
1000
0
200
400
600
800
1000
Offline Epochs
25%
50%
75%
100%
200%
300%
400%
Figure 4: Sensitivity analysis on datasct size for both Offline DV2 (walkor-walk random) and DrQ+BC (cheetah-run medexp). Both methods scale with more data, but receive diminishing returns past 2x the original size. Performance averaged over four random seeds.
Learning from large datasets presents huge opportunities for learning general agents for control. To make use of them, we need understand how our basclines scale with datasot size. We analyzo our basc choice of 100,000 observations for V-D4RL in Figure 4, where we vary the size of the walker- malker-walkk random dataset for Offline DV2 and the cheetah-run medexp dataset for DrQ+BC in the range of (0.25 4x) the size of the original dataset. We observe monotonic increase in the performance of both Offline DV2 and DrQ+BC with increasing dataset size but hit diminishing returns past 2 the original size. We note, perhaps surprisingly, that Offline DV2 can reach x500 total return from random data that average 10x less.
For the walker and cheetah datasets with fixed distributions random, medium, medium-expert and expert Table shows an average increase of 42.1% for Offline DV2 and 32.5% for DrQ+BC compared to 10.8% for BC when we scale the dataset size 500,000, showing that the reinforcement learning algorithms make far better use of increased data than supervised bchavioral cloning. However, crucial difference between Offline DV2 and DCQ+BC is that DCQ handles larger oflline dalasets far more readily. DrQ+BC policy training for the same number of epochs on 500,000 and 100,000 observations takes and 1.6 hours respectively on a V100 GPU, This is significantly quicker than Offline DV2, which takes 10 hours to train on 100,000 observations; we discuss this further in Appendix D.4.1. This significant computational discrepancy leads clear open problem:
Challenge 4: How can we scale model-based methods to larger datasets?
12

---

Published in Transactions on Machine Learning Research (07 (27)))
6 Conclusion and Limitations
In this paper, we took the first steps towards establishing fully open-sourced benchmarking tasks featuring broad range of behavioral policies and compelitive baselines for offline reinforcement learning from visual observations. Until now, work in this space has been nascent, with ad-hoc analyses leading to unclear comparisons. To address the lack of meaningful evaluations and comparative analyses in this space, we provided a set of straightforward and standardized benchmarking tasks that follow popular low-dimensional equivalent experiment setups and presented competitive model-based and model-free baselines. We analyzed key factors that help explain the performance of these approaches, while also demonstrating their ability to generalize in more challenging settings that are unique to visual observations.
With particular focus the DeepMind Control Suite, which features wide array of tasks of varying difficulties, we hope that V-D4RL will be useful to practitioners and researchers alike and that it will provide a springboard for developing oflline reinforcement learning methods for real-world continuous-contro problems and spark further progress in this space. Complementing and extending V-D4RL to more domains that feature complex manipulation tasks and physical robotics systems presents an exciting direction for future rescarch.
Acknowledgements
Cong Lu and Tim G. J. Rudner funded by the Engineering and Physical Sciences Research Council (EPSRC). Philip J. Ball is funded through the Willowgrove Studentship. Tim G. J. Rudner is also funded by Qualcomm Innovation Fellowship. We are grateful to Rafacl Rafailov for sharing the data used with LOMPO and Edoardo Cetin for his NumPy Array replay buffer implementation. We thank Samuel Daulton, Clare Lyle, and Muhammad Faaiz Taufiq for detailed feedback on an early draft of this paper. The authors would also like to thank the anonymous TMLR reviewers for constructive feedback that helped improve the paper.
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline rein- forcement learning. In International Conference on Machine Learning, 2020.
Arthur Argenson and Gabriel Dulac- Model-based offline planning. In International Conference Learning Representations, 2021.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103 139, 1995.
Philip J Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang (eds.), Proceedings of the 98th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 619-629. PMIRR, 2021.
Ivan Bratko, Tanja Urbančić, and Claude Sammut. Behavioural cloning: Phenomena, results and prob- lems. IFAC Proceedings Volumes, 28(21):143 149, September 1995. ISSN 1474-6670. doi: 10.1016/
41474-6670
Christopher P. Burgess, Trina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Undorstanding discntangling in 3-vac. CoRR, abs/1804.03399, 2018.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and batch reinforeement learning. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. doi: (0.15607/RSS.2200...) 2020
13

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1518 1528. PMLR, 18 24 Jul 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learming via sequence modeling, In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Newral Information Processing Systems, volume 34, pp. 15084 15097. Curran Associates, Inc., 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research (JMLR), 6(18):503-55) -566, 2005.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Tgor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In 5th Annual Conference on Robot Learning, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data- driven reinforcement learning, 2021.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offiine reinforcement learning. In Thirty- Fifth Conference Neural Information Processing Systems, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2062-2062, 2019.
Caglar Gulechre, Ziyu Wang, Alexander Novikov, Tom Le Painc, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi, Matt Hoffman, Ofir Nachum, George Tucker, Nicolas Heess, and Nando deFreitas. Rl unplugged: Benchmarks for offline reinforcement learning, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/ abs/1812.05905, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent. dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, pp. 2555 2565, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020a.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. arXiv preprint 0011009000, 2020b.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Courna- peau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Picrre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Wockesser, Hameer Ab- basi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825): 357-362, September 2020. doi: 10. (038//411880000-2222
14

---

Published in Transactions on Machine Learning Research (07/2023)
Arnav Kumar Jain, Shiva Kanth Sujit, Shruti Joshi, Vincent Michalski, Danijar Hafner, and Samira Ebrahimi Kahou. Learning robust dynamics through variational sparse gating. In Deep RL. Workshop NewrIPS 2021, 2021.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision- based robotic manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 651 673. PMLR, 29 31 Oct 2018.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 21810 21823. Curran Associates, Inc.,
2020.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Doshi-Velez. Robust and efficient. transfer learning with hidden parameter markov decision processes. Guyon, U. V. Luxburg, Bengio, H. Wallach, Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
lly Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1179 1191. Curran Associates, Inc., 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should run oflline reinforcement learning or behavioral cloning? In Deep RL Workshop NearIPS 2021, 2021.
Vitaly Kurin, Alessandro De Palma, Nya Kostrikov, Shimon Whiteson, and M. Pawan Kumar. In defense of the unitary scalarization for deep multi-task learning, 2022.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6405 6416, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19884 19895. Curran Associates, Inc., 2020.
Timothée Lesort, Natalia Díaz Rodriguz,, Jean-Francohi Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-30 392, December 2018. TSSN 0893-6080. doi: htw.://6...eueee.218.......
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, and Stephen J. Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2022.
Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. year, 1000 km: The oxford RobotCar dataset. The International Journal of Robotics Research, 36(1):3-15, November 2016. ISSN 0278-3649, 1741-3176. doi:
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning, 2021.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in vari- ational autoencoders. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4402 4412. PMLR, 09 15 Jun 2019.
Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal prodictive coding for model-based planning in latent space. In Marina Meila and Tong Zhang (eds.), Proceealings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8130 8139. PMLR, 18 24 Jul 2021.
D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Newral Networks (ICNN'94), volume 1, pp. 55-60 vol.1. IEEE, 1994. doi: 10.1109/icnn. 199......99..71113.
Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforccment learning by latent imagina- tion without reconstruction. In 2021 IEEE International Conference on Robolics and Aulomation (ICRA), pp. 9209-4215, 2021. doi: 10.1109 10.1109/TCRA440 TCRA48506 2021 .9500344.
Matthew F. Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop Representation Learning for NLP (RepL4NLP-2001), pp. 7 14, Florence, Italy, August 2019. Association for Computational Linguisties. doi: 00.18653/v1/w199-1
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Proceedings of the 3rd Conference Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pp. 1154-1168. PMLR, 2021.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In Marina. Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8787 8798. PMLR, 2021.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Con- ference on Machine Learning, 1CML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97, pp. 5331-5340. PMLR, 2019.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. arXio preprint wwXiv:2006.16971, 2020.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Plan ning to explore via self-supervised world models. In ICML, 2020.
16

---

Published in Transactions on Machine Learning Research (07/2023)
Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. ViNG: Learning Open-World Navigation with Visual Goals. In IEEE International Conference Robotács and Automation (ICRA), 2021. URL https://arxiv http://wwwrrg/bbb/22/...5550
Shagun Sodhani, Ludovic Denoyer, Pierre-Alexande Kamienny, and Olivier Delalleau. MTEny environment interface for mulit-task reinforcement learning. Github, 2021.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforeement learning. In International Conference on Learning Representations, 2020.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite a challenging benchmark for reinforcement learning from pixels. arXiv preprinl arXiv:2101.02712, 2021.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning. Springer US, second edition, 1992. ISBN
978146136600 46166666,,, 9781461536185 doi: 10. 11617//771-14411-3311
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yolam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess, dm control: Software and tasks for continuous control, 2020.
Yifan Wu, George Tucker, and Ofrr Nachum, Behavior regularized offline reinforcement learning, 2019.
Yiting Xie and David Richmond. Pre-training grayscale ImageNet improves medical image classification. In Laura Leal-Taixé and Stefan Roth (eds.), Computer Vision - ECCV 2018 Workshops, pp. 476-484, Cham, 2019. Springer International Publishing. ISBN 978-3-030-11024-6.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-angeented reinforcement learning. arXiv preprint arXiv:2107.096/ 2021a.
Denis Yarals, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual coutinuous control: Improved auggeetedented reinforcement learning, 2021b.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image angmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021c
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving datasct for heterogeneous multitask learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020a. doi: 10.1109 (tt00//eypP42002220.0000
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.
14129 14142. Curran Associales, Inc., 2020b.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative oflline model-based policy optimization, 2021.
Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden-paramest block MDPs. In International Conference on Learning Representations, 2021.
Luisa. Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Varin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: Variational Bayes-adaptive deep RL via meta- veetalearning. Jour- nal of Machine Leurning Research (JMLR), 22(289):1 39, 2021.
17

---

Published in Transactions on Machine Learning Research (07 (02)))
Supplementary Material
Table of Contents
A Offline Dataset Characteristics
19
B Algorithmic Details
19
B.1 Offiine DV2
20
B.2 DrQ+BC
20
B.3 CQL
20
C Hyperparameter and Experiment Setup
21
C.1 Offline DV2
21
C.2 DrQ+BC
22
C.3 Behavioral Cloning
22
C.4 CQL
23
C.5 LOMPO
23
C.6 Computational Cost.
D Further Results and Ablation Studies
23
D.1. Full Tabular Distracted Results
23
D.2 Further Offline DV2 Results Multitask Datasets
25
D.3 DrQ+BC Expert Ablation Studies
25
D.4 Analysis on World Models for Visual Observations
25
D.5 Understanding Model-Basee Extrapolation
26
18

---

Published in Transactions on Machine Learning Research (07 (072))
A Offline Dataset Characteristics
We provide explicit statistics on the returns of each episode for the datasets used in our main evaluation. This provides reasonable proxy to how diverse each dataset is.
Table 5: Full summary statistics of per-episode return in the V-D4RL benchmark.
Datasct
Timesteps
Mean
Std. Dev.
Min.
P25
Median
P75
Max.
random
100K
42.3
8.7
30.0
34.9
41.3
46.8
74.6
mixed
100K
144.5
155.9
10.9
44.3
69.4
162.4
604.9
walker
medium
100K
439.6
48.4
176.2
423.1
445.5
466.7
538.0
medexp
200K
704.1
267.7
176.2
445.5
538.0
969.1
990.6
expert
100K
969.8
12.4
909.2
963.9
969.1
979.5
990.6
random
100K
6.6
2.6
1.1
4.7
6.3
8.4
16.3
mixed
200K
191.2
144.6
2.5
48.3
191.9
303.4
473.8
cheetah
medium
100K
523.8
25.5
325.3
509.1
524.2
538.3
578.3
medexp
200K
707.0
184.9
325.3
524.2
578.3
894.1
905.7
expert
100K
891.1
11.2
843.0
886.9
894.2
898.5
905.7
random
100K
1.1
0.8
0.0
0.5
1.0
1.5
5.7
mixed
600K
275.7
176.1
0.0
93.7
341.7
423.1
529.0
humanoid
medium
100K
573.0
16.7
526.5
560.6
572.9
584.9
609.4
medexp
200K
715.9
146.1
526.5
572.9
620.6
877.4
889.8
expert
100K
858.1
42.4
631.8
846.4
877.6
885.5
889.8
We compare this to the LOMPO datasets and find that they more widely distributed, due to the differing data-collection method.
Table 6: Summary statistics of per-episode return in the LOMPO DMC walker-walk datasets.
Dataset
Timesteps
Mean
Std. Dev.
Min.
P25
Median
P75
Max.
mixed
100K
208.9
144.1
33.3
75.1
172.4
340.5
496.7
walker
medexp
100K
674.4
92.9
501.3
596.8
679.8
752.5
869.1
expert
100K
920.6
81.6
17.9
905.9
950.3
957.9
987.0
As we see in Table 6 and Figure 3, the LOMPO walker-wald walk expert dataset has standard deviation roughly higher than our expert dataset and has an extremely wide [min, max] range. Furthermore, whilst our medexp dataset bimodal, the LOMPO medexp dataset's 's returns are continuous progression. This reflects that the LOMPO data is sampled from the second half of replay buffer after medium-level performance is attained, akin to the medium-replay (mixed) datasets.
A.0.1 Broader Issues with Visual Data
Large datasets consisting of images often contain systematic biases, which can damage generalization. The datasets constructed in this paper are all synthetic from simulated reinforcement learning environments. However, as we move towards applying oflline RL from visual observations lo real-world tasks, it is important to take these potential dangers into account and extend existing work algorithmic fairness from computer vision our setting.
B Algorithmic Details
We provide additional details for both algorithms here and indicate where modifications have been made.
19

---

Published in Transactions on Machine Learning Research (07 (0722)23)
walker_ walk medium_ _rppla
walker_ _alk medium_ _xpeert
walker walk expert
60
200
V-D4RL LOMPO
150
40
Count
50
200
400
600
200
600
800
1000
0
200
400
600
800
1000
Episode Return
Figure 5: Comparison of the episodic returns from the LOMPO and V-D4RL. We see LOMPO has signifi- cantly more diversity in the medium-expert and expert datasets. Note that there is a single episode in the LOMPO expert dataset which has a low return of 17.9.
B.1 Offline DV2
In the oflline setting, it suffices to simply perform one phase of model training and one phase of policy training for the DreamerV2 (Hafner et al., 2020b) algorithm. Each episode in the offline dataset is ordered sequentially to facilitate sequence learning. To this instantiation of DreamerV2, we simply add reward penalty corresponding to the mean disagreement of the dynamics ensemble. During standard DreamerV2 policy training, imagined latent trajectories ((*r, (r)) are assigned reward r, F. [go (- 8+)] according to the mean output of the reward predictor. The imagined latent states consist of a deterministic component h, implemented as the recurrent state of a GRU, and a stochastic component 2, with categorical distribution. The logits of the categorical distribution are computed from an ensemble (with input h₂) over which we compute the mean disagrcement.
B.2 DrQ+BC
Here, we simply modify the policy loss term in DrQ-v2 (Yarats et al., 2021a) to match the loss given in Fujimoto Gu (2021). Following the notation from Yarats et al. (2021a), the DrQ-v2 actor na is trained with the following loss:
(D) = -E IQ0 (h,, ax)l
where h; = fe (aug (s:)) the encoded augmented visual observation, a; = =s (he) + e is the action with clipped noise to smooth the targets c ~ clip (N (0, -c,).. Note that we also do not update encoder weights with the policy gradient. We also train a pair of two fully connected Q-networks, which both use features from a single encoder, and take their minimum when calculating target values and actor losses. The resultant algorithm can be viewed as TD3 (Fujimoto et al., 2018), but with decaying smoothing noise parameters
In DrQ+BC, this loss becomes:
(D) -E,,aaD [AQo (h,, a.) (T (h:) a
where A ((,,,,)) [(h.a)]] is adaptive normalization term computed over minibatches. is behav-
ioral cloning weight, always set 2.5 in Fujimoto & Gu (2021), which we also adopt. We experimented performing extensive grid search over a, but did not observe any noticeable benefit in our ofline datasets by deviating away from the default value.
B.3 CQL
Our CQL implementation was also built on top of the DrQv2 codebase for comparability. We use the CQL(H) objective with fixed weight, as we found this be the most performant. This corresponds choosing the KT-divergence uniform prior the regularizer R(n) in Kumar et al. (2020). Concretely,
20

---

Published in Transactions on Machine Learning Research (07 (02)))
the Q-function objective becomes
BaassD - B kh)) xx))) |
min acQu, BawD Q
log exp(Q(s, a)) - (a (Q(s
where ACQL is a trade-off factor, à refers to the empirical behavioral policy and _ to the empirical Bellman operator that backs up a single sample. This approximated by taking gradient stops and sampling actions from the given bounds.
C Hyperparameter and Experiment Setup
C.1 Offline DV2
Our Offiine DV2 implementation was built on top of the official Dreamer V2 repository at: https://gite com/ com/dani /tdd//aar/drraarrrrrrrrr with minor modifications. The code was relcased under the MIT License. Table 7 lists the hyperparameters used for Offline DV2. For other hyperparameter values, we used the default values in the DreamerV2 repository.
Table 7: Offline DV2 hyperparamoters.
Parameter
Valuc(s)
ensemble member count (K)
7
imagination horizon (H)
5
batch size
64
sequence length (L)
50
action repeat
observation size
[64, 64]
discount (7)
0.99
optimizer
Adam
learning rate
{model = x 10 actor-critic = 8 10-5}
model training epochs
800
agent training epochs uncertainty penalty uncertainty weight (A)
2,400
mcan disagreement in [3, 10]
We found a default value of works for most settings. The only settings where this changes are À 3 for both random datasets and A 8 for the walker-walk mixed dataset.
For the penalty choice, chose mean disagreement of the ensemble because it comprises half of the ensemble variance, which was shown to be an optimal choice for offline model-based reinforcement learning in Lu et al. (2022). We found that the other component of the ensemble variance, the average variance over the ensemble, was uninformative and so discarded it.
C.1.1 Understanding the impact of hyperparameters
We run additional experiments to understand whether the key default online hyperparameters perform well in the offline setting; for proprioceptive environments, this is not always the case (Lu et al., 2022). Using V-D4RL we are able to better understand the impact of these design choices, and we show the results in Table 8. Concretely, the optimal hyperparameters are very different between Offlinc DV2 and the online DreamerV2. In the oflline selting, it is better to increase in batch size from 16 to 64, and decrease the imagined horizon (H) from 15 to 5. With our setup, the increase in batch size takes around 76% more wall- clock time per batch, but this leads of data that is trained on. We believe the reduction in horizon required due to the lack of online 'remedial' sampling (i.e., sampling data from the true environment that corrects for errors in the imagined rollouts),
huge gain performance when normalizing for the actual amount
21

---

Published in Transactions on Machine Learning Research (07/2023)
thus resulting in exploitation when training solely the offline data, with uncertainty penalty. However, was also observed in Lu et al. (2022), choosing too low horizon (i.e., H 2) is ill-advised, as this results in over-reliance on the offline state distribution, with less chance of policy improvement. Consoquently, we choose batch size = 16 and H = 5 for all our experiments.
Table 8: Ablations on hyperparameters of Offline DV2 where they differ from the online DreamerV2 We report final performance mapped from [0, 1000] to [0, 100) averaged over six seeds.
Environment
Default
batch size=16
H=2
H=15
random
28.7 +33.0
22.6 + 8.4
31.9 = 7.6
16.9 + 8.5
mixed
56.5 +18.1
27.4 +16.5
54.7 114.
45.5 +15.1
walker
medium
34.1 119.7
33.3 121.4
47.8 118.8
21.0 120.3
medexp
43.9 134.
24.5 +30.1
22.4 118.1
22.9 +25.7
expert
4.8
2.7 + 1.0
3.4 = 0.6
3.1 0.8
random
31.7
29.3 1
29.8 + 4.3
32.4 3.7
mixed
61.6
53.7 +
56.5 = 2.7
61.3 2.8
cheetah
medium
17.2 1 3.5
17.4 1 8.0
17.2 1 6.2
14.3 1 4.7
medexp
10.4 + 3.5
9.4 + 5.3
8.1 = 3.8
8.7 + 5.2
expert
10.9 + 3.2
11.1 + 5.2
8.1 + 4.8
9.7 + 3.7
Average Change
-21.6%
-6.1%
-15.8%
C.2 DrQ+BC
Our DCQ+BC implementation was built top of the official DrQ-v2 repository at: https://github. com/ facebookresearcch/wae The code was released under the MIT License. Table 9 lists the hyperparameters used for DrQ+BC. For other hyperparameter values, we used the default values in the DrQ-v2 repository. Due to the size some of our offline datasets, we found the defanlt replay buffer would not scale to the offline datasets. Thus, we used a NumPy (Harris et al., 2020) array implementation instead.
Table DCQ+BC hyperparamcters.
Parameter
Value
batch size action repeat observation size
256
[84, 84]
discount (7)
0.99
optimizer
Adam
learning rate
x 10
agent training epochs n-step returns.
256
3
Exploration stddev. clip Exploration stddev. schedule. BC Weight (a+)
0.3
Cnnar((10. 0.1, 500000)
2.5
We also further tuned a within (1.5, 2.5, 3.5} but as Fujimoto & Gu (2021) found, we did not observe any noticeable benefit from deviating away from the default value for a.
C.3 Behavioral Cloning
Our BC implementation shares the exact same policy network and hyperparameters DrQ+BC but just minimizes MSE on the offline data. Consoquently, we must also optimize the learned encoder using the supervised learning loss (unlike in DrQ+BC, where the TD-loss only contributes the encoder representation learning, and not the policy loss).
22

---

Published in Transactions on Machine Learning Research (07 (07/2023)
C.4 CQL
Similarly to BC, our CQL implementation is also based on the same networks and hyperparameters in DrQ+BC. CQL introduces one extra hyperparameter, the trade-off factor aCQL. We perform hyperpa- rameter sweep for this over the range: (0.5, 1,2,5, 10, 20). We chose the following values per environment:
Table 10: CQL trade-off factor per environment for Walker and Cheetah. Humanoid omitted as all choices performed equally.
Dataset
Trade-off Factor (aCQL)
random
0.5
mixed
0.5
walker
medium
2
medexp
2
expert
random
0.5
mixed
0.5
cheetah
medium
10
medexp
expert
20
C.5 LOMPO
For our LOMPO evaluation, we used the official repository at: https://github. com//mraafa1111//0000... The code open-sourced without license. We perform hyperparameter search over the uncortainty weight À in the range (1,5). The default value used in Rafailov et al. (2021) is x= 5, but we found that X = worked better for random datasets. The accompanying data for the DMC Walker- Walker-Walk data from Rafailov et al. (2021) was very kindly provided by the authors.
C.6 Computational Cost
The experiments in this paper were run on NVIDIA V100 GPUs. On the standard V-D4RL 100K datascts, DCQ+BC took 1.6 hours, Offline DV2 took 10 hours, and CQL took 12 hours.
Since we wish to compare several offline algorithms using the same dataset, we define a notion "offline epoch" for all algorithms to show training performance over time. We simply normalize the total number of gradient steps, so training progress falls within [0, 1000).
Further Results and Ablation Studies
D.1 Full Tabular Distracted Results
We further present full tabular results from Figure 3 in Section 5.1 for both Offline DV2 and DrQ+BC in Table 11 and Table 12 respectively. The highlighted base unshifted results in both tables are the same as in Table for Oflline DV2 and Table 4 for DrQ+BC.
23

---

Published in Transactions on Machine Learning Research (07/2023)
Table 11: Offline DV2 shows surprisingly good generalization to unseen distractions and can handle mixed datascts. Final mean performance averaged over six random sceds and base undistracted performance is highlighted. The environment used is walker-wall random.
Shift Severity
Evaluation Return
% Shifted
Original
Distraction Train
Distraction Test
0%
28.7
25.9
20.0
25%
28.1
22.0
14.0
low
50%
22.5
16.0
17.7
75%
16.3
21.9
20.1
100%
29.4
29.7
18.6
0%
28.7
21.0
15.1
25%
28.6
19.1
14.9
moderate
50%
19.8
20.5
11.2
75%
24.4
20.0
11.9
100%
20.1
22.9
15.5
0%
28.7
14.1
10.7
25%
24.6
10.1
6.4
high
50%
10.7
19.1
6.0
75%
20.3
18.9
5.54
100%
3.2
25.4
5.2
Table 12: DrQ+BC can adapt to multiple different distractions but is extremely brittle to settings it has not seen and struggles to generalize. Final mean performance is averaged over six random seeds and base undistracted performance is highlighted. The environment used cheetah-run medexp.
Shift
Evaluation Return
% Shifted
Severity
Original
Distraction Train
Distraction Test:
0%
79.1
0.5
1.1
25%
73.9
77.5
8.7
low
50%
72.3
82.2
8.5
75%
67.6
85,5
7.6
100%
4.2
86.6
4.5
0%
79.1
0.2
0.9
25%
74.6
77.9
3.1
moderate
50%
68.0
83.6
3.4
75%
55.4
86.1
3.7
100%
0.9
86.7
2.0
0%
79.1
0.1
0.5
25%
76.7
76.8
1.3
high
50%
66.7
82.5
1.1
75%
48.4
84.8
1.2
100%
0.4
86.1
0.9
24

---

Published in Transactions on Machine Learning Research (07 (07/2023) 2023)
D.2 Further Offline DV2 Results Multitask Datasets
To complement the results in Table 3, show additional results for Offline DV2 on medexp multitask data. Here, Offline DV2 learns a slightly reduced quality policy compared to that on the base environment, but experiences no deterioration in performance on the test environments in walker or cheotah.
Table 13: Evaluation on the DMControl-
benchmark using mederp data for Offine DV2. Normal- ized performance from [0, 1000] to [0, 100) is averaged over six seeds. Offline DV2 shows a strong ability to gencralize to the extrapolation test environments.
Eval. Return
Algorithm
Environment
Train Tasks
Test Interp.
Test Extrap.
walker cheetah
23.2
16.5
19.8
Offline DV2
8.2
7.2
9.6
D.3
DrQ+BC Random-
Random-Expert Expert Ablation Studies
walker-walk random-expert
cheetah-run random-expert
800
Eval Return
600
400
200
BC
DrQ+BC
%
200
400
600
800
1000
0
200
400
600
800
1000
Offline Epochs
Figure 6: Comparison of DrQ+BC and BC on random-expert datasets with 500,000 of each datatype, results averaged over six seeds.
Analogously Fujimoto & Gu (2021), we continue to see that DrQ+BQ does well on mixed datasets with high reward with experiments on the concatenated random-expert datasets. Surprisingly, we behavioral cloning also does reasonably well on cheetah-run with random-expert data. This is likely to be because there is significant distribution shift between the states of random and expert trajectories for that environment.
This would lcad to minimal destructive interference between similar states which contain completely different actions, as these state distributions largely do not overlap. This is in contrast to the mediun-expert dataset, which experiences higher state overlap between its two modes (i.e., states generated by a medium and an expert policy respectively), resulting in a marginal "average" action being learned which is likely suboptimal, as can be seen by the poor performance of the BC agent in Tables and
D.4 Analysis on World Models for Visual Observations
In this section, we seek to better understand the differences between model-based and model- algorithms by presenting further analysis for the Offline DreamerV2 algorithm. In particular, we begin address our open questions and understand why Offline DreamerV2 deals well with unseen distractions but has trouble handling larger more narrowly concentrated datasets.
D.4.1 Model Training Time
One of the major factors preventing algorithms that use an RSSM like Offline DV2 and LOMPO from scaling larger datasets is the time required for model training. The standard number of epochs of model training for our 100,000 datasets in Section 4 is 800 epochs, which takes around six hours V100 GPU. This scales
25

---

Published in Transactions on Machine Learning Research (07/2023)
linearly with the number of training points if maintain the same batch size. As show in Figure this is mandatory for performance and is a fundamental limitation of current model-based methods. We can see that the evaluated return increases and becomes more tightly distributed as the number of training epochs increases up to 800.
Offline DV2 on 400K random dataset
450
11292
400
11290
Return
Model Loss
11288
Eval. Return
Model
11286
300
11284
250
11282
200
100
300
600
700
800
Model Train Epochs
Figure 7: Evaluated return against model train epochs for Offline DV2 on a random dataset of size 400,000, We train a single model up to 800 epochs and evaluate the model periodically on three random seeds. We see that full model training, which scales linearly with training points, is mandatory for good performance.
D.4.2 Uncertainty Quantification
In Table 1, see that Offline DV2 is considerably weaker the expert datasets that have narrow data distributions. On these expert datasets, we found that the model uncertainty often had lower variance across trajectory compared to models trained on other datasets, and was thus uninformative. We give summary statistios for the uncertainty penalty on states generated by random policy in the walker-walk environment in Table 14. Since we followed the author's Dreamer implementation (Hafner et al., 2020b), only the stochastic portion of the latent state predicted by the ensemble; this may mean crucial calibration is lost when ignoring the impact of the deterministic latent. Future work could involve investigating SVSG (Jain et al., 2021), extension to DreamerV2 with purely stochastic latent state.
Table 14: Mean and standard deviation of the uncertainty penalty computed over 1,024 states sampled from the 'random' dataset on the walker-walt euvironment. Note that the model trained on expert data reports considerably smaller and tighter uncertainty values (compared 'medium' and 'medexp'), despite the large distribution shift that exists from 'random' to 'expert' data. Instead, we'd we'd expect the 'expert' trained model to exhibit the largest mean uncertainty when tested on the 'random'' data.
Dataset Type
Mean
St.d.
random mixed medium medexp expert
0.223
0.040
0.226
0.035
0.341
0.034
0.338
0.034
0.262
0.020
D.5 Understanding Model-Based Extrapolation
Finally, we investigate the reasons why our model-based bascline, Offline DV2, appears to generalize unseen
distractions as seen in Section 5.1. The RSSM includes latent decoder for the visual observations, which is primarily used during training to provide self-supervised reconstruction loss. During deployment, the
26

---

Published in Transactions on Machine Learning Research (07/2023)
policy solely relies on the latent states from the encoder, and the decoder is effectively discarded. However, can still use the decoder at test time in order to understand the information contained in the latent states. To do this, we take the latent states generated during rollouts in the extrapolation experiments, and 'translate' them into natural images by passing them through the decoder.
We first investigate the setting where each RSSM is trained only on data from a single distraction and then transferred. This setting is the most successful, can be seen in Figure Thus, in Figure 8 we first show the ground truth observation provided to the agent, then below this we show the decoder output reconstructed from the latent. In many cases, visual surroundings. This is an indication that despite the decoder overfitting to the exogenous factors in the visual input (e.g,, background, colors), the latent captures the salient state information of the agent (e.g.. joint positions and angles), explaining the strong test-time transfer performance.
recover the original pose despite ending up in different
In Figure 3, we note that scttings with a mixture of distractors often had worse test-time transfer perfor- mance than those with a single distractor. To explain this, we consider an RSSM trained images with combination of two fixed distractors, and examine the reconstruction of an episode under third distraction. We see in Figure 9, that the RSSM latents are split between the two modes of the data and the reconstruction switches robot color and background midway. This significantly confuses the recovered pose of the walker and likely causes a degradation in performance. Disentangling the factors of variation (Burgess et al., 2018; Mathieu et al., 2019) represents important future work for offline RL from visual observations.
27

---

Published in Transactions on Machine Learning Research (07/202) 2023)
Figure 8: Reconstruction of random episodes shifted by fixed distractor by models trained data shifted by a different distractor. The top row shows the original ground truth, and the bottom two rows the model reconstruction for a different RSSM. We can see that in many cases, the original pose of the walker still able to be recovered.
Figure 9: Bottom row shows the reconstruction of distracted random episode (on the top row) using an RSSM trained on a mixture of two differently distracted environments. We can see that the reconstructed states are split between the two modes the data and the reconstruction switches robot color and background midway.
28


Reviews:
[{"review_id": "lNY3PBYKKq", "reviewer": "Reviewer_YCLk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nStandardizing benchmarks for visual offline RL is an\nimportant topic as papers are otherwise forced to use\ntheir own datasets or datasets privately sent from the\nauthors of other papers. Creating this benchmark is a\nserious undertaking that I commend the authors for.\nI believe the paper has a great potential to be a snapshot\nof the current state of methods and results in this space.\n\n# Weaknesses\n\nThe main weakness is in the comparison to LOMPO:\n\n**W1.** The numbers in Table 2 should correspond to\n   [Table 1 of the LOMPO paper](http://proceedings.mlr.press/v144/rafailov21a/rafailov21a.pdf),\n   but the LOMPO results in this paper do not match the originally\n   published results. What is causing this difference?\n   This makes the claim of Offline DV2 and DrQ+BC outperforming LOMPO\n   potentially not sound because the originally published LOMPO\n   results give better performance in some settings.\n\n**W2.** The walker-walk is only one setting considered in LOMPO:\n   Omitting the other environments used in the LOMPO paper\n   also make the claim of Offline DV2 and DrQ+BC outperforming LOMPO\n   potentially not sound.\n\nOne other minor weakness is:\n\n**W3.** It seems slightly confusing to map from the original environmental\n   rewards from [0, 1000] to [0,100]. I much prefer the normalized\n   scores, e.g. from D4RL, that normalize using the dataset's score.\n   It would be nice to add the expert's performance to the figures\n   in some way because there is a significant gap:\n   Table 5 shows the expert on humanoid-walk attains a\n   mean reward of 858 while Figure 2 shows that the best model\n   isn't able to surpass 100.\n\nAnd an extremely minor weakness:\n\n**W4.** I do not find the open problem statements very interesting as\n   they are generically applicable to most\n   machine/reinforcement learning settings\n   (1. can a single algorithm outperform all baselines? and\n   2/3. how to improve generalization to visual\n   distractions/unseen dynamics?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a standardized benchmark and baselines\nfor offline reinforcement from visual observations using\nthe walker-walk, cheetah-run, and humanoid-walk environments\nfrom the DeepMind control suite.\nSection 2 describes the basic offline RL setup.\nSection 4.1 describes the environments, how expert\npolicies were obtained by training a SAC agent on the\nproprioceptive states, and how the data was obtained from\nthe expert's replay buffer.\nSection 4.2 describes a model-based baseline based on DreamerV2\nand a model-free baseline based on DrQ-v2 with a behavioral\ncloning regularization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**RC1.** Clarify the inconsistent and incomplete comparisons to LOMPO (for W1/W2)\n\n**RC2.** Normalize the scores using the dataset (as in D4RL) or make\n   the dataset's mean reward more visible, e.g., as a reference line in Figure 2 (for W3)\n\nI have one other question out of curiosity:\n\n**Q1.** Did you consider adding BC to the policy in Dreamer-v2?\n", "broader_impact_concerns": "I do not have any concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gJByh49eVL", "reviewer": "Reviewer_jAZh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper creates a new dataset that fulfill the existing gap of missing public availabile visual observations for offline reinforcement learning. In this sense, this paper is very intriguing. It systemically considers both model-based and model-free algorithms, which provides a complete study for RL. It studies several key aspects including: Robustness to Visual Distractions, Generalization Across Environment Dynamics, and Improved Performance with Scale. \n\nWeakness: although the v-d4rl dataset is a nice contribution, the environment included is limited. Concretely, only walker-walk, cheetah-run, and humanoid-walk are included, which makes the scale of the dataset small. From my opinion, more experiments/environments are needed for this work to receive higher attention. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel benchmark called \"v-d4rl: Vision Datasets for Deep Data-Driven RL\" which focuses on offline RL using visual observations of DMControl Suite (DMC) tasks (Tassa et al., 2020). This benchmark builds upon the design principles of the popular d4rl benchmark (Fu et al., 2021). It is the first publicly available benchmark for continuous control that encompasses a wide range of behavioral policies.\n\nSecond, to address the challenges of realistic offline RL from visual observations, this paper identifies three crucial aspects: robustness to distractions (Stone et al., 2021), generalization across dynamics (Zhang et al., 2021), and improved performance for offline reinforcement learning at scale. To establish baselines for offline RL from visual observations, this paper adapted state-of-the-art online RL algorithms, namely DreamerV2 (Hafner et al., 2020b) and DrQ-v2 (Yarats et al., 2021a). These modified algorithms serve as simple baselines for measuring progress and guiding future advancements in this domain.\n\n\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address accordingly as suggested above.", "broader_impact_concerns": "I do not observe any explicit discussion on the Broader Impact Concerns of the topic studied.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vD3qp86Nud", "reviewer": "Reviewer_W9te", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nA thorough analysis of what characteristics should have for an offline RL benchmark is provided, together with empirical evaluations.\n\nThis paper proposes some simple model-based and model-free baselines for offline RL from visual observations, which are based on the classic algorithms DreamerV2 and DrQ under online learning. The authors have made reasonable and effective optimizations for offline training. The experimental results show the advantages of the model-based algorithm in datasets with broad data distributions, such as random/mixed, and the advantages of behavior cloning in datasets with high data quality, which are consistent with expectations. \n\nWeaknesses:\n\nDespite the detailed introduction to the dataset and baseline algorithms in this paper, there are still some areas that need improvement to enable researchers to efficiently conduct further studies based on this paper, from a more detailed perspective. The main shortcomings of the paper are as follows:\n\n1. Regarding the dataset, some details are not clearly introduced, which may cause confusion for researchers.\n\n>a) It is recommended to add a column to Table 1 to show the mean return of the dataset or provide results under an expert policy similar to D4RL.\n\n>b) Sections 5.1 and 5.3 introduce new datasets (datasets with Visual Distractions and larger-scale datasets) for evaluation, but the paper does not provide information about these datasets. Appendix A only provides information about the 15 basic datasets. Therefore, it would be better to list all available datasets and their characteristics.\n\n2. Open Problem 3 is not a problem unique to visual observation offline RL, but rather a problem of extrapolation error caused by unseen data, which is common in offline RL, and more specifically, offline model-based RL. The author needs to provide a more in-depth discussion and insight.\n\n3. Other details:\n3a. DreamerV2 and DrQv2 are no longer state-of-the-art vision-based online reinforcement learning algorithms, and this statement should be corrected.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an offline RL benchmark with visual observations by considering some key features/requirements that should be satisfied. Two baselines evaluation on the presented benchmark is also provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Details about the dataset should be clarified.\n\nMore in-depth insights should be provided.\n\nSome statements should be carefully clarified, in more appropriate language.\n\nMore details are included in the pros and cons part.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance and the AE agrees.
Decision: Accept (poster)